2024-09-16T05:21:35.938 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T05:21:35.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T05:21:36.037 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162 branch: squid description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906162' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) mon_bind_msgr2: false sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFlkKbOObKZbHVFWJLBxZzrukUnFl1HZyMiqBSfFs4bQOtb5bqVNz173Cfq9Z4l7j78FXIzL9mhQ101oT5a5jQ= smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCFFkFoHe7x+wAmwAKEWqC7x/6QIMVmZhtDBz9W8oCwoQW1aVaiUfsADK3f9alaCQmDL7v69HQbzuq6llgt5KYM= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T05:21:36.037 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T05:21:36.038 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T05:21:36.038 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T05:21:36.040 INFO:teuthology.task.internal:Checking packages... 2024-09-16T05:21:36.060 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T05:21:36.060 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T05:21:36.060 INFO:teuthology.packaging:ref: None 2024-09-16T05:21:36.060 INFO:teuthology.packaging:tag: None 2024-09-16T05:21:36.060 INFO:teuthology.packaging:branch: squid 2024-09-16T05:21:36.060 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:21:36.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-16T05:21:36.250 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:21:36.252 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T05:21:36.259 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T05:21:36.259 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T05:21:36.266 INFO:teuthology.task.internal:Saving configuration 2024-09-16T05:21:36.279 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T05:21:36.285 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T05:21:36.310 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 05:16:02.028731', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICftBB6CihfARL49+dWxwZT4hIlEoL35hEve4bzsNU1c'} 2024-09-16T05:21:36.332 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 05:16:02.027432', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINuOE3683El9ebwfJkb6H6xbsOypm4U+y2gyrrpWmxSr'} 2024-09-16T05:21:36.332 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T05:21:36.339 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T05:21:36.339 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T05:21:36.339 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T05:21:36.406 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1c5a2b6e60>, signals=[15]) 2024-09-16T05:21:36.406 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T05:21:36.412 INFO:teuthology.task.internal:Opening connections... 2024-09-16T05:21:36.412 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:21:36.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:21:36.498 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:21:36.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:21:36.578 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T05:21:36.585 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-09-16T05:21:36.590 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-09-16T05:21:36.590 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Ubuntu" 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="22.04" 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_CODENAME=jammy 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:ID=ubuntu 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE=debian 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T05:21:36.638 INFO:teuthology.orchestra.run.smithi006.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T05:21:36.639 INFO:teuthology.orchestra.run.smithi006.stdout:UBUNTU_CODENAME=jammy 2024-09-16T05:21:36.639 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-09-16T05:21:36.661 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-09-16T05:21:36.667 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-09-16T05:21:36.667 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-09-16T05:21:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T05:21:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="Ubuntu" 2024-09-16T05:21:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="22.04" 2024-09-16T05:21:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T05:21:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_CODENAME=jammy 2024-09-16T05:21:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:ID=ubuntu 2024-09-16T05:21:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE=debian 2024-09-16T05:21:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T05:21:36.714 INFO:teuthology.orchestra.run.smithi063.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T05:21:36.714 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T05:21:36.714 INFO:teuthology.orchestra.run.smithi063.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T05:21:36.714 INFO:teuthology.orchestra.run.smithi063.stdout:UBUNTU_CODENAME=jammy 2024-09-16T05:21:36.714 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-09-16T05:21:36.737 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T05:21:36.745 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T05:21:36.750 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T05:21:36.750 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-09-16T05:21:36.753 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-09-16T05:21:36.760 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T05:21:36.766 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T05:21:36.767 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-09-16T05:21:36.799 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-09-16T05:21:36.811 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T05:21:36.871 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T05:21:36.886 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T05:21:36.886 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T05:21:36.886 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T05:21:36.887 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T05:21:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.891 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-16T05:21:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:21:36.892 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-09-16T05:21:36.937 INFO:teuthology.orchestra.run.smithi063.stdout:5.15.0-56-generic 2024-09-16T05:21:36.938 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.15.0-56-generic 2024-09-16T05:21:36.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get clean 2024-09-16T05:21:36.939 INFO:teuthology.orchestra.run.smithi006.stdout:5.15.0-56-generic 2024-09-16T05:21:36.939 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.15.0-56-generic 2024-09-16T05:21:36.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-09-16T05:21:37.056 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2024-09-16T05:21:37.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-16T05:21:37.212 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T05:21:37.228 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T05:21:37.289 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T05:21:37.324 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T05:21:37.324 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T05:21:37.325 INFO:teuthology.orchestra.run.smithi063.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T05:21:37.390 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T05:21:37.461 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T05:21:37.527 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T05:21:37.562 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T05:21:37.565 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T05:21:37.566 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T05:21:37.588 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T05:21:37.588 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T05:21:37.592 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T05:21:37.593 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T05:21:37.602 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T05:21:37.608 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T05:21:37.610 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T05:21:37.611 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T05:21:37.611 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T05:21:37.643 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T05:21:37.643 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T05:21:37.643 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T05:21:37.644 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T05:21:37.645 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T05:21:37.645 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T05:21:37.645 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T05:21:37.646 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T05:21:37.646 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T05:21:37.646 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T05:21:37.747 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T05:21:37.776 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T05:21:37.799 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T05:21:37.887 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T05:21:38.070 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T05:21:38.205 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T05:21:38.221 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T05:21:38.237 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T05:21:38.240 INFO:teuthology.orchestra.run.smithi063.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T05:21:38.244 INFO:teuthology.orchestra.run.smithi063.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T05:21:38.245 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T05:21:38.246 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T05:21:38.248 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T05:21:38.254 INFO:teuthology.orchestra.run.smithi063.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T05:21:38.255 INFO:teuthology.orchestra.run.smithi063.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T05:21:38.257 INFO:teuthology.orchestra.run.smithi063.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T05:21:38.257 INFO:teuthology.orchestra.run.smithi063.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T05:21:38.258 INFO:teuthology.orchestra.run.smithi063.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T05:21:38.260 INFO:teuthology.orchestra.run.smithi063.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T05:21:38.344 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T05:21:38.345 INFO:teuthology.orchestra.run.smithi063.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T05:21:38.355 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T05:21:38.355 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T05:21:38.371 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T05:21:38.400 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T05:21:38.406 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T05:21:38.406 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T05:21:38.406 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T05:21:38.407 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T05:21:38.425 INFO:teuthology.orchestra.run.smithi063.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T05:21:38.425 INFO:teuthology.orchestra.run.smithi063.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T05:21:38.425 INFO:teuthology.orchestra.run.smithi063.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T05:21:38.434 INFO:teuthology.orchestra.run.smithi063.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T05:21:38.434 INFO:teuthology.orchestra.run.smithi063.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T05:21:38.444 INFO:teuthology.orchestra.run.smithi063.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T05:21:38.450 INFO:teuthology.orchestra.run.smithi063.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T05:21:38.457 INFO:teuthology.orchestra.run.smithi063.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T05:21:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T05:21:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T05:21:38.462 INFO:teuthology.orchestra.run.smithi063.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T05:21:38.463 INFO:teuthology.orchestra.run.smithi063.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T05:21:38.469 INFO:teuthology.orchestra.run.smithi063.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T05:21:38.472 INFO:teuthology.orchestra.run.smithi063.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T05:21:38.473 INFO:teuthology.orchestra.run.smithi063.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T05:21:38.473 INFO:teuthology.orchestra.run.smithi063.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T05:21:38.474 INFO:teuthology.orchestra.run.smithi063.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T05:21:38.486 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T05:21:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T05:21:38.510 INFO:teuthology.orchestra.run.smithi063.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T05:21:38.556 INFO:teuthology.orchestra.run.smithi063.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T05:21:38.556 INFO:teuthology.orchestra.run.smithi063.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T05:21:38.557 INFO:teuthology.orchestra.run.smithi063.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T05:21:38.557 INFO:teuthology.orchestra.run.smithi063.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T05:21:38.558 INFO:teuthology.orchestra.run.smithi063.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T05:21:38.592 INFO:teuthology.orchestra.run.smithi063.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T05:21:38.592 INFO:teuthology.orchestra.run.smithi063.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T05:21:38.593 INFO:teuthology.orchestra.run.smithi063.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T05:21:44.489 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 16.0 MB in 2s (7,219 kB/s) 2024-09-16T05:21:44.874 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 16.0 MB in 3s (5,606 kB/s) 2024-09-16T05:21:45.981 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-16T05:21:46.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T05:21:46.061 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-16T05:21:46.274 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-16T05:21:46.275 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-16T05:21:46.393 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-09-16T05:21:46.411 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T05:21:46.471 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T05:21:46.471 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T05:21:46.471 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T05:21:46.472 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T05:21:46.472 INFO:teuthology.orchestra.run.smithi006.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T05:21:46.472 INFO:teuthology.orchestra.run.smithi006.stdout: python2.7-minimal 2024-09-16T05:21:46.472 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T05:21:46.473 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-09-16T05:21:46.474 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T05:21:46.474 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T05:21:46.474 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T05:21:46.475 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-09-16T05:21:46.475 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-09-16T05:21:46.476 INFO:teuthology.orchestra.run.smithi006.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T05:21:46.476 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-09-16T05:21:46.476 INFO:teuthology.orchestra.run.smithi006.stdout: thermald 2024-09-16T05:21:46.523 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-09-16T05:21:46.523 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T05:21:46.523 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T05:21:46.524 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T05:21:46.525 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages will be upgraded: 2024-09-16T05:21:46.525 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T05:21:46.577 INFO:teuthology.orchestra.run.smithi006.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T05:21:46.577 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 113 MB of archives. 2024-09-16T05:21:46.577 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T05:21:46.577 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T05:21:46.687 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2024-09-16T05:21:46.687 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2024-09-16T05:21:46.883 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T05:21:46.883 INFO:teuthology.orchestra.run.smithi063.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T05:21:46.883 INFO:teuthology.orchestra.run.smithi063.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T05:21:46.883 INFO:teuthology.orchestra.run.smithi063.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T05:21:46.884 INFO:teuthology.orchestra.run.smithi063.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T05:21:46.884 INFO:teuthology.orchestra.run.smithi063.stdout: python2.7-minimal 2024-09-16T05:21:46.884 INFO:teuthology.orchestra.run.smithi063.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T05:21:46.885 INFO:teuthology.orchestra.run.smithi063.stdout:The following additional packages will be installed: 2024-09-16T05:21:46.885 INFO:teuthology.orchestra.run.smithi063.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T05:21:46.885 INFO:teuthology.orchestra.run.smithi063.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T05:21:46.886 INFO:teuthology.orchestra.run.smithi063.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T05:21:46.887 INFO:teuthology.orchestra.run.smithi063.stdout:Suggested packages: 2024-09-16T05:21:46.887 INFO:teuthology.orchestra.run.smithi063.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T05:21:46.887 INFO:teuthology.orchestra.run.smithi063.stdout:Recommended packages: 2024-09-16T05:21:46.887 INFO:teuthology.orchestra.run.smithi063.stdout: thermald 2024-09-16T05:21:46.941 INFO:teuthology.orchestra.run.smithi063.stdout:The following NEW packages will be installed: 2024-09-16T05:21:46.941 INFO:teuthology.orchestra.run.smithi063.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T05:21:46.941 INFO:teuthology.orchestra.run.smithi063.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T05:21:46.941 INFO:teuthology.orchestra.run.smithi063.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T05:21:46.943 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages will be upgraded: 2024-09-16T05:21:46.943 INFO:teuthology.orchestra.run.smithi063.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T05:21:46.995 INFO:teuthology.orchestra.run.smithi063.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T05:21:46.995 INFO:teuthology.orchestra.run.smithi063.stdout:Need to get 113 MB of archives. 2024-09-16T05:21:46.995 INFO:teuthology.orchestra.run.smithi063.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T05:21:46.995 INFO:teuthology.orchestra.run.smithi063.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T05:21:47.025 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T05:21:47.122 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T05:21:47.468 INFO:teuthology.orchestra.run.smithi063.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T05:21:47.573 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T05:21:47.969 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T05:21:47.970 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T05:21:47.970 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T05:21:48.101 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T05:21:48.123 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T05:21:48.365 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T05:21:48.365 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T05:21:48.365 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T05:21:48.491 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T05:21:48.513 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T05:21:48.531 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 113 MB in 2s (71.2 MB/s) 2024-09-16T05:21:48.690 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T05:21:48.929 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 113 MB in 2s (72.5 MB/s) 2024-09-16T05:21:49.112 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T05:21:50.710 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T05:21:50.715 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T05:21:50.963 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:21:51.079 INFO:teuthology.orchestra.run.smithi063.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T05:21:51.083 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T05:21:51.309 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:21:53.048 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T05:21:53.069 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T05:21:53.159 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:21:53.751 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T05:21:53.773 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T05:21:53.801 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:21:53.836 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T05:21:53.857 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T05:21:53.939 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:21:54.364 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T05:21:54.385 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T05:21:54.414 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:00.294 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T05:22:00.421 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T05:22:00.729 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T05:22:00.773 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T05:22:00.833 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T05:22:00.940 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T05:22:01.119 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T05:22:01.142 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T05:22:01.186 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T05:22:01.307 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T05:22:01.452 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T05:22:01.730 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T05:22:01.741 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T05:22:01.813 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T05:22:06.716 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T05:22:06.753 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T05:22:06.791 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:07.460 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T05:22:07.497 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T05:22:07.543 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:09.547 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T05:22:09.669 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T05:22:10.012 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T05:22:10.141 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:10.267 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T05:22:10.395 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:11.007 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T05:22:11.146 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T05:22:11.522 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T05:22:11.650 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:11.746 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T05:22:11.746 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T05:22:11.746 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T05:22:11.746 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T05:22:11.776 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T05:22:11.906 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:11.938 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:13.413 INFO:teuthology.orchestra.run.smithi063.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T05:22:13.413 INFO:teuthology.orchestra.run.smithi063.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T05:22:13.413 INFO:teuthology.orchestra.run.smithi063.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T05:22:13.413 INFO:teuthology.orchestra.run.smithi063.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T05:22:13.824 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:13.932 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:15.258 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:15.359 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T05:22:15.493 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T05:22:15.620 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:15.689 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T05:22:15.689 INFO:teuthology.orchestra.run.smithi006.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T05:22:16.726 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T05:22:16.852 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T05:22:16.978 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T05:22:17.058 INFO:teuthology.orchestra.run.smithi063.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T05:22:17.058 INFO:teuthology.orchestra.run.smithi063.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T05:22:30.254 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T05:22:30.254 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub' 2024-09-16T05:22:30.261 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T05:22:30.288 INFO:teuthology.orchestra.run.smithi006.stdout:Generating grub configuration file ... 2024-09-16T05:22:30.624 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T05:22:30.645 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T05:22:31.032 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T05:22:31.039 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T05:22:31.106 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T05:22:31.112 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T05:22:31.275 INFO:teuthology.orchestra.run.smithi006.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T05:22:31.275 INFO:teuthology.orchestra.run.smithi006.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T05:22:31.275 INFO:teuthology.orchestra.run.smithi006.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T05:22:31.311 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-09-16T05:22:31.398 INFO:teuthology.orchestra.run.smithi063.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T05:22:31.398 INFO:teuthology.orchestra.run.smithi063.stdout:Sourcing file `/etc/default/grub' 2024-09-16T05:22:31.421 INFO:teuthology.orchestra.run.smithi063.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T05:22:31.440 INFO:teuthology.orchestra.run.smithi063.stdout:Generating grub configuration file ... 2024-09-16T05:22:31.758 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-09-16T05:22:31.788 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 21 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.15.0.121.121 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-09-16T05:22:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-09-16T05:22:31.790 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-09-16T05:22:31.790 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-09-16T05:22:31.790 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T05:22:31.790 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T05:22:31.791 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T05:22:31.791 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T05:22:31.791 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T05:22:31.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-09-16T05:22:31.819 INFO:teuthology.orchestra.run.smithi063.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T05:22:31.842 INFO:teuthology.orchestra.run.smithi063.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T05:22:31.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-16T05:22:32.105 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T05:22:32.105 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T05:22:32.116 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T05:22:32.133 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T05:22:32.203 INFO:teuthology.orchestra.run.smithi063.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T05:22:32.209 INFO:teuthology.orchestra.run.smithi063.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T05:22:32.278 INFO:teuthology.orchestra.run.smithi063.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T05:22:32.284 INFO:teuthology.orchestra.run.smithi063.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T05:22:32.452 INFO:teuthology.orchestra.run.smithi063.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T05:22:32.452 INFO:teuthology.orchestra.run.smithi063.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T05:22:32.452 INFO:teuthology.orchestra.run.smithi063.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T05:22:32.493 INFO:teuthology.orchestra.run.smithi063.stdout:done 2024-09-16T05:22:33.088 DEBUG:teuthology.orchestra.run.smithi063:> dpkg -s linux-image-generic 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Package: linux-image-generic 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Status: install ok installed 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Priority: optional 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Section: kernel 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Installed-Size: 21 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: amd64 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Source: linux-meta 2024-09-16T05:22:33.115 INFO:teuthology.orchestra.run.smithi063.stdout:Version: 5.15.0.121.121 2024-09-16T05:22:33.116 INFO:teuthology.orchestra.run.smithi063.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T05:22:33.116 INFO:teuthology.orchestra.run.smithi063.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T05:22:33.116 INFO:teuthology.orchestra.run.smithi063.stdout:Recommends: thermald 2024-09-16T05:22:33.116 INFO:teuthology.orchestra.run.smithi063.stdout:Description: Generic Linux kernel image 2024-09-16T05:22:33.116 INFO:teuthology.orchestra.run.smithi063.stdout: This package will always depend on the latest generic kernel image 2024-09-16T05:22:33.116 INFO:teuthology.orchestra.run.smithi063.stdout: available. 2024-09-16T05:22:33.116 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T05:22:33.116 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T05:22:33.116 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T05:22:33.116 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T05:22:33.116 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T05:22:33.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get clean 2024-09-16T05:22:33.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2024-09-16T05:22:33.417 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T05:22:33.538 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T05:22:33.555 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-16T05:22:33.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T05:22:33.634 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T05:22:33.640 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-16T05:22:33.701 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T05:22:33.849 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-16T05:22:33.849 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-16T05:22:34.037 INFO:teuthology.orchestra.run.smithi006.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T05:22:34.037 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T05:22:34.037 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T05:22:34.038 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T05:22:34.038 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T05:22:34.038 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T05:22:34.038 INFO:teuthology.orchestra.run.smithi006.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T05:22:34.038 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T05:22:34.078 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T05:22:34.080 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-09-16T05:22:34.092 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-09-16T05:22:34.092 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-09-16T05:22:34.092 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 21 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.15.0.121.121 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-09-16T05:22:34.093 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-09-16T05:22:34.094 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-16T05:22:34.137 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.DvtK5bQoCs 2024-09-16T05:22:34.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub/grub.cfg /tmp/tmp.DvtK5bQoCs 2024-09-16T05:22:34.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.DvtK5bQoCs 2024-09-16T05:22:34.323 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.DvtK5bQoCs is 10KB 2024-09-16T05:22:34.336 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.DvtK5bQoCs 2024-09-16T05:22:34.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T05:22:34.398 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:22:34.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T05:22:34.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T05:22:34.468 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T05:22:34.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo update-grub 2024-09-16T05:22:34.880 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-09-16T05:22:34.898 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T05:22:34.960 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-09-16T05:22:35.176 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2024-09-16T05:22:35.177 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2024-09-16T05:22:35.380 INFO:teuthology.orchestra.run.smithi063.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T05:22:35.380 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T05:22:35.380 INFO:teuthology.orchestra.run.smithi063.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T05:22:35.380 INFO:teuthology.orchestra.run.smithi063.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T05:22:35.381 INFO:teuthology.orchestra.run.smithi063.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T05:22:35.381 INFO:teuthology.orchestra.run.smithi063.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T05:22:35.381 INFO:teuthology.orchestra.run.smithi063.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T05:22:35.381 INFO:teuthology.orchestra.run.smithi063.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T05:22:35.427 INFO:teuthology.orchestra.run.smithi063.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T05:22:35.429 DEBUG:teuthology.orchestra.run.smithi063:> dpkg -s linux-image-generic 2024-09-16T05:22:35.442 INFO:teuthology.orchestra.run.smithi063.stdout:Package: linux-image-generic 2024-09-16T05:22:35.442 INFO:teuthology.orchestra.run.smithi063.stdout:Status: install ok installed 2024-09-16T05:22:35.442 INFO:teuthology.orchestra.run.smithi063.stdout:Priority: optional 2024-09-16T05:22:35.442 INFO:teuthology.orchestra.run.smithi063.stdout:Section: kernel 2024-09-16T05:22:35.442 INFO:teuthology.orchestra.run.smithi063.stdout:Installed-Size: 21 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: amd64 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout:Source: linux-meta 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout:Version: 5.15.0.121.121 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout:Recommends: thermald 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout:Description: Generic Linux kernel image 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout: This package will always depend on the latest generic kernel image 2024-09-16T05:22:35.443 INFO:teuthology.orchestra.run.smithi063.stdout: available. 2024-09-16T05:22:35.443 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-09-16T05:22:35.489 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.rZ69XHhAPx 2024-09-16T05:22:35.490 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub/grub.cfg /tmp/tmp.rZ69XHhAPx 2024-09-16T05:22:35.544 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.rZ69XHhAPx 2024-09-16T05:22:35.674 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.rZ69XHhAPx is 10KB 2024-09-16T05:22:35.687 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.rZ69XHhAPx 2024-09-16T05:22:35.693 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T05:22:35.717 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub' 2024-09-16T05:22:35.719 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T05:22:35.721 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-09-16T05:22:35.749 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:22:35.750 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T05:22:35.750 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T05:22:35.834 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T05:22:35.834 DEBUG:teuthology.orchestra.run.smithi063:> sudo update-grub 2024-09-16T05:22:35.950 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T05:22:35.957 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T05:22:36.289 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T05:22:36.295 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T05:22:36.337 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T05:22:36.343 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T05:22:36.474 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T05:22:36.474 INFO:teuthology.orchestra.run.smithi006.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T05:22:36.475 INFO:teuthology.orchestra.run.smithi006.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T05:22:36.494 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-09-16T05:22:36.503 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-09-16T05:22:37.101 INFO:teuthology.orchestra.run.smithi063.stderr:Sourcing file `/etc/default/grub' 2024-09-16T05:22:37.102 INFO:teuthology.orchestra.run.smithi063.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T05:22:37.106 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-09-16T05:22:37.282 INFO:teuthology.orchestra.run.smithi063.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T05:22:37.288 INFO:teuthology.orchestra.run.smithi063.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T05:22:37.635 INFO:teuthology.orchestra.run.smithi063.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T05:22:37.641 INFO:teuthology.orchestra.run.smithi063.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T05:22:37.688 INFO:teuthology.orchestra.run.smithi063.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T05:22:37.693 INFO:teuthology.orchestra.run.smithi063.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T05:22:37.844 INFO:teuthology.orchestra.run.smithi063.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T05:22:37.844 INFO:teuthology.orchestra.run.smithi063.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T05:22:37.844 INFO:teuthology.orchestra.run.smithi063.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T05:22:37.881 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-09-16T05:22:37.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-09-16T05:23:06.533 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T05:23:06.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-16T05:23:06.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:23:07.894 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T05:23:07.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T05:23:07.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:23:24.861 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-09-16T05:23:26.401 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-09-16T05:23:33.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-16T05:23:33.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:23:35.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T05:23:35.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:23:36.925 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-09-16T05:23:38.461 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-09-16T05:23:48.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-16T05:23:48.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:23:50.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T05:23:50.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:24:08.897 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-09-16T05:24:20.809 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-16T05:24:21.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-16T05:24:21.582 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T05:24:21.582 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-16T05:24:21.630 INFO:teuthology.orchestra.run.smithi006.stdout:5.15.0-121-generic 2024-09-16T05:24:21.630 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T05:24:21.630 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T05:24:21.630 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T05:24:22.631 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T05:24:22.632 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T05:24:22.723 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-09-16T05:24:22.755 DEBUG:teuthology.parallel:result is None 2024-09-16T05:24:23.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T05:24:23.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:24:24.363 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-09-16T05:24:25.498 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T05:24:25.498 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T05:24:25.498 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-09-16T05:24:25.503 INFO:teuthology.orchestra.run.smithi063.stdout:5.15.0-121-generic 2024-09-16T05:24:25.504 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T05:24:25.504 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T05:24:25.504 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T05:24:26.505 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T05:24:26.505 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T05:24:26.688 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-09-16T05:24:26.718 DEBUG:teuthology.parallel:result is None 2024-09-16T05:24:26.718 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T05:24:26.727 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T05:24:26.727 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T05:24:26.730 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T05:24:26.736 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T05:24:26.742 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T05:24:26.748 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T05:24:26.749 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T05:24:26.779 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T05:24:26.813 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T05:24:26.819 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T05:24:26.819 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T05:24:26.847 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T05:24:26.868 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:26.877 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:26.878 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:26.885 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:26.887 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T05:24:26.897 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T05:24:26.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T05:24:26.928 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T05:24:26.943 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T05:24:26.952 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T05:24:26.953 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T05:24:26.983 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T05:24:26.991 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T05:24:27.034 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T05:24:27.082 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:24:27.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T05:24:27.143 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T05:24:27.149 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T05:24:27.200 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:24:27.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T05:24:27.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-09-16T05:24:27.310 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-09-16T05:24:27.397 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T05:24:27.404 INFO:teuthology.task.internal:Starting timer... 2024-09-16T05:24:27.404 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T05:24:27.415 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T05:24:27.423 DEBUG:teuthology.task.selinux:Excluding smithi006: OS 'ubuntu' does not support SELinux 2024-09-16T05:24:27.423 DEBUG:teuthology.task.selinux:Excluding smithi063: OS 'ubuntu' does not support SELinux 2024-09-16T05:24:27.423 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T05:24:27.424 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T05:24:27.424 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T05:24:27.424 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T05:24:27.432 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T05:24:27.435 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T05:24:27.552 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T05:24:27.563 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T05:24:27.565 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi063.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T05:32:09.973 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi063.front.sepia.ceph.com')] 2024-09-16T05:32:09.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-16T05:32:09.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:32:10.066 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-16T05:32:10.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-16T05:32:10.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T05:32:10.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:32:10.227 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-09-16T05:32:10.305 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-16T05:32:10.305 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T05:32:10.316 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T05:32:10.316 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T05:32:10.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:32:10.319 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T05:32:10.319 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:32:10.347 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T05:32:10.347 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Command line: ntpd -gq 2024-09-16T05:32:10.347 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: ---------------------------------------------------- 2024-09-16T05:32:10.347 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T05:32:10.347 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T05:32:10.347 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: corporation. Support and training for ntp-4 are 2024-09-16T05:32:10.347 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: available at https://www.nwtime.org/support 2024-09-16T05:32:10.347 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: ---------------------------------------------------- 2024-09-16T05:32:10.348 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: proto: precision = 0.048 usec (-24) 2024-09-16T05:32:10.348 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: basedate set to 2022-02-04 2024-09-16T05:32:10.348 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: gps base set to 2022-02-06 (week 2196) 2024-09-16T05:32:10.348 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T05:32:10.349 INFO:teuthology.orchestra.run.smithi006.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T05:32:10.349 INFO:teuthology.orchestra.run.smithi006.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T05:32:10.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T05:32:10.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T05:32:10.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T05:32:10.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T05:32:10.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Listen normally on 3 ens1f0 172.21.15.6:123 2024-09-16T05:32:10.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Listen normally on 4 lo [::1]:123 2024-09-16T05:32:10.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e4%3]:123 2024-09-16T05:32:10.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:10 ntpd[18385]: Listening on routing socket on fd #22 for interface updates 2024-09-16T05:32:10.380 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T05:32:10.380 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Command line: ntpd -gq 2024-09-16T05:32:10.380 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: ---------------------------------------------------- 2024-09-16T05:32:10.380 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T05:32:10.380 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T05:32:10.380 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: corporation. Support and training for ntp-4 are 2024-09-16T05:32:10.380 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: available at https://www.nwtime.org/support 2024-09-16T05:32:10.380 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: ---------------------------------------------------- 2024-09-16T05:32:10.381 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: proto: precision = 0.047 usec (-24) 2024-09-16T05:32:10.381 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: basedate set to 2022-02-04 2024-09-16T05:32:10.381 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: gps base set to 2022-02-06 (week 2196) 2024-09-16T05:32:10.382 INFO:teuthology.orchestra.run.smithi063.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T05:32:10.382 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T05:32:10.382 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T05:32:10.382 INFO:teuthology.orchestra.run.smithi063.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T05:32:10.383 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T05:32:10.383 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T05:32:10.383 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T05:32:10.383 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Listen normally on 3 enp3s0f1 172.21.15.63:123 2024-09-16T05:32:10.383 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Listen normally on 4 lo [::1]:123 2024-09-16T05:32:10.384 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8d7%5]:123 2024-09-16T05:32:10.384 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:10 ntpd[18463]: Listening on routing socket on fd #22 for interface updates 2024-09-16T05:32:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:11.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:11 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:13.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:13 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:13 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:13 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:13 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:13 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:13 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:13 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:13.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:13 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:13.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:13.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:13.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:13.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:13.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:13 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:15 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:15.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:15.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:15.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:15.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:15.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:15.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:15 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:17 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:17.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.381 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:17.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:17.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:17.382 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:17.382 INFO:teuthology.orchestra.run.smithi063.stdout:16 Sep 05:32:17 ntpd[18463]: ntpd: time slew +0.002244 s 2024-09-16T05:32:17.382 INFO:teuthology.orchestra.run.smithi063.stdout:ntpd: time slew +0.002244s 2024-09-16T05:32:17.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T05:32:17.383 INFO:teuthology.orchestra.run.smithi063.stderr:16 Sep 05:32:17 ntpd[18463]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T05:32:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T05:32:17.433 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================== 2024-09-16T05:32:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:32:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:32:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:32:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:32:19.348 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T05:32:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T05:32:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T05:32:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T05:32:19.350 INFO:teuthology.orchestra.run.smithi006.stdout:16 Sep 05:32:19 ntpd[18385]: ntpd: time slew -0.000352 s 2024-09-16T05:32:19.350 INFO:teuthology.orchestra.run.smithi006.stdout:ntpd: time slew -0.000352s 2024-09-16T05:32:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T05:32:19.350 INFO:teuthology.orchestra.run.smithi006.stderr:16 Sep 05:32:19 ntpd[18385]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T05:32:19.404 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T05:32:19.404 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-09-16T05:32:19.404 INFO:teuthology.orchestra.run.smithi006.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:32:19.404 INFO:teuthology.orchestra.run.smithi006.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:32:19.404 INFO:teuthology.orchestra.run.smithi006.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:32:19.405 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T05:32:19.406 INFO:teuthology.run_tasks:Running task install... 2024-09-16T05:32:19.418 DEBUG:teuthology.task.install:project ceph 2024-09-16T05:32:19.418 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T05:32:19.418 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T05:32:19.418 INFO:teuthology.task.install:Using flavor: default 2024-09-16T05:32:19.424 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T05:32:19.424 INFO:teuthology.task.install:extra packages: [] 2024-09-16T05:32:19.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-key list | grep Ceph 2024-09-16T05:32:19.426 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-key list | grep Ceph 2024-09-16T05:32:19.492 INFO:teuthology.orchestra.run.smithi063.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T05:32:19.507 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T05:32:19.525 INFO:teuthology.orchestra.run.smithi063.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T05:32:19.525 INFO:teuthology.orchestra.run.smithi063.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T05:32:19.526 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T05:32:19.526 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:32:19.538 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T05:32:19.538 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T05:32:19.539 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T05:32:19.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:32:19.677 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T05:32:19.677 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:32:19.681 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T05:32:19.681 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:32:19.789 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:32:19.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T05:32:19.795 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:32:19.795 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T05:32:19.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-16T05:32:19.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2024-09-16T05:32:19.976 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T05:32:19.980 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T05:32:19.985 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T05:32:20.002 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T05:32:20.031 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T05:32:20.034 INFO:teuthology.orchestra.run.smithi063.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T05:32:20.055 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T05:32:20.055 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T05:32:20.076 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T05:32:20.076 INFO:teuthology.orchestra.run.smithi063.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T05:32:20.097 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T05:32:20.097 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T05:32:20.103 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T05:32:20.105 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T05:32:20.125 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T05:32:20.125 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T05:32:20.182 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T05:32:20.264 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T05:32:20.334 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 29.0 kB in 0s (84.0 kB/s) 2024-09-16T05:32:20.416 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 29.0 kB in 0s (69.2 kB/s) 2024-09-16T05:32:21.393 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-16T05:32:21.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:32:21.464 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-16T05:32:21.492 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-09-16T05:32:21.509 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:32:21.573 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-09-16T05:32:21.654 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-16T05:32:21.655 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-16T05:32:21.785 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2024-09-16T05:32:21.785 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2024-09-16T05:32:21.848 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T05:32:21.848 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T05:32:21.848 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T05:32:21.848 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T05:32:21.849 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T05:32:21.849 INFO:teuthology.orchestra.run.smithi006.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T05:32:21.849 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T05:32:21.850 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-09-16T05:32:21.850 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T05:32:21.850 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T05:32:21.850 INFO:teuthology.orchestra.run.smithi006.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T05:32:21.850 INFO:teuthology.orchestra.run.smithi006.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T05:32:21.851 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T05:32:21.851 INFO:teuthology.orchestra.run.smithi006.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T05:32:21.851 INFO:teuthology.orchestra.run.smithi006.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T05:32:21.852 INFO:teuthology.orchestra.run.smithi006.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet zip 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T05:32:21.853 INFO:teuthology.orchestra.run.smithi006.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T05:32:21.854 INFO:teuthology.orchestra.run.smithi006.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T05:32:21.854 INFO:teuthology.orchestra.run.smithi006.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T05:32:21.854 INFO:teuthology.orchestra.run.smithi006.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T05:32:21.854 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-09-16T05:32:21.854 INFO:teuthology.orchestra.run.smithi006.stdout: btrfs-tools 2024-09-16T05:32:21.960 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-09-16T05:32:21.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T05:32:21.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T05:32:21.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T05:32:21.960 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T05:32:21.960 INFO:teuthology.orchestra.run.smithi006.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T05:32:21.960 INFO:teuthology.orchestra.run.smithi006.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T05:32:21.961 INFO:teuthology.orchestra.run.smithi006.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T05:32:21.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T05:32:21.963 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T05:32:21.963 INFO:teuthology.orchestra.run.smithi006.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T05:32:21.963 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T05:32:21.963 INFO:teuthology.orchestra.run.smithi006.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T05:32:21.963 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T05:32:21.963 INFO:teuthology.orchestra.run.smithi006.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T05:32:22.003 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T05:32:22.003 INFO:teuthology.orchestra.run.smithi063.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T05:32:22.004 INFO:teuthology.orchestra.run.smithi063.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T05:32:22.004 INFO:teuthology.orchestra.run.smithi063.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T05:32:22.004 INFO:teuthology.orchestra.run.smithi063.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T05:32:22.004 INFO:teuthology.orchestra.run.smithi063.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T05:32:22.004 INFO:teuthology.orchestra.run.smithi063.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T05:32:22.005 INFO:teuthology.orchestra.run.smithi063.stdout:The following additional packages will be installed: 2024-09-16T05:32:22.005 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T05:32:22.005 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T05:32:22.006 INFO:teuthology.orchestra.run.smithi063.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T05:32:22.006 INFO:teuthology.orchestra.run.smithi063.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T05:32:22.007 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T05:32:22.008 INFO:teuthology.orchestra.run.smithi063.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T05:32:22.008 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T05:32:22.008 INFO:teuthology.orchestra.run.smithi063.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T05:32:22.008 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T05:32:22.008 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T05:32:22.008 INFO:teuthology.orchestra.run.smithi063.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T05:32:22.008 INFO:teuthology.orchestra.run.smithi063.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T05:32:22.009 INFO:teuthology.orchestra.run.smithi063.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T05:32:22.009 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T05:32:22.009 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T05:32:22.009 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet zip 2024-09-16T05:32:22.010 INFO:teuthology.orchestra.run.smithi063.stdout:Suggested packages: 2024-09-16T05:32:22.010 INFO:teuthology.orchestra.run.smithi063.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T05:32:22.010 INFO:teuthology.orchestra.run.smithi063.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T05:32:22.010 INFO:teuthology.orchestra.run.smithi063.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T05:32:22.010 INFO:teuthology.orchestra.run.smithi063.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T05:32:22.010 INFO:teuthology.orchestra.run.smithi063.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T05:32:22.010 INFO:teuthology.orchestra.run.smithi063.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T05:32:22.010 INFO:teuthology.orchestra.run.smithi063.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T05:32:22.011 INFO:teuthology.orchestra.run.smithi063.stdout:Recommended packages: 2024-09-16T05:32:22.011 INFO:teuthology.orchestra.run.smithi063.stdout: btrfs-tools 2024-09-16T05:32:22.105 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T05:32:22.105 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 181 MB of archives. 2024-09-16T05:32:22.105 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T05:32:22.106 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T05:32:22.117 INFO:teuthology.orchestra.run.smithi063.stdout:The following NEW packages will be installed: 2024-09-16T05:32:22.117 INFO:teuthology.orchestra.run.smithi063.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T05:32:22.117 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T05:32:22.117 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T05:32:22.118 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T05:32:22.118 INFO:teuthology.orchestra.run.smithi063.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T05:32:22.118 INFO:teuthology.orchestra.run.smithi063.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T05:32:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T05:32:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T05:32:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T05:32:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T05:32:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T05:32:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T05:32:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T05:32:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T05:32:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T05:32:22.150 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T05:32:22.263 INFO:teuthology.orchestra.run.smithi063.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T05:32:22.264 INFO:teuthology.orchestra.run.smithi063.stdout:Need to get 181 MB of archives. 2024-09-16T05:32:22.264 INFO:teuthology.orchestra.run.smithi063.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T05:32:22.264 INFO:teuthology.orchestra.run.smithi063.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T05:32:22.291 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T05:32:22.308 INFO:teuthology.orchestra.run.smithi063.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T05:32:22.339 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T05:32:22.351 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T05:32:22.356 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T05:32:22.357 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T05:32:22.360 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T05:32:22.361 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T05:32:22.366 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T05:32:22.491 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T05:32:22.493 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T05:32:22.499 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T05:32:22.574 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T05:32:22.586 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T05:32:22.667 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T05:32:22.763 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T05:32:22.867 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T05:32:22.872 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T05:32:22.897 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T05:32:22.907 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T05:32:22.937 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T05:32:22.945 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T05:32:22.946 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T05:32:22.948 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T05:32:23.014 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T05:32:23.075 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T05:32:23.081 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T05:32:23.082 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T05:32:23.086 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T05:32:23.089 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T05:32:23.090 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T05:32:23.094 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T05:32:23.099 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T05:32:23.102 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T05:32:23.103 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T05:32:23.104 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T05:32:23.116 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T05:32:23.143 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T05:32:23.150 INFO:teuthology.orchestra.run.smithi063.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T05:32:23.151 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T05:32:23.154 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T05:32:23.155 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T05:32:23.159 INFO:teuthology.orchestra.run.smithi063.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T05:32:23.160 INFO:teuthology.orchestra.run.smithi063.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T05:32:23.162 INFO:teuthology.orchestra.run.smithi063.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T05:32:23.170 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T05:32:23.180 INFO:teuthology.orchestra.run.smithi063.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T05:32:23.184 INFO:teuthology.orchestra.run.smithi063.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T05:32:23.187 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T05:32:23.187 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T05:32:23.188 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T05:32:23.188 INFO:teuthology.orchestra.run.smithi063.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T05:32:23.189 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T05:32:23.190 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T05:32:23.190 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T05:32:23.191 INFO:teuthology.orchestra.run.smithi063.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T05:32:23.191 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T05:32:23.196 INFO:teuthology.orchestra.run.smithi006.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T05:32:23.225 INFO:teuthology.orchestra.run.smithi006.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T05:32:23.267 INFO:teuthology.orchestra.run.smithi063.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T05:32:23.272 INFO:teuthology.orchestra.run.smithi063.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T05:32:23.276 INFO:teuthology.orchestra.run.smithi063.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T05:32:23.279 INFO:teuthology.orchestra.run.smithi063.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T05:32:23.280 INFO:teuthology.orchestra.run.smithi063.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T05:32:23.284 INFO:teuthology.orchestra.run.smithi006.stdout:Get:47 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T05:32:23.285 INFO:teuthology.orchestra.run.smithi063.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T05:32:23.306 INFO:teuthology.orchestra.run.smithi006.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T05:32:23.307 INFO:teuthology.orchestra.run.smithi006.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T05:32:23.309 INFO:teuthology.orchestra.run.smithi006.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T05:32:23.309 INFO:teuthology.orchestra.run.smithi006.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T05:32:23.310 INFO:teuthology.orchestra.run.smithi006.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T05:32:23.313 INFO:teuthology.orchestra.run.smithi006.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T05:32:23.314 INFO:teuthology.orchestra.run.smithi006.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T05:32:23.315 INFO:teuthology.orchestra.run.smithi006.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T05:32:23.350 INFO:teuthology.orchestra.run.smithi063.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T05:32:23.350 INFO:teuthology.orchestra.run.smithi063.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T05:32:23.351 INFO:teuthology.orchestra.run.smithi063.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T05:32:23.351 INFO:teuthology.orchestra.run.smithi063.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T05:32:23.388 INFO:teuthology.orchestra.run.smithi006.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T05:32:23.388 INFO:teuthology.orchestra.run.smithi006.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T05:32:23.393 INFO:teuthology.orchestra.run.smithi063.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T05:32:23.408 INFO:teuthology.orchestra.run.smithi006.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T05:32:23.409 INFO:teuthology.orchestra.run.smithi006.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T05:32:23.411 INFO:teuthology.orchestra.run.smithi006.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T05:32:23.415 INFO:teuthology.orchestra.run.smithi006.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T05:32:23.431 INFO:teuthology.orchestra.run.smithi063.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T05:32:23.432 INFO:teuthology.orchestra.run.smithi063.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T05:32:23.432 INFO:teuthology.orchestra.run.smithi063.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T05:32:23.437 INFO:teuthology.orchestra.run.smithi063.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T05:32:23.439 INFO:teuthology.orchestra.run.smithi063.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T05:32:23.439 INFO:teuthology.orchestra.run.smithi063.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T05:32:23.471 INFO:teuthology.orchestra.run.smithi006.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T05:32:23.474 INFO:teuthology.orchestra.run.smithi006.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T05:32:23.475 INFO:teuthology.orchestra.run.smithi006.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T05:32:23.480 INFO:teuthology.orchestra.run.smithi006.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T05:32:23.504 INFO:teuthology.orchestra.run.smithi063.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T05:32:23.513 INFO:teuthology.orchestra.run.smithi063.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T05:32:23.514 INFO:teuthology.orchestra.run.smithi063.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T05:32:23.515 INFO:teuthology.orchestra.run.smithi063.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T05:32:23.516 INFO:teuthology.orchestra.run.smithi063.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T05:32:23.552 INFO:teuthology.orchestra.run.smithi006.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T05:32:23.554 INFO:teuthology.orchestra.run.smithi006.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T05:32:23.558 INFO:teuthology.orchestra.run.smithi006.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T05:32:23.560 INFO:teuthology.orchestra.run.smithi063.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T05:32:23.562 INFO:teuthology.orchestra.run.smithi063.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T05:32:23.565 INFO:teuthology.orchestra.run.smithi063.stdout:Get:41 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T05:32:23.567 INFO:teuthology.orchestra.run.smithi063.stdout:Get:42 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T05:32:23.594 INFO:teuthology.orchestra.run.smithi063.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T05:32:23.595 INFO:teuthology.orchestra.run.smithi063.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T05:32:23.596 INFO:teuthology.orchestra.run.smithi063.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T05:32:23.607 INFO:teuthology.orchestra.run.smithi063.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T05:32:23.607 INFO:teuthology.orchestra.run.smithi063.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T05:32:23.609 INFO:teuthology.orchestra.run.smithi063.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T05:32:23.621 INFO:teuthology.orchestra.run.smithi063.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T05:32:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T05:32:23.678 INFO:teuthology.orchestra.run.smithi063.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T05:32:23.683 INFO:teuthology.orchestra.run.smithi063.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T05:32:23.684 INFO:teuthology.orchestra.run.smithi063.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T05:32:23.757 INFO:teuthology.orchestra.run.smithi063.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T05:32:23.759 INFO:teuthology.orchestra.run.smithi063.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T05:32:23.763 INFO:teuthology.orchestra.run.smithi063.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T05:32:23.764 INFO:teuthology.orchestra.run.smithi063.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T05:32:23.766 INFO:teuthology.orchestra.run.smithi063.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T05:32:23.770 INFO:teuthology.orchestra.run.smithi063.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T05:32:23.798 INFO:teuthology.orchestra.run.smithi006.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T05:32:23.851 INFO:teuthology.orchestra.run.smithi006.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T05:32:23.852 INFO:teuthology.orchestra.run.smithi006.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T05:32:23.852 INFO:teuthology.orchestra.run.smithi006.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T05:32:23.854 INFO:teuthology.orchestra.run.smithi006.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T05:32:23.855 INFO:teuthology.orchestra.run.smithi006.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T05:32:23.858 INFO:teuthology.orchestra.run.smithi006.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T05:32:23.866 INFO:teuthology.orchestra.run.smithi006.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T05:32:23.903 INFO:teuthology.orchestra.run.smithi006.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T05:32:23.905 INFO:teuthology.orchestra.run.smithi006.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T05:32:23.907 INFO:teuthology.orchestra.run.smithi006.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T05:32:23.907 INFO:teuthology.orchestra.run.smithi006.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T05:32:23.915 INFO:teuthology.orchestra.run.smithi006.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T05:32:23.917 INFO:teuthology.orchestra.run.smithi006.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T05:32:23.935 INFO:teuthology.orchestra.run.smithi063.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T05:32:23.937 INFO:teuthology.orchestra.run.smithi063.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T05:32:23.940 INFO:teuthology.orchestra.run.smithi063.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T05:32:23.940 INFO:teuthology.orchestra.run.smithi006.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T05:32:23.941 INFO:teuthology.orchestra.run.smithi063.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T05:32:23.943 INFO:teuthology.orchestra.run.smithi063.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T05:32:23.957 INFO:teuthology.orchestra.run.smithi006.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T05:32:23.957 INFO:teuthology.orchestra.run.smithi006.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T05:32:24.000 INFO:teuthology.orchestra.run.smithi063.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T05:32:24.000 INFO:teuthology.orchestra.run.smithi063.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T05:32:24.002 INFO:teuthology.orchestra.run.smithi063.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T05:32:24.008 INFO:teuthology.orchestra.run.smithi063.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T05:32:24.009 INFO:teuthology.orchestra.run.smithi063.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T05:32:24.012 INFO:teuthology.orchestra.run.smithi063.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T05:32:24.017 INFO:teuthology.orchestra.run.smithi006.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T05:32:24.018 INFO:teuthology.orchestra.run.smithi006.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T05:32:24.019 INFO:teuthology.orchestra.run.smithi006.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T05:32:24.021 INFO:teuthology.orchestra.run.smithi006.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T05:32:24.021 INFO:teuthology.orchestra.run.smithi006.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T05:32:24.022 INFO:teuthology.orchestra.run.smithi006.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T05:32:24.030 INFO:teuthology.orchestra.run.smithi006.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T05:32:24.042 INFO:teuthology.orchestra.run.smithi063.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T05:32:24.042 INFO:teuthology.orchestra.run.smithi063.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T05:32:24.099 INFO:teuthology.orchestra.run.smithi006.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T05:32:24.104 INFO:teuthology.orchestra.run.smithi006.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T05:32:24.105 INFO:teuthology.orchestra.run.smithi006.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T05:32:24.105 INFO:teuthology.orchestra.run.smithi063.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T05:32:24.105 INFO:teuthology.orchestra.run.smithi063.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T05:32:24.105 INFO:teuthology.orchestra.run.smithi063.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T05:32:24.106 INFO:teuthology.orchestra.run.smithi063.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T05:32:24.106 INFO:teuthology.orchestra.run.smithi063.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T05:32:24.108 INFO:teuthology.orchestra.run.smithi063.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T05:32:24.110 INFO:teuthology.orchestra.run.smithi006.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T05:32:24.110 INFO:teuthology.orchestra.run.smithi006.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T05:32:24.115 INFO:teuthology.orchestra.run.smithi063.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T05:32:24.180 INFO:teuthology.orchestra.run.smithi006.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T05:32:24.184 INFO:teuthology.orchestra.run.smithi063.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T05:32:24.186 INFO:teuthology.orchestra.run.smithi006.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T05:32:24.188 INFO:teuthology.orchestra.run.smithi063.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T05:32:24.189 INFO:teuthology.orchestra.run.smithi006.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T05:32:24.189 INFO:teuthology.orchestra.run.smithi063.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T05:32:24.190 INFO:teuthology.orchestra.run.smithi006.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T05:32:24.201 INFO:teuthology.orchestra.run.smithi006.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T05:32:24.202 INFO:teuthology.orchestra.run.smithi006.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T05:32:24.210 INFO:teuthology.orchestra.run.smithi006.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T05:32:24.262 INFO:teuthology.orchestra.run.smithi006.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T05:32:24.263 INFO:teuthology.orchestra.run.smithi006.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T05:32:24.264 INFO:teuthology.orchestra.run.smithi006.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T05:32:24.266 INFO:teuthology.orchestra.run.smithi063.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T05:32:24.272 INFO:teuthology.orchestra.run.smithi063.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T05:32:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T05:32:24.276 INFO:teuthology.orchestra.run.smithi063.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T05:32:24.287 INFO:teuthology.orchestra.run.smithi063.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T05:32:24.288 INFO:teuthology.orchestra.run.smithi063.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T05:32:24.296 INFO:teuthology.orchestra.run.smithi063.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T05:32:24.343 INFO:teuthology.orchestra.run.smithi006.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T05:32:24.345 INFO:teuthology.orchestra.run.smithi006.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T05:32:24.348 INFO:teuthology.orchestra.run.smithi063.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T05:32:24.349 INFO:teuthology.orchestra.run.smithi063.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T05:32:24.350 INFO:teuthology.orchestra.run.smithi063.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T05:32:24.362 INFO:teuthology.orchestra.run.smithi006.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T05:32:24.363 INFO:teuthology.orchestra.run.smithi006.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T05:32:24.364 INFO:teuthology.orchestra.run.smithi006.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T05:32:24.369 INFO:teuthology.orchestra.run.smithi006.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T05:32:24.371 INFO:teuthology.orchestra.run.smithi006.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T05:32:24.429 INFO:teuthology.orchestra.run.smithi063.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T05:32:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T05:32:24.448 INFO:teuthology.orchestra.run.smithi063.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T05:32:24.449 INFO:teuthology.orchestra.run.smithi063.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T05:32:24.449 INFO:teuthology.orchestra.run.smithi063.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T05:32:24.455 INFO:teuthology.orchestra.run.smithi063.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T05:32:24.457 INFO:teuthology.orchestra.run.smithi063.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T05:32:25.400 INFO:teuthology.orchestra.run.smithi063.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T05:32:25.482 INFO:teuthology.orchestra.run.smithi063.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T05:32:25.501 INFO:teuthology.orchestra.run.smithi063.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T05:32:25.503 INFO:teuthology.orchestra.run.smithi063.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T05:32:25.535 INFO:teuthology.orchestra.run.smithi063.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T05:32:25.634 INFO:teuthology.orchestra.run.smithi006.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T05:32:25.636 INFO:teuthology.orchestra.run.smithi006.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T05:32:25.638 INFO:teuthology.orchestra.run.smithi006.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T05:32:25.642 INFO:teuthology.orchestra.run.smithi063.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T05:32:25.958 INFO:teuthology.orchestra.run.smithi006.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T05:32:26.247 INFO:teuthology.orchestra.run.smithi063.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T05:32:26.248 INFO:teuthology.orchestra.run.smithi063.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T05:32:26.259 INFO:teuthology.orchestra.run.smithi063.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T05:32:26.290 INFO:teuthology.orchestra.run.smithi063.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T05:32:26.299 INFO:teuthology.orchestra.run.smithi063.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T05:32:26.301 INFO:teuthology.orchestra.run.smithi063.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T05:32:26.322 INFO:teuthology.orchestra.run.smithi063.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T05:32:26.461 INFO:teuthology.orchestra.run.smithi063.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T05:32:26.462 INFO:teuthology.orchestra.run.smithi063.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T05:32:26.552 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 181 MB in 4s (45.3 MB/s) 2024-09-16T05:32:26.718 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T05:32:26.821 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T05:32:26.827 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T05:32:26.900 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T05:32:27.221 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T05:32:27.247 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T05:32:27.287 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T05:32:27.571 INFO:teuthology.orchestra.run.smithi063.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T05:32:27.578 INFO:teuthology.orchestra.run.smithi063.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T05:32:27.580 INFO:teuthology.orchestra.run.smithi063.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T05:32:27.640 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T05:32:27.667 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T05:32:27.706 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T05:32:27.772 INFO:teuthology.orchestra.run.smithi063.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T05:32:28.085 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T05:32:28.112 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:32:28.160 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:32:28.348 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 181 MB in 6s (32.1 MB/s) 2024-09-16T05:32:28.536 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T05:32:28.639 INFO:teuthology.orchestra.run.smithi063.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T05:32:28.644 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T05:32:28.680 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T05:32:28.706 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:32:28.726 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T05:32:28.746 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:32:29.041 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T05:32:29.063 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T05:32:29.067 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:32:29.090 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T05:32:29.107 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:32:29.121 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T05:32:29.449 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T05:32:29.469 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T05:32:29.476 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T05:32:29.496 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T05:32:29.516 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T05:32:29.535 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T05:32:29.969 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T05:32:29.997 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:32:30.039 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librados2. 2024-09-16T05:32:30.044 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:32:30.066 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:30.105 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:30.422 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T05:32:30.450 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:32:30.480 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:32:30.533 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librbd1. 2024-09-16T05:32:30.559 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:30.599 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:30.825 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T05:32:30.852 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T05:32:30.891 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:32:31.044 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T05:32:31.070 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:31.111 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:31.261 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T05:32:31.288 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T05:32:31.328 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T05:32:31.414 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rados. 2024-09-16T05:32:31.440 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:31.488 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:31.672 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librados2. 2024-09-16T05:32:31.699 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:31.738 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:31.774 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T05:32:31.789 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:32:31.823 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:32.043 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T05:32:32.069 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:32.109 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:32.166 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librbd1. 2024-09-16T05:32:32.182 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:32.216 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:32.437 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T05:32:32.464 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:32:32.504 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:32.644 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T05:32:32.670 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:32.710 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:32.823 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T05:32:32.850 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T05:32:32.890 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T05:32:33.113 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rados. 2024-09-16T05:32:33.140 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:33.179 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:33.209 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T05:32:33.236 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T05:32:33.276 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T05:32:33.482 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T05:32:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:32:33.540 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:33.712 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T05:32:33.739 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:33.759 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T05:32:33.775 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:33.778 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:33.817 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:34.107 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T05:32:34.133 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T05:32:34.145 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T05:32:34.172 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:32:34.173 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T05:32:34.212 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:34.442 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T05:32:34.469 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T05:32:34.509 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T05:32:34.539 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T05:32:34.567 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T05:32:34.606 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T05:32:34.812 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T05:32:34.828 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T05:32:34.870 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T05:32:34.892 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T05:32:34.919 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T05:32:34.958 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T05:32:35.194 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T05:32:35.218 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:35.261 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:35.315 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua5.1. 2024-09-16T05:32:35.342 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T05:32:35.381 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T05:32:35.605 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T05:32:35.633 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T05:32:35.672 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T05:32:35.684 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-any. 2024-09-16T05:32:35.711 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T05:32:35.751 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T05:32:35.958 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T05:32:35.973 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T05:32:36.012 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package zip. 2024-09-16T05:32:36.016 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T05:32:36.039 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T05:32:36.079 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T05:32:36.327 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T05:32:36.343 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T05:32:36.385 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T05:32:36.390 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package unzip. 2024-09-16T05:32:36.417 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T05:32:36.456 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T05:32:36.705 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package lua5.1. 2024-09-16T05:32:36.732 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T05:32:36.772 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T05:32:36.810 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package luarocks. 2024-09-16T05:32:36.837 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T05:32:36.868 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T05:32:37.074 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package lua-any. 2024-09-16T05:32:37.090 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T05:32:37.133 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T05:32:37.228 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librgw2. 2024-09-16T05:32:37.253 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:37.292 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:37.377 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package zip. 2024-09-16T05:32:37.404 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T05:32:37.443 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T05:32:37.780 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package unzip. 2024-09-16T05:32:37.807 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T05:32:37.838 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T05:32:37.862 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T05:32:37.878 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:37.920 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:38.217 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package luarocks. 2024-09-16T05:32:38.244 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T05:32:38.265 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T05:32:38.283 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T05:32:38.292 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T05:32:38.331 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T05:32:38.618 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T05:32:38.645 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:38.649 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librgw2. 2024-09-16T05:32:38.676 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:38.676 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:38.707 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:39.113 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-common. 2024-09-16T05:32:39.140 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:39.179 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:39.218 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T05:32:39.233 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:39.276 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:39.621 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T05:32:39.648 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T05:32:39.687 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T05:32:40.007 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T05:32:40.034 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:40.073 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:40.426 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-common. 2024-09-16T05:32:40.453 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:40.484 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:40.676 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-base. 2024-09-16T05:32:40.703 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:40.752 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:41.324 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T05:32:41.349 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T05:32:41.388 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T05:32:41.691 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T05:32:41.718 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T05:32:41.758 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T05:32:41.931 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-base. 2024-09-16T05:32:41.958 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:42.008 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:42.060 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T05:32:42.077 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T05:32:42.110 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T05:32:42.388 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T05:32:42.415 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T05:32:42.455 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T05:32:42.593 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T05:32:42.620 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T05:32:42.660 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T05:32:42.758 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T05:32:42.785 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T05:32:42.824 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T05:32:43.127 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T05:32:43.154 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T05:32:43.193 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T05:32:43.196 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T05:32:43.223 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T05:32:43.263 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T05:32:43.447 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-portend. 2024-09-16T05:32:43.473 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T05:32:43.505 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T05:32:43.582 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T05:32:43.609 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T05:32:43.649 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T05:32:43.926 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T05:32:43.943 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T05:32:43.976 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T05:32:43.983 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T05:32:44.010 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T05:32:44.050 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T05:32:44.254 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T05:32:44.281 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T05:32:44.321 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T05:32:44.352 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T05:32:44.379 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T05:32:44.419 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T05:32:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T05:32:44.651 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T05:32:44.691 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T05:32:44.756 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T05:32:44.783 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T05:32:44.828 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T05:32:44.993 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-portend. 2024-09-16T05:32:45.020 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T05:32:45.059 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T05:32:45.133 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T05:32:45.160 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T05:32:45.200 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T05:32:45.303 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T05:32:45.331 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T05:32:45.370 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T05:32:45.478 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T05:32:45.505 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T05:32:45.544 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T05:32:45.656 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T05:32:45.684 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T05:32:45.723 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T05:32:45.822 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-mako. 2024-09-16T05:32:45.849 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T05:32:45.888 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T05:32:46.076 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T05:32:46.103 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T05:32:46.143 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T05:32:46.200 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T05:32:46.227 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T05:32:46.258 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T05:32:46.438 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T05:32:46.465 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T05:32:46.494 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T05:32:46.495 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T05:32:46.521 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T05:32:46.552 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T05:32:46.798 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T05:32:46.826 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T05:32:46.864 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webob. 2024-09-16T05:32:46.865 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T05:32:46.891 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T05:32:46.930 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T05:32:47.167 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-mako. 2024-09-16T05:32:47.195 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T05:32:47.234 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T05:32:47.275 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T05:32:47.302 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T05:32:47.344 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T05:32:47.562 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T05:32:47.589 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T05:32:47.637 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T05:32:47.669 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T05:32:47.696 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T05:32:47.735 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T05:32:47.915 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T05:32:47.942 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T05:32:47.973 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T05:32:48.022 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-paste. 2024-09-16T05:32:48.049 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T05:32:48.080 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T05:32:48.234 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-webob. 2024-09-16T05:32:48.261 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T05:32:48.300 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T05:32:48.358 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T05:32:48.374 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T05:32:48.408 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T05:32:48.637 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T05:32:48.664 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T05:32:48.677 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T05:32:48.704 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T05:32:48.709 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T05:32:48.744 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T05:32:48.998 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T05:32:49.014 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T05:32:49.047 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T05:32:49.048 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T05:32:49.074 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T05:32:49.114 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T05:32:49.392 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-paste. 2024-09-16T05:32:49.419 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T05:32:49.425 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T05:32:49.452 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T05:32:49.467 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T05:32:49.491 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T05:32:49.786 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T05:32:49.803 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T05:32:49.838 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T05:32:49.844 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T05:32:49.865 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T05:32:49.904 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T05:32:50.157 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T05:32:50.184 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T05:32:50.207 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-idna. 2024-09-16T05:32:50.223 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T05:32:50.224 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T05:32:50.257 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T05:32:50.534 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T05:32:50.535 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T05:32:50.561 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T05:32:50.562 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T05:32:50.600 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T05:32:50.601 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T05:32:50.870 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T05:32:50.897 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T05:32:50.898 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests. 2024-09-16T05:32:50.925 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T05:32:50.927 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T05:32:50.965 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T05:32:51.243 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T05:32:51.270 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T05:32:51.274 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T05:32:51.301 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T05:32:51.309 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T05:32:51.332 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T05:32:51.596 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T05:32:51.623 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:32:51.653 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:51.668 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-idna. 2024-09-16T05:32:51.696 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T05:32:51.735 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T05:32:52.055 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T05:32:52.082 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T05:32:52.092 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T05:32:52.119 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:52.121 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T05:32:52.158 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:52.424 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-requests. 2024-09-16T05:32:52.452 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T05:32:52.461 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T05:32:52.489 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T05:32:52.490 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T05:32:52.527 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T05:32:52.780 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T05:32:52.802 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T05:32:52.809 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T05:32:52.829 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T05:32:52.838 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T05:32:52.868 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T05:32:53.154 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T05:32:53.170 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:32:53.175 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T05:32:53.203 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:53.213 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:53.241 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:53.617 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T05:32:53.636 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T05:32:53.645 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:53.664 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:53.684 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:53.703 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:53.995 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T05:32:54.022 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T05:32:54.061 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T05:32:54.273 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T05:32:54.300 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:54.331 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:54.347 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T05:32:54.375 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T05:32:54.414 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T05:32:54.691 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T05:32:54.708 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:54.741 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:55.136 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T05:32:55.164 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:55.202 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:55.823 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T05:32:55.850 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:55.914 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:56.202 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph. 2024-09-16T05:32:56.218 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:56.260 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T05:32:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:56.959 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph. 2024-09-16T05:32:56.975 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:56.999 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T05:32:57.008 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:57.027 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:57.065 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:57.295 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T05:32:57.322 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:57.361 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:57.393 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package cephadm. 2024-09-16T05:32:57.421 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:57.476 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:57.756 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T05:32:57.784 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:57.796 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T05:32:57.823 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:57.823 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T05:32:57.862 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T05:32:58.140 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T05:32:58.164 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T05:32:58.206 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T05:32:58.225 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package cephadm. 2024-09-16T05:32:58.253 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:32:58.292 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:32:58.562 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T05:32:58.589 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T05:32:58.619 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T05:32:58.974 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T05:32:59.001 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T05:32:59.039 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T05:32:59.378 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-babel. 2024-09-16T05:32:59.406 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T05:32:59.444 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T05:32:59.764 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T05:32:59.791 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T05:32:59.839 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T05:33:00.052 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-babel. 2024-09-16T05:33:00.080 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T05:33:00.118 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T05:33:00.226 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T05:33:00.253 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:00.292 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:00.430 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T05:33:00.457 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T05:33:00.498 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T05:33:00.637 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T05:33:00.664 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T05:33:00.703 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T05:33:00.876 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T05:33:00.903 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:00.942 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:00.989 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-routes. 2024-09-16T05:33:01.017 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T05:33:01.056 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T05:33:01.279 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T05:33:01.306 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T05:33:01.345 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T05:33:01.409 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T05:33:01.437 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:01.476 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:01.648 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-routes. 2024-09-16T05:33:01.675 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T05:33:01.714 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T05:33:02.068 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T05:33:02.095 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:02.134 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:02.689 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T05:33:02.717 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T05:33:02.747 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T05:33:03.217 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T05:33:03.245 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T05:33:03.284 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T05:33:03.422 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T05:33:03.444 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T05:33:03.489 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T05:33:03.637 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T05:33:03.665 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T05:33:03.703 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T05:33:03.989 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T05:33:03.992 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T05:33:04.017 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T05:33:04.020 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T05:33:04.056 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T05:33:04.058 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T05:33:04.404 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T05:33:04.431 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T05:33:04.470 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T05:33:04.731 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T05:33:04.758 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T05:33:04.773 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T05:33:04.789 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T05:33:04.801 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:04.933 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:05.489 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T05:33:05.506 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:05.649 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:05.853 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T05:33:05.881 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T05:33:05.920 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T05:33:06.156 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T05:33:06.172 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T05:33:06.214 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T05:33:06.534 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T05:33:06.562 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T05:33:06.578 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T05:33:06.601 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T05:33:06.606 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T05:33:06.644 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T05:33:06.912 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T05:33:06.940 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T05:33:06.947 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T05:33:06.975 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T05:33:06.978 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T05:33:07.013 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T05:33:07.307 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T05:33:07.335 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T05:33:07.342 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T05:33:07.358 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T05:33:07.374 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T05:33:07.400 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T05:33:07.660 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T05:33:07.686 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T05:33:07.688 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T05:33:07.714 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T05:33:07.727 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T05:33:07.744 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T05:33:08.013 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T05:33:08.041 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T05:33:08.047 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T05:33:08.076 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T05:33:08.079 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T05:33:08.114 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T05:33:08.416 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T05:33:08.444 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T05:33:08.451 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T05:33:08.479 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T05:33:08.482 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T05:33:08.517 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T05:33:08.895 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T05:33:08.923 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T05:33:08.970 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T05:33:09.155 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T05:33:09.172 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:09.204 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:09.348 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T05:33:09.376 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T05:33:09.423 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T05:33:09.490 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T05:33:09.518 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T05:33:09.557 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T05:33:09.860 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T05:33:09.888 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T05:33:09.926 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T05:33:10.053 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T05:33:10.071 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:10.111 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:10.154 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package jq. 2024-09-16T05:33:10.182 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T05:33:10.221 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T05:33:10.439 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T05:33:10.467 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T05:33:10.473 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package socat. 2024-09-16T05:33:10.490 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T05:33:10.522 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T05:33:10.531 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T05:33:10.809 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T05:33:10.835 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T05:33:10.837 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T05:33:10.863 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T05:33:10.866 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T05:33:10.901 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T05:33:11.127 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package jq. 2024-09-16T05:33:11.144 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T05:33:11.186 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T05:33:11.239 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-test. 2024-09-16T05:33:11.256 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:33:11.288 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:11.438 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package socat. 2024-09-16T05:33:11.455 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T05:33:11.496 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T05:33:11.800 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T05:33:11.817 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T05:33:11.858 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T05:33:12.220 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-test. 2024-09-16T05:33:12.249 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:33:12.287 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:13.660 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T05:33:13.688 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:13.726 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:14.030 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T05:33:14.058 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:33:14.096 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:14.425 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T05:33:14.452 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T05:33:14.491 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T05:33:14.699 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T05:33:14.729 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T05:33:14.774 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:15.154 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T05:33:15.182 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T05:33:15.220 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T05:33:15.236 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T05:33:15.252 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:33:15.294 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:15.582 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T05:33:15.610 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T05:33:15.638 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T05:33:15.657 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T05:33:15.666 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T05:33:15.705 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T05:33:16.059 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T05:33:16.062 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package pkg-config. 2024-09-16T05:33:16.076 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T05:33:16.090 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T05:33:16.117 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T05:33:16.129 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T05:33:16.390 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T05:33:16.406 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T05:33:16.463 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T05:33:16.488 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T05:33:16.490 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T05:33:16.529 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T05:33:16.910 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T05:33:16.926 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package pkg-config. 2024-09-16T05:33:16.938 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T05:33:16.954 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T05:33:16.976 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T05:33:16.992 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T05:33:17.253 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T05:33:17.269 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T05:33:17.279 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T05:33:17.307 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T05:33:17.311 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T05:33:17.345 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T05:33:17.673 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T05:33:17.701 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T05:33:17.740 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T05:33:17.747 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T05:33:17.776 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T05:33:17.814 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T05:33:18.093 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T05:33:18.117 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T05:33:18.121 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T05:33:18.145 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T05:33:18.159 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T05:33:18.183 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T05:33:18.487 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-py. 2024-09-16T05:33:18.511 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T05:33:18.515 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T05:33:18.539 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T05:33:18.553 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T05:33:18.578 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T05:33:18.839 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T05:33:18.867 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T05:33:18.874 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T05:33:18.897 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T05:33:18.902 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T05:33:18.940 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T05:33:19.234 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-py. 2024-09-16T05:33:19.262 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T05:33:19.300 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T05:33:19.420 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T05:33:19.448 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T05:33:19.486 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T05:33:19.578 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T05:33:19.595 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T05:33:19.628 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T05:33:19.797 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-toml. 2024-09-16T05:33:19.825 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T05:33:19.863 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T05:33:20.099 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T05:33:20.127 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T05:33:20.165 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T05:33:20.200 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T05:33:20.228 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T05:33:20.268 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T05:33:20.518 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-toml. 2024-09-16T05:33:20.534 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T05:33:20.576 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T05:33:20.645 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T05:33:20.674 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T05:33:20.711 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T05:33:20.854 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T05:33:20.870 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T05:33:20.911 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T05:33:20.988 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T05:33:21.005 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T05:33:21.047 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T05:33:21.248 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T05:33:21.276 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T05:33:21.315 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T05:33:21.601 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T05:33:21.629 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T05:33:21.667 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T05:33:21.751 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package radosgw. 2024-09-16T05:33:21.779 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:33:21.817 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:22.346 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package radosgw. 2024-09-16T05:33:22.374 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:33:22.404 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:22.678 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T05:33:22.695 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:33:22.736 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:23.041 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T05:33:23.240 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T05:33:23.257 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T05:33:23.275 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T05:33:23.298 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:23.401 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T05:33:23.612 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T05:33:23.680 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T05:33:23.845 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T05:33:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T05:33:24.005 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T05:33:24.249 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T05:33:24.421 INFO:teuthology.orchestra.run.smithi006.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T05:33:24.494 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:24.610 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T05:33:25.019 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user cephadm....done 2024-09-16T05:33:25.044 INFO:teuthology.orchestra.run.smithi063.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T05:33:25.131 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:25.176 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T05:33:25.485 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T05:33:25.656 INFO:teuthology.orchestra.run.smithi063.stdout:Adding system user cephadm....done 2024-09-16T05:33:25.761 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T05:33:25.812 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T05:33:25.887 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T05:33:26.088 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T05:33:26.147 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T05:33:26.339 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T05:33:26.398 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T05:33:26.465 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T05:33:26.507 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T05:33:26.716 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T05:33:26.859 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T05:33:26.968 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T05:33:27.069 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T05:33:27.236 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T05:33:27.362 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T05:33:27.495 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T05:33:27.739 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T05:33:27.779 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T05:33:27.990 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T05:33:28.023 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T05:33:28.257 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T05:33:28.266 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T05:33:28.508 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T05:33:28.559 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T05:33:28.794 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T05:33:28.847 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:29.045 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T05:33:29.210 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T05:33:29.296 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:29.336 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T05:33:29.462 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T05:33:29.563 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T05:33:29.588 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T05:33:29.689 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T05:33:29.815 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T05:33:29.847 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T05:33:29.966 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T05:33:30.192 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T05:33:30.225 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T05:33:30.229 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T05:33:30.318 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T05:33:30.444 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T05:33:30.601 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T05:33:30.632 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T05:33:30.687 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T05:33:30.811 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T05:33:30.813 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T05:33:30.937 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T05:33:31.189 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T05:33:31.213 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T05:33:31.315 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T05:33:31.481 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T05:33:31.599 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T05:33:31.774 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T05:33:31.859 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T05:33:31.917 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T05:33:32.185 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T05:33:32.286 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T05:33:32.619 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T05:33:32.895 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T05:33:32.971 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T05:33:33.021 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T05:33:33.147 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T05:33:33.230 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T05:33:33.339 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T05:33:33.439 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T05:33:33.440 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T05:33:33.741 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T05:33:33.832 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T05:33:34.083 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T05:33:34.143 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T05:33:34.393 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T05:33:34.460 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T05:33:34.753 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T05:33:34.787 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T05:33:35.104 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T05:33:35.121 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T05:33:35.372 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T05:33:35.473 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:35.489 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T05:33:35.740 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T05:33:35.766 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T05:33:35.866 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:36.134 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T05:33:36.176 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T05:33:36.493 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T05:33:37.348 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T05:33:37.599 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:33:37.690 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T05:33:37.736 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T05:33:37.959 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:33:38.012 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T05:33:38.085 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T05:33:38.138 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T05:33:38.255 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T05:33:38.344 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T05:33:38.445 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T05:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T05:33:38.562 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T05:33:38.799 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T05:33:38.880 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T05:33:39.051 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:33:39.140 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T05:33:39.177 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T05:33:39.382 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:33:39.487 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T05:33:39.517 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T05:33:39.746 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T05:33:39.835 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T05:33:40.030 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T05:33:40.102 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T05:33:40.298 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T05:33:40.356 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T05:33:40.533 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T05:33:40.638 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T05:33:40.801 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T05:33:40.889 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T05:33:40.926 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T05:33:41.148 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T05:33:41.219 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T05:33:41.274 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T05:33:41.345 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T05:33:41.575 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T05:33:41.647 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T05:33:41.701 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T05:33:41.948 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T05:33:41.986 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T05:33:42.191 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T05:33:42.270 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T05:33:42.316 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T05:33:42.442 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T05:33:42.529 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T05:33:42.532 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T05:33:42.655 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T05:33:42.778 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T05:33:42.781 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T05:33:42.871 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T05:33:42.903 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T05:33:43.116 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T05:33:43.250 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T05:33:43.330 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T05:33:43.581 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T05:33:43.652 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T05:33:43.690 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T05:33:43.903 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T05:33:43.950 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:33:44.029 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T05:33:44.076 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T05:33:44.296 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T05:33:44.394 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T05:33:44.422 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T05:33:44.695 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T05:33:44.870 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T05:33:44.979 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T05:33:45.097 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T05:33:45.183 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T05:33:45.365 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T05:33:45.459 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T05:33:45.577 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T05:33:45.624 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T05:33:45.853 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T05:33:46.112 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T05:33:46.303 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T05:33:46.457 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T05:33:46.687 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T05:33:46.860 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T05:33:46.989 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T05:33:47.145 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T05:33:47.333 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T05:33:47.459 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:47.498 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T05:33:47.586 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:47.624 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:47.711 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T05:33:47.750 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:47.893 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T05:33:48.048 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.149 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.187 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.275 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T05:33:48.313 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.439 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T05:33:48.551 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.678 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.716 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.804 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.842 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:48.968 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:49.025 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T05:33:49.026 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T05:33:49.204 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T05:33:49.204 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T05:33:49.530 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:49.647 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:49.686 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:49.749 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:49.812 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:49.866 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T05:33:49.930 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:50.031 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T05:33:51.170 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.296 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.318 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.405 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.428 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.506 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.529 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.607 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.646 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:51.772 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:52.150 INFO:teuthology.orchestra.run.smithi006.stdout:Adding group ceph....done 2024-09-16T05:33:52.307 INFO:teuthology.orchestra.run.smithi063.stdout:Adding group ceph....done 2024-09-16T05:33:52.510 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user ceph....done 2024-09-16T05:33:52.678 INFO:teuthology.orchestra.run.smithi006.stdout:Setting system user ceph properties....done 2024-09-16T05:33:52.684 INFO:teuthology.orchestra.run.smithi063.stdout:Adding system user ceph....done 2024-09-16T05:33:52.690 INFO:teuthology.orchestra.run.smithi006.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T05:33:52.832 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T05:33:52.844 INFO:teuthology.orchestra.run.smithi063.stdout:Setting system user ceph properties....done 2024-09-16T05:33:52.861 INFO:teuthology.orchestra.run.smithi063.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T05:33:53.013 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T05:33:53.146 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T05:33:53.335 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T05:33:53.626 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:53.751 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:53.832 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:53.958 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:54.236 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:33:54.236 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:33:54.454 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:33:54.454 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:33:54.695 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:54.927 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:55.059 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T05:33:55.288 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T05:33:55.530 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:55.753 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T05:33:55.753 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T05:33:55.754 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:56.057 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T05:33:56.057 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T05:33:56.223 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:56.452 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T05:33:56.452 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T05:33:56.598 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:56.819 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T05:33:56.819 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T05:33:56.950 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:57.241 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T05:33:57.241 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T05:33:57.299 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:57.582 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T05:33:57.582 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T05:33:57.727 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:57.853 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:58.012 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:58.068 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:58.194 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:58.233 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T05:33:58.233 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T05:33:58.345 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:58.565 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T05:33:58.565 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T05:33:58.747 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:58.881 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:58.982 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:59.071 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:59.133 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:59.222 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:59.323 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:59.474 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T05:33:59.595 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T05:33:59.754 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T05:33:59.852 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T05:33:59.906 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T05:33:59.970 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T05:34:00.113 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T05:34:00.940 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T05:34:01.377 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T05:34:01.591 INFO:teuthology.orchestra.run.smithi006.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T05:34:01.597 DEBUG:teuthology.parallel:result is None 2024-09-16T05:34:02.026 INFO:teuthology.orchestra.run.smithi063.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T05:34:02.031 DEBUG:teuthology.parallel:result is None 2024-09-16T05:34:02.032 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:34:02.188 DEBUG:teuthology.orchestra.run.smithi006:> dpkg-query -W -f '${Version}' ceph 2024-09-16T05:34:02.209 INFO:teuthology.orchestra.run.smithi006.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:34:02.209 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:34:02.210 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T05:34:02.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:34:02.352 DEBUG:teuthology.orchestra.run.smithi063:> dpkg-query -W -f '${Version}' ceph 2024-09-16T05:34:02.372 INFO:teuthology.orchestra.run.smithi063.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:34:02.372 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T05:34:02.373 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T05:34:02.374 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T05:34:02.375 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:02.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T05:34:02.389 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:02.389 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T05:34:02.428 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T05:34:02.428 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:02.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T05:34:02.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T05:34:02.500 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:02.500 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T05:34:02.514 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T05:34:02.570 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T05:34:02.570 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:02.570 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T05:34:02.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T05:34:02.639 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:02.639 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T05:34:02.652 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T05:34:02.707 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T05:34:02.708 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:02.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T05:34:02.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T05:34:02.779 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:02.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T05:34:02.794 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T05:34:02.850 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T05:34:02.980 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T05:34:02.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-09-16T05:34:02.983 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /var/log/ceph 2024-09-16T05:34:02.997 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T05:34:02.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T05:34:03.041 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T05:34:03.054 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T05:34:03.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T05:34:03.097 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T05:34:03.135 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T05:34:03.135 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-16T05:34:03.136 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162', 'branch': 'squid', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906162', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'mon_bind_msgr2': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFlkKbOObKZbHVFWJLBxZzrukUnFl1HZyMiqBSfFs4bQOtb5bqVNz173Cfq9Z4l7j78FXIzL9mhQ101oT5a5jQ=', 'smithi063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCFFkFoHe7x+wAmwAKEWqC7x/6QIMVmZhtDBz9W8oCwoQW1aVaiUfsADK3f9alaCQmDL7v69HQbzuq6llgt5KYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T05:34:03.136 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T05:34:03.153 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T05:34:03.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T05:34:03.201 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T05:34:03.215 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:03.215 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T05:34:03.259 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T05:34:03.260 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-09-16T05:34:03.308 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T05:34:03.308 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:34:03.308 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 880 Links: 1 2024-09-16T05:34:03.308 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:34:03.308 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-16 05:28:33.941857078 +0000 2024-09-16T05:34:03.308 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-16 05:28:33.681859400 +0000 2024-09-16T05:34:03.309 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-16 05:28:33.681859400 +0000 2024-09-16T05:34:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-16T05:34:03.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T05:34:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-16T05:34:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-16T05:34:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000456283 s, 1.1 MB/s 2024-09-16T05:34:03.364 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T05:34:03.413 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-09-16T05:34:03.460 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T05:34:03.460 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:34:03.460 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 897 Links: 1 2024-09-16T05:34:03.460 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:34:03.461 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-16 05:28:34.413852862 +0000 2024-09-16T05:34:03.461 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-16 05:28:34.165855077 +0000 2024-09-16T05:34:03.461 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-16 05:28:34.165855077 +0000 2024-09-16T05:34:03.461 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-16T05:34:03.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T05:34:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-16T05:34:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-16T05:34:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000443283 s, 1.2 MB/s 2024-09-16T05:34:03.516 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T05:34:03.569 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-09-16T05:34:03.616 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T05:34:03.616 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:34:03.616 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-09-16T05:34:03.616 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:34:03.616 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-16 05:28:34.949848074 +0000 2024-09-16T05:34:03.616 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-16 05:28:34.669850576 +0000 2024-09-16T05:34:03.616 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-16 05:28:34.669850576 +0000 2024-09-16T05:34:03.616 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-16T05:34:03.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T05:34:03.674 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-16T05:34:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-16T05:34:03.675 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000426848 s, 1.2 MB/s 2024-09-16T05:34:03.676 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T05:34:03.726 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-09-16T05:34:03.772 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T05:34:03.772 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:34:03.772 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 931 Links: 1 2024-09-16T05:34:03.772 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:34:03.772 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-16 05:28:35.429843787 +0000 2024-09-16T05:34:03.772 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-16 05:28:35.193845895 +0000 2024-09-16T05:34:03.772 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-16 05:28:35.193845895 +0000 2024-09-16T05:34:03.772 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-16T05:34:03.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T05:34:03.826 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-16T05:34:03.827 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-16T05:34:03.827 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.00031736 s, 1.6 MB/s 2024-09-16T05:34:03.828 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T05:34:03.877 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T05:34:03.877 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:03.877 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T05:34:03.883 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T05:34:03.883 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-09-16T05:34:03.931 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T05:34:03.931 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:34:03.931 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 876 Links: 1 2024-09-16T05:34:03.932 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:34:03.932 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-16 05:28:34.042901947 +0000 2024-09-16T05:34:03.932 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-16 05:28:33.806904055 +0000 2024-09-16T05:34:03.932 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-16 05:28:33.806904055 +0000 2024-09-16T05:34:03.932 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-09-16T05:34:03.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T05:34:03.986 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-16T05:34:03.986 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-16T05:34:03.986 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000383127 s, 1.3 MB/s 2024-09-16T05:34:03.987 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T05:34:04.036 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-09-16T05:34:04.083 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T05:34:04.083 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:34:04.083 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-09-16T05:34:04.084 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:34:04.084 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-16 05:28:34.530897587 +0000 2024-09-16T05:34:04.084 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-16 05:28:34.286899768 +0000 2024-09-16T05:34:04.084 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-16 05:28:34.286899768 +0000 2024-09-16T05:34:04.084 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-09-16T05:34:04.084 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T05:34:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-16T05:34:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-16T05:34:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000425529 s, 1.2 MB/s 2024-09-16T05:34:04.139 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T05:34:04.188 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-09-16T05:34:04.235 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T05:34:04.235 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:34:04.235 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-09-16T05:34:04.235 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:34:04.235 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-16 05:28:35.018893229 +0000 2024-09-16T05:34:04.235 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-16 05:28:34.782895337 +0000 2024-09-16T05:34:04.236 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-16 05:28:34.782895337 +0000 2024-09-16T05:34:04.236 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-09-16T05:34:04.236 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T05:34:04.289 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-16T05:34:04.289 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-16T05:34:04.289 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000351043 s, 1.5 MB/s 2024-09-16T05:34:04.291 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T05:34:04.340 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-09-16T05:34:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T05:34:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T05:34:04.387 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 931 Links: 1 2024-09-16T05:34:04.387 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T05:34:04.387 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-16 05:28:35.618887870 +0000 2024-09-16T05:34:04.387 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-16 05:28:35.366890121 +0000 2024-09-16T05:34:04.387 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-16 05:28:35.366890121 +0000 2024-09-16T05:34:04.387 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-09-16T05:34:04.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T05:34:04.441 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-16T05:34:04.442 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-16T05:34:04.442 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000349791 s, 1.5 MB/s 2024-09-16T05:34:04.443 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T05:34:04.493 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T05:34:04.493 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T05:34:04.493 INFO:tasks.ceph:Generating config... 2024-09-16T05:34:04.494 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T05:34:04.494 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T05:34:04.495 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T05:34:04.495 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T05:34:04.495 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-16T05:34:04.495 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T05:34:04.495 INFO:tasks.ceph:[global] ms type = async 2024-09-16T05:34:04.495 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T05:34:04.495 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T05:34:04.495 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T05:34:04.495 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T05:34:04.496 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T05:34:04.496 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T05:34:04.496 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T05:34:04.496 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T05:34:04.496 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T05:34:04.496 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T05:34:04.496 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T05:34:04.496 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T05:34:04.497 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T05:34:04.497 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T05:34:04.498 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T05:34:04.498 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T05:34:04.498 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T05:34:04.498 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T05:34:04.498 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T05:34:04.498 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T05:34:04.498 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T05:34:04.498 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T05:34:04.499 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T05:34:04.499 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T05:34:04.499 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T05:34:04.499 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T05:34:04.499 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T05:34:04.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T05:34:04.534 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T05:34:04.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T05:34:04.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T05:34:04.674 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.6:6789]'), ('mon.c', '[v1:172.21.15.6:6790]'), ('mon.b', '[v1:172.21.15.63:6789]')] 2024-09-16T05:34:04.675 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.63:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T05:34:04.677 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:04.677 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T05:34:04.721 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.6:6789]' --addv c '[v1:172.21.15.6:6790]' --addv b '[v1:172.21.15.63:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:04.807 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:04.808 INFO:teuthology.orchestra.run.smithi006.stdout:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:04.809 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T05:34:04.810 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T05:34:04.815 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 22154726-d7f4-4d5e-b601-06ee1f3e7219... 2024-09-16T05:34:04.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T05:34:04.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-09-16T05:34:04.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.891 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.892 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.63:6789]" 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: mon election default strategy = 1 2024-09-16T05:34:04.893 INFO:teuthology.orchestra.run.smithi006.stdout: ms bind msgr2 = False 2024-09-16T05:34:04.894 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 5000 2024-09-16T05:34:04.894 INFO:teuthology.orchestra.run.smithi006.stdout: ms type = async 2024-09-16T05:34:04.894 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:04.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.894 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-09-16T05:34:04.894 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-09-16T05:34:04.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.894 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-09-16T05:34:04.895 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-09-16T05:34:04.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.896 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-09-16T05:34:04.896 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-09-16T05:34:04.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.896 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-09-16T05:34:04.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.896 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-09-16T05:34:04.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = lz4 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-09-16T05:34:04.897 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore zero block detection = True 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify cached snaps = True 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify missing on start = True 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock override recovery settings = True 2024-09-16T05:34:04.898 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue = debug_random 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue cut off = debug_random 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.899 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T05:34:04.900 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon min osdmap epochs = 25 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune interval = 2 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune min = 15 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune txsize = 2 2024-09-16T05:34:04.901 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub inject missing keys = 0.05 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub interval = 300 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: paxos service trim min = 5 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-09-16T05:34:04.902 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-09-16T05:34:04.903 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-09-16T05:34:04.904 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-09-16T05:34:04.904 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2024-09-16T05:34:04.904 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T05:34:04.904 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2024-09-16T05:34:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2024-09-16T05:34:04.907 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = "[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.63:6789]" 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: mon client directed command retry = 5 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: mon election default strategy = 1 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: ms bind msgr2 = False 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: ms inject socket failures = 5000 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: ms type = async 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2024-09-16T05:34:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock profile = high_recovery_ops 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug trim objects = true 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.910 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore block size = 96636764160 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore compression algorithm = lz4 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore compression mode = aggressive 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore fsck on mount = True 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore zero block detection = True 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluefs = 20 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluestore = 20 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: debug rocksdb = 10 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd full ratio = 0.9 2024-09-16T05:34:04.911 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd nearfull ratio = 0.8 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify cached snaps = True 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify missing on start = True 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: osd failsafe full ratio = 0.95 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock override recovery settings = True 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: osd objectstore = bluestore 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: osd op queue = debug_random 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: osd op queue cut off = debug_random 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-09-16T05:34:04.912 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2024-09-16T05:34:04.913 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: # 1m isn't quite enough 2024-09-16T05:34:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: mon_down_mkfs_grace = 2m 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_filestore_osds = false 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: mon min osdmap epochs = 25 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune interval = 2 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune min = 15 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune txsize = 2 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: mon scrub inject missing keys = 0.05 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: mon scrub interval = 300 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: paxos service trim min = 5 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2024-09-16T05:34:04.915 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 20 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.c] 2024-09-16T05:34:04.916 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.b] 2024-09-16T05:34:04.916 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T05:34:04.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T05:34:04.998 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T05:34:04.998 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:04.998 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T05:34:05.003 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:05.003 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T05:34:05.051 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:34:05.051 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:05.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T05:34:05.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T05:34:05.115 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:05.115 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:05.164 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:34:05.164 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:05.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T05:34:05.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T05:34:05.188 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:05.188 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:05.235 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T05:34:05.235 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T05:34:05.235 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T05:34:05.322 INFO:teuthology.orchestra.run.smithi063.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T05:34:05.342 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T05:34:05.342 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T05:34:05.342 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T05:34:05.420 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T05:34:05.431 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T05:34:05.431 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T05:34:05.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T05:34:05.445 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T05:34:05.445 INFO:tasks.ceph:role: osd.0 2024-09-16T05:34:05.445 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:34:05.446 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T05:34:05.507 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:34:05.507 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:34:05.507 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:34:05.507 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:34:05.507 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:34:05.507 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:34:05.507 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:34:05.508 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:34:05.508 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:34:05.508 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:34:05.961 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-16T05:34:05.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-16T05:34:05.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T05:34:06.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T05:34:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:34:06.075 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:34:06.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T05:34:06.129 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T05:34:06.130 INFO:tasks.ceph:role: osd.1 2024-09-16T05:34:06.130 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:34:06.130 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:34:06.195 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:34:06.196 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:34:06.660 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-16T05:34:06.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-16T05:34:06.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T05:34:06.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T05:34:06.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:34:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:34:06.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T05:34:06.860 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T05:34:06.860 INFO:tasks.ceph:role: osd.2 2024-09-16T05:34:06.860 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:34:06.860 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T05:34:06.924 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:34:06.924 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:34:06.924 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:34:06.924 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:34:06.924 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:34:06.924 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:34:06.924 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:34:06.924 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:34:06.925 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:34:06.925 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:34:07.412 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-16T05:34:07.415 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-16T05:34:07.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T05:34:07.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T05:34:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:34:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:34:07.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:07.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:07.675+0000 7f4d60a027c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T05:34:07.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:07.675+0000 7f4d60a027c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T05:34:07.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:07.675+0000 7f4d60a027c0 -1 bdev(0x55568a9e7800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T05:34:07.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:07.675+0000 7f4d60a027c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T05:34:10.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T05:34:10.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:10.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:10.891+0000 7fb4c33dc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T05:34:10.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:10.895+0000 7fb4c33dc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T05:34:10.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:10.895+0000 7fb4c33dc7c0 -1 bdev(0x55e8e1d8d800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T05:34:10.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:10.895+0000 7fb4c33dc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T05:34:14.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T05:34:14.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:14.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:14.163+0000 7f446c1207c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T05:34:14.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:14.163+0000 7f446c1207c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T05:34:14.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:14.163+0000 7f446c1207c0 -1 bdev(0x55d5cb9dd800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T05:34:14.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:14.163+0000 7f446c1207c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T05:34:17.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T05:34:17.365 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T05:34:17.380 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T05:34:17.380 INFO:tasks.ceph:role: osd.3 2024-09-16T05:34:17.380 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:34:17.380 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T05:34:17.442 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:34:17.443 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:34:17.896 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-09-16T05:34:17.898 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-09-16T05:34:17.898 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T05:34:18.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T05:34:18.010 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:34:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:34:18.011 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T05:34:18.066 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T05:34:18.066 INFO:tasks.ceph:role: osd.4 2024-09-16T05:34:18.066 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:34:18.066 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:34:18.131 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:34:18.601 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-09-16T05:34:18.603 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-09-16T05:34:18.603 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T05:34:18.736 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T05:34:18.747 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:34:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:34:18.748 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T05:34:18.803 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T05:34:18.803 INFO:tasks.ceph:role: osd.5 2024-09-16T05:34:18.803 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:34:18.804 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T05:34:18.869 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T05:34:18.869 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T05:34:18.869 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T05:34:18.870 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T05:34:18.870 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T05:34:18.870 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-09-16T05:34:18.870 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T05:34:18.870 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T05:34:18.870 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T05:34:18.870 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T05:34:19.361 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-09-16T05:34:19.363 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-09-16T05:34:19.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T05:34:19.509 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T05:34:19.522 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T05:34:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:34:19.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:19.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:19.655+0000 7fd5736b17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T05:34:19.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:19.655+0000 7fd5736b17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T05:34:19.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:19.655+0000 7fd5736b17c0 -1 bdev(0x555a73e75800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T05:34:19.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:19.655+0000 7fd5736b17c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T05:34:22.787 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T05:34:22.801 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:22.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:22.883+0000 7fc6d24ae7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T05:34:22.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:22.883+0000 7fc6d24ae7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T05:34:22.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:22.883+0000 7fc6d24ae7c0 -1 bdev(0x5629f9eb3800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T05:34:22.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:22.883+0000 7fc6d24ae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T05:34:26.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T05:34:26.084 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:26.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:26.167+0000 7f7e1f1407c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T05:34:26.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:26.167+0000 7f7e1f1407c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T05:34:26.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:26.167+0000 7f7e1f1407c0 -1 bdev(0x563e40b7f800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T05:34:26.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:26.167+0000 7f7e1f1407c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T05:34:29.302 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T05:34:29.359 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T05:34:29.360 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:29.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T05:34:29.375 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:29.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T05:34:29.432 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:29.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T05:34:29.493 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:29.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T05:34:29.507 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:29.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T05:34:29.563 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:29.563 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T05:34:29.619 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:29.619 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T05:34:29.674 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:29.674 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T05:34:29.723 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T05:34:29.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T05:34:29.726 DEBUG:teuthology.orchestra.run.smithi063:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBPw+dmpRifKBAAwxahtWI25AOXiDK9M+mung== 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBSw+dmI86TNRAAIibU2KaGvbo7N0wMhNSaew== 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBWw+dmlAsCChAAiPp4XE1gQy6ITV+0PNwD5w== 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBNw+dmWiBUExAAbHPXiiz73lqdWi2iV/Whpw== 2024-09-16T05:34:29.769 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-09-16T05:34:29.770 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBbw+dmlictJxAAN0gSsZALn/VZQMwMO/4pxw== 2024-09-16T05:34:29.770 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-09-16T05:34:29.770 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBew+dmMDPINBAANV6AI8zf+GYJgL/S0l2U6g== 2024-09-16T05:34:29.770 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-09-16T05:34:29.770 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBiw+dmOKkSChAAn1Q6i+vHPfYtk7+M4/w3/w== 2024-09-16T05:34:29.770 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-09-16T05:34:29.770 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBNw+dmRgcwGRAAD4LTCyd4HdSxElzS1pgRIQ== 2024-09-16T05:34:29.777 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBPw+dmpRifKBAAwxahtWI25AOXiDK9M+mung== 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBSw+dmI86TNRAAIibU2KaGvbo7N0wMhNSaew== 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBWw+dmlAsCChAAiPp4XE1gQy6ITV+0PNwD5w== 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.x] 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBNw+dmWiBUExAAbHPXiiz73lqdWi2iV/Whpw== 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.3] 2024-09-16T05:34:29.778 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBbw+dmlictJxAAN0gSsZALn/VZQMwMO/4pxw== 2024-09-16T05:34:29.779 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.4] 2024-09-16T05:34:29.779 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBew+dmMDPINBAANV6AI8zf+GYJgL/S0l2U6g== 2024-09-16T05:34:29.779 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.5] 2024-09-16T05:34:29.779 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBiw+dmOKkSChAAn1Q6i+vHPfYtk7+M4/w3/w== 2024-09-16T05:34:29.779 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-09-16T05:34:29.779 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBNw+dmRgcwGRAAD4LTCyd4HdSxElzS1pgRIQ== 2024-09-16T05:34:29.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:29.817 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:29.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:29.871 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:29.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:29.959 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:30.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T05:34:30.045 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T05:34:30.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:30.092 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:30.171 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:30.176 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:30.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:30.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T05:34:30.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T05:34:30.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T05:34:30.426 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T05:34:30.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T05:34:30.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T05:34:30.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T05:34:30.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T05:34:30.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T05:34:30.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T05:34:30.698 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T05:34:30.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T05:34:30.807 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T05:34:30.817 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:30.820 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T05:34:30.864 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T05:34:30.865 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T05:34:30.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T05:34:30.868 INFO:tasks.ceph.mon.a:Started 2024-09-16T05:34:30.868 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T05:34:30.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T05:34:30.871 INFO:tasks.ceph.mon.c:Started 2024-09-16T05:34:30.871 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:34:30.871 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:34:30.908 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:34:30.908 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T05:34:30.908 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T05:34:30.908 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T05:34:30.911 INFO:tasks.ceph.mgr.x:Started 2024-09-16T05:34:30.911 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T05:34:30.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T05:34:31.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.023+0000 7f6dea18f640 10 monclient: get_monmap_and_config 2024-09-16T05:34:31.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.023+0000 7f6dea18f640 10 monclient: build_initial_monmap 2024-09-16T05:34:31.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.023+0000 7f6dea18f640 10 monclient: monmap: 2024-09-16T05:34:31.030 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:31.030248+0000 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:31.030248+0000 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:31.031 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:31.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 1 Processor -- start 2024-09-16T05:34:31.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 1 -- start start 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 10 monclient: init 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 10 monclient(hunting): picked mon.noname-a con 0x7f6de40a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 10 monclient(hunting): picked mon.noname-b con 0x7f6de40a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 10 monclient(hunting): picked mon.noname-c con 0x7f6de40a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de406e520 con 0x7f6de40a37a0 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de40a0780 con 0x7f6de40a1a00 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de40a4c70 con 0x7f6de40a4290 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6dea18f640 10 monclient(hunting): authenticate will time out at 932.836907s 2024-09-16T05:34:31.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6de2ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6de40a4290 0x7f6de413bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:44072/0 (socket says 172.21.15.6:44072) 2024-09-16T05:34:31.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:31.027+0000 7f6de2ffd640 1 -- 172.21.15.6:0/2390471246 learned_addr learned my addr 172.21.15.6:0/2390471246 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 10 monclient(hunting): tick 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 1 monclient(hunting): continuing hunt 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 1 -- 172.21.15.6:0/2390471246 >> v1:172.21.15.6:6790/0 conn(0x7f6de40a4290 legacy=0x7f6de413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 1 -- 172.21.15.6:0/2390471246 >> v1:172.21.15.63:6789/0 conn(0x7f6de40a1a00 legacy=0x7f6de413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 1 -- 172.21.15.6:0/2390471246 >> v1:172.21.15.6:6789/0 conn(0x7f6de40a37a0 legacy=0x7f6de40a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 10 monclient(hunting): picked mon.noname-c con 0x7f6dd00036f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 10 monclient(hunting): picked mon.noname-a con 0x7f6dd00073b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 10 monclient(hunting): picked mon.noname-b con 0x7f6dd000b0e0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 1 -- 172.21.15.6:0/2390471246 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de40a65f0 con 0x7f6dd00073b0 2024-09-16T05:34:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 1 -- 172.21.15.6:0/2390471246 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dd000fdf0 con 0x7f6dd00036f0 2024-09-16T05:34:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 1 -- 172.21.15.6:0/2390471246 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dd0011020 con 0x7f6dd000b0e0 2024-09-16T05:34:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:34.027+0000 7f6de17fa640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26684082 0 0) 0x7f6de40a65f0 con 0x7f6dd00073b0 2024-09-16T05:34:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): my global_id is 4098 2024-09-16T05:34:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dcc003530 con 0x7f6dd00073b0 2024-09-16T05:34:36.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715088982 0 0) 0x7f6dd0011020 con 0x7f6dd000b0e0 2024-09-16T05:34:36.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): my global_id is 4103 2024-09-16T05:34:36.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6de40a65f0 con 0x7f6dd000b0e0 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4037045553 0 0) 0x7f6dd000fdf0 con 0x7f6dd00036f0 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 10 monclient(hunting): my global_id is 4102 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd0011020 con 0x7f6dd00036f0 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2102330727 0 0) 0x7f6de40a65f0 con 0x7f6dd000b0e0 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 >> v1:172.21.15.63:6789/0 conn(0x7f6dd00036f0 legacy=0x7f6dd0005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.251+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 >> v1:172.21.15.6:6789/0 conn(0x7f6dd00073b0 legacy=0x7f6dd0009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 1 monclient: found mon.noname-b 2024-09-16T05:34:36.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:36.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:36.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd0012250 con 0x7f6dd000b0e0 2024-09-16T05:34:36.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: _finish_auth 0 2024-09-16T05:34:36.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:34:36.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:36.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6ddc001a20 con 0x7f6dd000b0e0 2024-09-16T05:34:36.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 5 monclient: authenticate success, global_id 4103 2024-09-16T05:34:36.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: dump: 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:36.272 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ddc001cc0 con 0x7f6dd000b0e0 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 1 -- 172.21.15.6:0/2390471246 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6ddc0020d0 con 0x7f6dd000b0e0 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 4 set_mon_vals no callback set 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6de27fc640 10 monclient: dump: 2024-09-16T05:34:36.273 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 10 monclient: shutdown 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:36.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 1 -- 172.21.15.6:0/2390471246 >> v1:172.21.15.6:6790/0 conn(0x7f6dd000b0e0 legacy=0x7f6dd000d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 1 -- 172.21.15.6:0/2390471246 shutdown_connections 2024-09-16T05:34:36.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 1 -- 172.21.15.6:0/2390471246 >> 172.21.15.6:0/2390471246 conn(0x7f6de40a58a0 msgr2=0x7f6de40a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:36.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 1 -- 172.21.15.6:0/2390471246 shutdown_connections 2024-09-16T05:34:36.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.255+0000 7f6dea18f640 1 -- 172.21.15.6:0/2390471246 wait complete. 2024-09-16T05:34:36.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient: build_initial_monmap 2024-09-16T05:34:36.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient: monmap: 2024-09-16T05:34:36.275 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 1 Processor -- start 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 1 -- start start 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient: init 2024-09-16T05:34:36.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient(hunting): picked mon.noname-c con 0x7f6dd00036f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient(hunting): picked mon.noname-a con 0x7f6dd00073b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient(hunting): picked mon.noname-b con 0x7f6dd000b0e0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de4124210 con 0x7f6dd00073b0 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de41244f0 con 0x7f6dd000b0e0 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de40a8710 con 0x7f6dd00036f0 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6de37fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6dd00036f0 0x7f6de41cb420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:44090/0 (socket says 172.21.15.6:44090) 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6de37fe640 1 -- 172.21.15.6:0/20124608 learned_addr learned my addr 172.21.15.6:0/20124608 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:36.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dea18f640 10 monclient(hunting): authenticate will time out at 938.070409s 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590929962 0 0) 0x7f6de41244f0 con 0x7f6dd000b0e0 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): my global_id is 4108 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd000ee70 con 0x7f6dd000b0e0 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6de3fff640 1 -- 172.21.15.6:0/20124608 >> v1:172.21.15.63:6789/0 conn(0x7f6dd000b0e0 legacy=0x7f6de41d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6de3fff640 1 -- 172.21.15.6:0/20124608 >> v1:172.21.15.63:6789/0 conn(0x7f6dd000b0e0 legacy=0x7f6de41d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6de3fff640 1 --1- 172.21.15.6:0/20124608 >> v1:172.21.15.63:6789/0 conn(0x7f6dd000b0e0 0x7f6de41d3780 :-1 s=OPENED pgs=13 cs=1 l=1).handle_message read tag failed 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6de3fff640 1 --1- 172.21.15.6:0/20124608 >> v1:172.21.15.63:6789/0 conn(0x7f6dd000b0e0 0x7f6de41d3780 :-1 s=OPENED pgs=13 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6789/0 2024-09-16T05:34:36.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3219797680 0 0) 0x7f6de40a8710 con 0x7f6dd00036f0 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 10 monclient(hunting): my global_id is 4109 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.259+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6de41244f0 con 0x7f6dd00036f0 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2206966217 0 0) 0x7f6de4124210 con 0x7f6dd00073b0 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient(hunting): my global_id is 4107 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6de40a8710 con 0x7f6dd00073b0 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 286271223 0 0) 0x7f6de40a8710 con 0x7f6dd00073b0 2024-09-16T05:34:36.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6de4124210 con 0x7f6dd00073b0 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6dd80017c0 con 0x7f6dd00073b0 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient(hunting): dump: 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:36.280 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:36.281 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:36.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:36.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2149700173 0 0) 0x7f6de4124210 con 0x7f6dd00073b0 2024-09-16T05:34:36.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 >> v1:172.21.15.6:6790/0 conn(0x7f6dd00036f0 legacy=0x7f6de41cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 >> v1:172.21.15.63:6789/0 conn(0x7f6dd000b0e0 legacy=0x7f6de41d3780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:34:36.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:34:36.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 monclient: found mon.a 2024-09-16T05:34:36.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:36.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:36.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de4137600 con 0x7f6dd00073b0 2024-09-16T05:34:36.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient: _finish_auth 0 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 5 monclient: authenticate success, global_id 4107 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 10 monclient: _renew_subs 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6de4123f30 con 0x7f6dd00073b0 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 10 monclient: _renew_subs 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6de4137240 con 0x7f6dd00073b0 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:36.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:36.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de40008e0 con 0x7f6dd00073b0 2024-09-16T05:34:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6dd80047f0 con 0x7f6dd00073b0 2024-09-16T05:34:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6dd8004b40 con 0x7f6dd00073b0 2024-09-16T05:34:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 10 monclient: dump: 2024-09-16T05:34:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:36.285 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:36.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6dd8005070 con 0x7f6dd00073b0 2024-09-16T05:34:36.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6de1ffb640 4 set_mon_vals no callback set 2024-09-16T05:34:36.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.263+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1721812870 0 0) 0x7f6dd8006580 con 0x7f6dd00073b0 2024-09-16T05:34:36.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.267+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f6dd8005af0 con 0x7f6dd00073b0 2024-09-16T05:34:36.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.267+0000 7f6dcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:36.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.267+0000 7f6dcbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:36.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.339+0000 7f6dea18f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T05:34:36.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.339+0000 7f6dea18f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T05:34:36.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.339+0000 7f6dea18f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:36.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.339+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6de40a3d90 con 0x7f6dd00073b0 2024-09-16T05:34:36.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.339+0000 7f6dcbfff640 1 -- 172.21.15.6:0/20124608 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7f6dd8005360 con 0x7f6dd00073b0 2024-09-16T05:34:36.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.339+0000 7f6dcbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T05:34:36.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.339+0000 7f6dcbfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T05:34:36.347 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-09-16T05:34:36.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.343+0000 7f6dea18f640 10 monclient: shutdown 2024-09-16T05:34:36.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.343+0000 7f6dea18f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:36.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.343+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 >> v1:172.21.15.6:6789/0 conn(0x7f6dd00073b0 legacy=0x7f6de4136b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.343+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 shutdown_connections 2024-09-16T05:34:36.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.343+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 >> 172.21.15.6:0/20124608 conn(0x7f6de40a58a0 msgr2=0x7f6de41beb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:36.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.343+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 shutdown_connections 2024-09-16T05:34:36.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:36.343+0000 7f6dea18f640 1 -- 172.21.15.6:0/20124608 wait complete. 2024-09-16T05:34:36.365 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T05:34:36.366 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T05:34:36.366 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:36.366 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T05:34:36.377 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:36.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T05:34:36.434 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-16T05:34:36.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T05:34:36.486 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:36.486 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T05:34:36.501 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:36.501 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T05:34:36.555 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:34:36.556 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T05:34:36.611 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 28573ac1-022a-424e-a024-a8ae14680604 0 2024-09-16T05:34:36.707 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:36.707+0000 7f650b3ec180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T05:34:36.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient: get_monmap_and_config 2024-09-16T05:34:36.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient: build_initial_monmap 2024-09-16T05:34:36.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient: monmap: 2024-09-16T05:34:36.780 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:36.780 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.781 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:36.782030+0000 2024-09-16T05:34:36.781 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:36.782030+0000 2024-09-16T05:34:36.781 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:36.781 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:36.781 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:36.781 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:36.781 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:36.781 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:36.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 1 Processor -- start 2024-09-16T05:34:36.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 1 -- start start 2024-09-16T05:34:36.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient: init 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient(hunting): picked mon.noname-a con 0x7f91b4134880 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient(hunting): picked mon.noname-b con 0x7f91b40a0f20 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient(hunting): picked mon.noname-c con 0x7f91b413ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b40a3180 con 0x7f91b4134880 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b409f2a0 con 0x7f91b413ae70 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b40a0260 con 0x7f91b40a0f20 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:36.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b88a9640 10 monclient(hunting): authenticate will time out at 933.559566s 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b2d9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f91b413ae70 0x7f91b413d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59180/0 (socket says 172.21.15.63:59180) 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.779+0000 7f91b2d9d640 1 -- 172.21.15.63:0/4123423222 learned_addr learned my addr 172.21.15.63:0/4123423222 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3038129124 0 0) 0x7f91b409f2a0 con 0x7f91b413ae70 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient(hunting): my global_id is 4111 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91a0003530 con 0x7f91b413ae70 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2859624240 0 0) 0x7f91a0003530 con 0x7f91b413ae70 2024-09-16T05:34:36.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 >> v1:172.21.15.6:6790/0 conn(0x7f91b40a0f20 legacy=0x7f91b4139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 >> v1:172.21.15.6:6789/0 conn(0x7f91b4134880 legacy=0x7f91b4134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 monclient: found mon.noname-c 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b40a0540 con 0x7f91b413ae70 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: _finish_auth 0 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:34:36.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f91ac0019a0 con 0x7f91b413ae70 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 5 monclient: authenticate success, global_id 4111 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: dump: 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.794 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f91ac001c90 con 0x7f91b413ae70 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 1 -- 172.21.15.63:0/4123423222 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f91ac002020 con 0x7f91b413ae70 2024-09-16T05:34:36.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 4 set_mon_vals no callback set 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b1d9b640 10 monclient: dump: 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:36.796 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: shutdown 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- 172.21.15.63:0/4123423222 >> v1:172.21.15.63:6789/0 conn(0x7f91b413ae70 legacy=0x7f91b413d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- 172.21.15.63:0/4123423222 shutdown_connections 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- 172.21.15.63:0/4123423222 >> 172.21.15.63:0/4123423222 conn(0x7f91b40a37e0 msgr2=0x7f91b40a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- 172.21.15.63:0/4123423222 shutdown_connections 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- 172.21.15.63:0/4123423222 wait complete. 2024-09-16T05:34:36.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: build_initial_monmap 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: monmap: 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:36.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 Processor -- start 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- start start 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: init 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient(hunting): picked mon.noname-a con 0x7f91b40a0f20 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient(hunting): picked mon.noname-c con 0x7f91b4134880 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient(hunting): picked mon.noname-b con 0x7f91b413ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b4133e10 con 0x7f91b40a0f20 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b4139fe0 con 0x7f91b413ae70 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b40a5bd0 con 0x7f91b4134880 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:36.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient(hunting): authenticate will time out at 933.562877s 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b2d9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f91b413ae70 0x7f91b41cfbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59190/0 (socket says 172.21.15.63:59190) 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b2d9d640 1 -- 172.21.15.63:0/1920137024 learned_addr learned my addr 172.21.15.63:0/1920137024 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 100542141 0 0) 0x7f91b4139fe0 con 0x7f91b413ae70 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): my global_id is 4114 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9194003530 con 0x7f91b413ae70 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 821233402 0 0) 0x7f9194003530 con 0x7f91b413ae70 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91b4139fe0 con 0x7f91b413ae70 2024-09-16T05:34:36.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f91ac001960 con 0x7f91b413ae70 2024-09-16T05:34:36.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:36.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:36.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): dump: 2024-09-16T05:34:36.801 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:36.801 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.802 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.802 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.802 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:36.802 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:36.802 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:36.802 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 126617660 0 0) 0x7f91b4133e10 con 0x7f91b40a0f20 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient(hunting): my global_id is 4113 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9194003530 con 0x7f91b40a0f20 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2779179528 0 0) 0x7f91b4139fe0 con 0x7f91b413ae70 2024-09-16T05:34:36.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 >> v1:172.21.15.6:6790/0 conn(0x7f91b4134880 legacy=0x7f91b41cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 >> v1:172.21.15.6:6789/0 conn(0x7f91b40a0f20 legacy=0x7f91b4133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 monclient: found mon.b 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b413a890 con 0x7f91b413ae70 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 5 monclient: authenticate success, global_id 4114 2024-09-16T05:34:36.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: _renew_subs 2024-09-16T05:34:36.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:36.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f91b40742c0 con 0x7f91b413ae70 2024-09-16T05:34:36.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: _renew_subs 2024-09-16T05:34:36.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:36.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f91b413a2c0 con 0x7f91b413ae70 2024-09-16T05:34:36.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f91ac004fc0 con 0x7f91b413ae70 2024-09-16T05:34:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f91ac005170 con 0x7f91b413ae70 2024-09-16T05:34:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 10 monclient: dump: 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:36.807 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f91ac0056a0 con 0x7f91b413ae70 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.783+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 117735440 0 0) 0x7f91ac006b60 con 0x7f91b413ae70 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.787+0000 7f91b159a640 4 set_mon_vals no callback set 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.787+0000 7f91b88a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.787+0000 7f91b88a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.787+0000 7f91b88a9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.787+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9198005090 con 0x7f91b413ae70 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.787+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f91ac005af0 con 0x7f91b413ae70 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.787+0000 7f91aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.787+0000 7f91aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:36.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.855+0000 7f91b88a9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "28573ac1-022a-424e-a024-a8ae14680604", "id": 0}] 2024-09-16T05:34:36.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.855+0000 7f91b88a9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "28573ac1-022a-424e-a024-a8ae14680604", "id": 0}] 2024-09-16T05:34:36.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.855+0000 7f91b88a9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:36.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.855+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "28573ac1-022a-424e-a024-a8ae14680604", "id": 0} v 0) -- 0x7f9198005380 con 0x7f91b413ae70 2024-09-16T05:34:36.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91aaffd640 1 -- 172.21.15.63:0/1920137024 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "28573ac1-022a-424e-a024-a8ae14680604", "id": 0}]=0 v3) ==== 112+0+2 (unknown 3569444994 0 732707515) 0x7f91ac005e20 con 0x7f91b413ae70 2024-09-16T05:34:36.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "28573ac1-022a-424e-a024-a8ae14680604", "id": 0}] 2024-09-16T05:34:36.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:36.865 INFO:teuthology.orchestra.run.smithi063.stdout:0 2024-09-16T05:34:36.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91b88a9640 10 monclient: shutdown 2024-09-16T05:34:36.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91b88a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:36.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 >> v1:172.21.15.63:6789/0 conn(0x7f91b413ae70 legacy=0x7f91b41cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:36.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 shutdown_connections 2024-09-16T05:34:36.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 >> 172.21.15.63:0/1920137024 conn(0x7f91b40a37e0 msgr2=0x7f91b406dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:36.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 shutdown_connections 2024-09-16T05:34:36.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.863+0000 7f91b88a9640 1 -- 172.21.15.63:0/1920137024 wait complete. 2024-09-16T05:34:36.883 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c 1 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient: get_monmap_and_config 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient: build_initial_monmap 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient: monmap: 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:36.992949+0000 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:36.992949+0000 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:36.991 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:36.992 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:36.992 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:36.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 Processor -- start 2024-09-16T05:34:36.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- start start 2024-09-16T05:34:36.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient: init 2024-09-16T05:34:36.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:36.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:36.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad009ba40 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:36.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad00a0c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:36.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad00a1050 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:36.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad009e230 con 0x7f9ad00a1050 2024-09-16T05:34:36.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad00a14a0 con 0x7f9ad009ba40 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad009c0c0 con 0x7f9ad00a0c40 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient(hunting): authenticate will time out at 933.770097s 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ace59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f9ad009ba40 0x7f9ad009db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59194/0 (socket says 172.21.15.63:59194) 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ace59c640 1 -- 172.21.15.63:0/2249484812 learned_addr learned my addr 172.21.15.63:0/2249484812 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549429359 0 0) 0x7f9ad00a14a0 con 0x7f9ad009ba40 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient(hunting): my global_id is 4117 2024-09-16T05:34:37.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9abc004640 con 0x7f9ad009ba40 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1769547891 0 0) 0x7f9abc004640 con 0x7f9ad009ba40 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 >> v1:172.21.15.6:6790/0 conn(0x7f9ad00a0c40 legacy=0x7f9ad0137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 >> v1:172.21.15.6:6789/0 conn(0x7f9ad00a1050 legacy=0x7f9ad013b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 monclient: found mon.noname-c 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad009c980 con 0x7f9ad009ba40 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ab8001620 con 0x7f9ad009ba40 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: dump: 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 5 monclient: authenticate success, global_id 4117 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ab8001910 con 0x7f9ad009ba40 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 1 -- 172.21.15.63:0/2249484812 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ab8001d20 con 0x7f9ad009ba40 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 4 set_mon_vals no callback set 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9acd59a640 10 monclient: dump: 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.003 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 10 monclient: shutdown 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- 172.21.15.63:0/2249484812 >> v1:172.21.15.63:6789/0 conn(0x7f9ad009ba40 legacy=0x7f9ad009db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- 172.21.15.63:0/2249484812 shutdown_connections 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- 172.21.15.63:0/2249484812 >> 172.21.15.63:0/2249484812 conn(0x7f9ad00a1f00 msgr2=0x7f9ad00a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- 172.21.15.63:0/2249484812 shutdown_connections 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.991+0000 7f9ad4804640 1 -- 172.21.15.63:0/2249484812 wait complete. 2024-09-16T05:34:37.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: monmap: 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 1 Processor -- start 2024-09-16T05:34:37.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 1 -- start start 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: init 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad009ba40 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad00a0c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad00a1050 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad01a80a0 con 0x7f9ad00a1050 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad009c420 con 0x7f9ad009ba40 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad01a9830 con 0x7f9ad00a0c40 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient(hunting): authenticate will time out at 933.773070s 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ace59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f9ad009ba40 0x7f9ad01bae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59200/0 (socket says 172.21.15.63:59200) 2024-09-16T05:34:37.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ace59c640 1 -- 172.21.15.63:0/3458440420 learned_addr learned my addr 172.21.15.63:0/3458440420 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 670794731 0 0) 0x7f9ad009c420 con 0x7f9ad009ba40 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): my global_id is 4120 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab0003530 con 0x7f9ad009ba40 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 317047184 0 0) 0x7f9ab0003530 con 0x7f9ad009ba40 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ad009c420 con 0x7f9ad009ba40 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ab80015e0 con 0x7f9ad009ba40 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): dump: 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.007 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3956993129 0 0) 0x7f9ad01a80a0 con 0x7f9ad00a1050 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): my global_id is 4119 2024-09-16T05:34:37.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab0003530 con 0x7f9ad00a1050 2024-09-16T05:34:37.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107078205 0 0) 0x7f9ad01a9830 con 0x7f9ad00a0c40 2024-09-16T05:34:37.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient(hunting): my global_id is 4121 2024-09-16T05:34:37.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ad01a80a0 con 0x7f9ad00a0c40 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 412839759 0 0) 0x7f9ad009c420 con 0x7f9ad009ba40 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 >> v1:172.21.15.6:6790/0 conn(0x7f9ad00a0c40 legacy=0x7f9ad01d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 >> v1:172.21.15.6:6789/0 conn(0x7f9ad00a1050 legacy=0x7f9ad01d3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 monclient: found mon.b 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad01a7e20 con 0x7f9ad009ba40 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 5 monclient: authenticate success, global_id 4120 2024-09-16T05:34:37.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: _renew_subs 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad01a9550 con 0x7f9ad009ba40 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: _renew_subs 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ad01bb570 con 0x7f9ad009ba40 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ab8001910 con 0x7f9ad009ba40 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ab8004ef0 con 0x7f9ad009ba40 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9accd99640 4 set_mon_vals no callback set 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 10 monclient: dump: 2024-09-16T05:34:37.011 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9ab8005420 con 0x7f9ad009ba40 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 1353020935 0 0) 0x7f9ab8006830 con 0x7f9ad009ba40 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.995+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab4005090 con 0x7f9ad009ba40 2024-09-16T05:34:37.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.999+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f9ab80059a0 con 0x7f9ad009ba40 2024-09-16T05:34:37.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.999+0000 7f9ac6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:36.999+0000 7f9ac6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:37.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.067+0000 7f9ad4804640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c", "id": 1}] 2024-09-16T05:34:37.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.067+0000 7f9ad4804640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c", "id": 1}] 2024-09-16T05:34:37.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.067+0000 7f9ad4804640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.067+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c", "id": 1} v 0) -- 0x7f9ab4005380 con 0x7f9ad009ba40 2024-09-16T05:34:37.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.071+0000 7f9ac6ffd640 1 -- 172.21.15.63:0/3458440420 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c", "id": 1}]=0 v4) ==== 112+0+2 (unknown 950400234 0 940483276) 0x7f9ab8005cd0 con 0x7f9ad009ba40 2024-09-16T05:34:37.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.071+0000 7f9ac6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c", "id": 1}] 2024-09-16T05:34:37.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.071+0000 7f9ac6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:37.077 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-09-16T05:34:37.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.075+0000 7f9ad4804640 10 monclient: shutdown 2024-09-16T05:34:37.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.075+0000 7f9ad4804640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.075+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 >> v1:172.21.15.63:6789/0 conn(0x7f9ad009ba40 legacy=0x7f9ad01bae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.075+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 shutdown_connections 2024-09-16T05:34:37.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.075+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 >> 172.21.15.63:0/3458440420 conn(0x7f9ad00a1f00 msgr2=0x7f9ad009b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.075+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 shutdown_connections 2024-09-16T05:34:37.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.075+0000 7f9ad4804640 1 -- 172.21.15.63:0/3458440420 wait complete. 2024-09-16T05:34:37.091 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 89377298-ce68-4fd4-bb63-2ffd60eeea3a 2 2024-09-16T05:34:37.189 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:37.187+0000 7f650b3ec180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient: get_monmap_and_config 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient: monmap: 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:37.206176+0000 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:37.206176+0000 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.204 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.205 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.205 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:37.205 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:37.205 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 1 Processor -- start 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 1 -- start start 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient: init 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc073320 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc073700 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient(hunting): picked mon.noname-b con 0x7f1bcc13d8e0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bcc06db40 con 0x7f1bcc073700 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bcc0a3160 con 0x7f1bcc073320 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bcc073b30 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bd2964640 10 monclient(hunting): authenticate will time out at 933.983329s 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcbfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f1bcc073320 0x7f1bcc06cd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59206/0 (socket says 172.21.15.63:59206) 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcbfff640 1 -- 172.21.15.63:0/3500240300 learned_addr learned my addr 172.21.15.63:0/3500240300 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1190711344 0 0) 0x7f1bcc0a3160 con 0x7f1bcc073320 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient(hunting): my global_id is 4123 2024-09-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bb8003530 con 0x7f1bcc073320 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4272191211 0 0) 0x7f1bb8003530 con 0x7f1bcc073320 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 >> v1:172.21.15.6:6790/0 conn(0x7f1bcc13d8e0 legacy=0x7f1bcc13fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 >> v1:172.21.15.6:6789/0 conn(0x7f1bcc073700 legacy=0x7f1bcc06d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 1 monclient: found mon.noname-c 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bcc09e940 con 0x7f1bcc073320 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1bbc001620 con 0x7f1bcc073320 2024-09-16T05:34:37.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.203+0000 7f1bcaffd640 10 monclient: dump: 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 5 monclient: authenticate success, global_id 4123 2024-09-16T05:34:37.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1bbc001910 con 0x7f1bcc073320 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bcaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bcaffd640 1 -- 172.21.15.63:0/3500240300 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1bbc001ca0 con 0x7f1bcc073320 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bcaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bcaffd640 10 monclient: dump: 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.212 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 4 set_mon_vals no callback set 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient: shutdown 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- 172.21.15.63:0/3500240300 >> v1:172.21.15.63:6789/0 conn(0x7f1bcc073320 legacy=0x7f1bcc06cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- 172.21.15.63:0/3500240300 shutdown_connections 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- 172.21.15.63:0/3500240300 >> 172.21.15.63:0/3500240300 conn(0x7f1bcc0a37e0 msgr2=0x7f1bcc0a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- 172.21.15.63:0/3500240300 shutdown_connections 2024-09-16T05:34:37.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- 172.21.15.63:0/3500240300 wait complete. 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient: monmap: 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:37.214 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 Processor -- start 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- start start 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient: init 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc073320 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc073700 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient(hunting): picked mon.noname-b con 0x7f1bcc13d8e0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bcc127810 con 0x7f1bcc073700 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bcc1263c0 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bcc126640 con 0x7f1bcc073320 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd2964640 10 monclient(hunting): authenticate will time out at 933.987124s 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd0f01640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f1bcc13d8e0 0x7f1bcc1d43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59212/0 (socket says 172.21.15.63:59212) 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bd0f01640 1 -- 172.21.15.63:0/708414535 learned_addr learned my addr 172.21.15.63:0/708414535 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077236712 0 0) 0x7f1bcc1263c0 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): my global_id is 4126 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bac003530 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3888211600 0 0) 0x7f1bac003530 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bcc1263c0 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1bc4001c50 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): dump: 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.220 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452959264 0 0) 0x7f1bcc127810 con 0x7f1bcc073700 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): my global_id is 4125 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bac003530 con 0x7f1bcc073700 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2487122897 0 0) 0x7f1bcc126640 con 0x7f1bcc073320 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 10 monclient(hunting): my global_id is 4127 2024-09-16T05:34:37.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bcc127810 con 0x7f1bcc073320 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 498426502 0 0) 0x7f1bcc1263c0 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.207+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 >> v1:172.21.15.6:6790/0 conn(0x7f1bcc073320 legacy=0x7f1bcc1cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 >> v1:172.21.15.6:6789/0 conn(0x7f1bcc073700 legacy=0x7f1bcc1d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 1 monclient: found mon.b 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bcc127af0 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 5 monclient: authenticate success, global_id 4126 2024-09-16T05:34:37.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 10 monclient: _renew_subs 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1bcc09ce10 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 10 monclient: _renew_subs 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1bcc124e50 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1bc4003a20 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1bc4003d70 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bca7fc640 4 set_mon_vals no callback set 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: dump: 2024-09-16T05:34:37.223 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1bc40042a0 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 3265508557 0 0) 0x7f1bc4005b50 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb0005090 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f1bc4004590 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.211+0000 7f1bc8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:37.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.283+0000 7f1bd2964640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "89377298-ce68-4fd4-bb63-2ffd60eeea3a", "id": 2}] 2024-09-16T05:34:37.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.283+0000 7f1bd2964640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "89377298-ce68-4fd4-bb63-2ffd60eeea3a", "id": 2}] 2024-09-16T05:34:37.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.283+0000 7f1bd2964640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.283+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "89377298-ce68-4fd4-bb63-2ffd60eeea3a", "id": 2} v 0) -- 0x7f1bb0005380 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.287+0000 7f1bc8ff9640 1 -- 172.21.15.63:0/708414535 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89377298-ce68-4fd4-bb63-2ffd60eeea3a", "id": 2}]=0 v5) ==== 112+0+2 (unknown 1868329398 0 216599125) 0x7f1bc40047f0 con 0x7f1bcc13d8e0 2024-09-16T05:34:37.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.287+0000 7f1bc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "89377298-ce68-4fd4-bb63-2ffd60eeea3a", "id": 2}] 2024-09-16T05:34:37.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.287+0000 7f1bc8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:37.290 INFO:teuthology.orchestra.run.smithi063.stdout:2 2024-09-16T05:34:37.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.291+0000 7f1bd2964640 10 monclient: shutdown 2024-09-16T05:34:37.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.291+0000 7f1bd2964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.291+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 >> v1:172.21.15.63:6789/0 conn(0x7f1bcc13d8e0 legacy=0x7f1bcc1d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.291+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 shutdown_connections 2024-09-16T05:34:37.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.291+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 >> 172.21.15.63:0/708414535 conn(0x7f1bcc0a37e0 msgr2=0x7f1bcc09fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.291+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 shutdown_connections 2024-09-16T05:34:37.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.291+0000 7f1bd2964640 1 -- 172.21.15.63:0/708414535 wait complete. 2024-09-16T05:34:37.305 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 7c5624a2-d552-4295-b112-d87972274f82 3 2024-09-16T05:34:37.401 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:37.399+0000 7f650b3ec180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T05:34:37.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.419+0000 7fc690700640 10 monclient: get_monmap_and_config 2024-09-16T05:34:37.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.419+0000 7fc690700640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.419+0000 7fc690700640 10 monclient: monmap: 2024-09-16T05:34:37.422 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.422 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.422 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:37.424405+0000 2024-09-16T05:34:37.423 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:37.424405+0000 2024-09-16T05:34:37.423 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.423 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.423 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.423 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:37.423 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:37.423 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 Processor -- start 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- start start 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient: init 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient(hunting): picked mon.noname-b con 0x7fc68806bc10 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient(hunting): picked mon.noname-c con 0x7fc68806bff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient(hunting): picked mon.noname-a con 0x7fc68813cb10 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc68809fe30 con 0x7fc68813cb10 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc68806ef70 con 0x7fc68806bff0 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6880a16f0 con 0x7fc68806bc10 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient(hunting): authenticate will time out at 934.201431s 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68dc9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc68806bff0 0x7fc68806c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59226/0 (socket says 172.21.15.63:59226) 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68dc9b640 1 -- 172.21.15.63:0/852062851 learned_addr learned my addr 172.21.15.63:0/852062851 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502809707 0 0) 0x7fc68806ef70 con 0x7fc68806bff0 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient(hunting): my global_id is 4129 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc674003530 con 0x7fc68806bff0 2024-09-16T05:34:37.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1999746581 0 0) 0x7fc674003530 con 0x7fc68806bff0 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 >> v1:172.21.15.6:6790/0 conn(0x7fc68806bc10 legacy=0x7fc688136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 >> v1:172.21.15.6:6789/0 conn(0x7fc68813cb10 legacy=0x7fc68813ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 monclient: found mon.noname-c 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc688143650 con 0x7fc68806bff0 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc678001620 con 0x7fc68806bff0 2024-09-16T05:34:37.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: dump: 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 5 monclient: authenticate success, global_id 4129 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc678001910 con 0x7fc68806bff0 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 1 -- 172.21.15.63:0/852062851 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc678001ce0 con 0x7fc68806bff0 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc68d49a640 10 monclient: dump: 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.431 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.431 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.431 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 4 set_mon_vals no callback set 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient: shutdown 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- 172.21.15.63:0/852062851 >> v1:172.21.15.63:6789/0 conn(0x7fc68806bff0 legacy=0x7fc68806c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- 172.21.15.63:0/852062851 shutdown_connections 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- 172.21.15.63:0/852062851 >> 172.21.15.63:0/852062851 conn(0x7fc6880a2580 msgr2=0x7fc6880a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- 172.21.15.63:0/852062851 shutdown_connections 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 1 -- 172.21.15.63:0/852062851 wait complete. 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.423+0000 7fc690700640 10 monclient: monmap: 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.432 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-09-16T05:34:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-09-16T05:34:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:37.433 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 1 Processor -- start 2024-09-16T05:34:37.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 1 -- start start 2024-09-16T05:34:37.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: init 2024-09-16T05:34:37.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:37.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient(hunting): picked mon.noname-b con 0x7fc68806bc10 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient(hunting): picked mon.noname-a con 0x7fc68806bff0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient(hunting): picked mon.noname-c con 0x7fc68813cb10 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc688140370 con 0x7fc68806bff0 2024-09-16T05:34:37.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc68813ff20 con 0x7fc68806bc10 2024-09-16T05:34:37.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc688070690 con 0x7fc68813cb10 2024-09-16T05:34:37.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient(hunting): authenticate will time out at 934.204697s 2024-09-16T05:34:37.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc68e49c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc68806bc10 0x7fc6881334a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59234/0 (socket says 172.21.15.63:59234) 2024-09-16T05:34:37.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc68e49c640 1 -- 172.21.15.63:0/2705455083 learned_addr learned my addr 172.21.15.63:0/2705455083 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3632038745 0 0) 0x7fc6880a16f0 con 0x7fc68806bc10 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): my global_id is 4132 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc66c003530 con 0x7fc68806bc10 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1023651961 0 0) 0x7fc66c003530 con 0x7fc68806bc10 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6880a16f0 con 0x7fc68806bc10 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc684001920 con 0x7fc68806bc10 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): dump: 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467214629 0 0) 0x7fc688070690 con 0x7fc68813cb10 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): my global_id is 4133 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc66c003530 con 0x7fc68813cb10 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254167324 0 0) 0x7fc688143650 con 0x7fc68806bff0 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient(hunting): my global_id is 4131 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc688070690 con 0x7fc68806bff0 2024-09-16T05:34:37.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3581037618 0 0) 0x7fc6880a16f0 con 0x7fc68806bc10 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 >> v1:172.21.15.6:6790/0 conn(0x7fc68813cb10 legacy=0x7fc6881d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 >> v1:172.21.15.6:6789/0 conn(0x7fc68806bff0 legacy=0x7fc6881d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 monclient: found mon.b 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc688070970 con 0x7fc68806bc10 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 5 monclient: authenticate success, global_id 4132 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: _renew_subs 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc688070c50 con 0x7fc68806bc10 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: _renew_subs 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc688139e40 con 0x7fc68806bc10 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc684001d30 con 0x7fc68806bc10 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc684003af0 con 0x7fc68806bc10 2024-09-16T05:34:37.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 10 monclient: dump: 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.445 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc68cc99640 4 set_mon_vals no callback set 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc6840040a0 con 0x7fc68806bc10 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 798537775 0 0) 0x7fc684006a20 con 0x7fc68806bc10 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.427+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc670005090 con 0x7fc68806bc10 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.431+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7fc68813ff20 con 0x7fc68806bc10 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.431+0000 7fc67effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.431+0000 7fc67effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.503+0000 7fc690700640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7c5624a2-d552-4295-b112-d87972274f82", "id": 3}] 2024-09-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.503+0000 7fc690700640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7c5624a2-d552-4295-b112-d87972274f82", "id": 3}] 2024-09-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.503+0000 7fc690700640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.503+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7c5624a2-d552-4295-b112-d87972274f82", "id": 3} v 0) -- 0x7fc670005380 con 0x7fc68806bc10 2024-09-16T05:34:37.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc67effd640 1 -- 172.21.15.63:0/2705455083 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7c5624a2-d552-4295-b112-d87972274f82", "id": 3}]=0 v6) ==== 112+0+2 (unknown 974738692 0 525046306) 0x7fc684006590 con 0x7fc68806bc10 2024-09-16T05:34:37.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc67effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7c5624a2-d552-4295-b112-d87972274f82", "id": 3}] 2024-09-16T05:34:37.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc67effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:37.508 INFO:teuthology.orchestra.run.smithi063.stdout:3 2024-09-16T05:34:37.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc690700640 10 monclient: shutdown 2024-09-16T05:34:37.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc690700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 >> v1:172.21.15.63:6789/0 conn(0x7fc68806bc10 legacy=0x7fc6881334a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 shutdown_connections 2024-09-16T05:34:37.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 >> 172.21.15.63:0/2705455083 conn(0x7fc6880a2580 msgr2=0x7fc688135950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 shutdown_connections 2024-09-16T05:34:37.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.507+0000 7fc690700640 1 -- 172.21.15.63:0/2705455083 wait complete. 2024-09-16T05:34:37.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new 52fa74b0-2bf6-4c82-acb0-4032eca1f08b 4 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient: get_monmap_and_config 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient: monmap: 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:37.637936+0000 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:37.637936+0000 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.636 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.637 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.637 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:37.637 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:37.637 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 1 Processor -- start 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 1 -- start start 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient: init 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient(hunting): picked mon.noname-b con 0x7f3e280a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient(hunting): picked mon.noname-c con 0x7f3e280a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient(hunting): picked mon.noname-a con 0x7f3e280a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e280a33a0 con 0x7f3e280a2980 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e280a07a0 con 0x7f3e280a2570 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e280a2d90 con 0x7f3e280a2190 2024-09-16T05:34:37.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient(hunting): authenticate will time out at 934.415019s 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e27fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f3e280a2570 0x7f3e2813b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59246/0 (socket says 172.21.15.63:59246) 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e27fff640 1 -- 172.21.15.63:0/4250967424 learned_addr learned my addr 172.21.15.63:0/4250967424 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3900527079 0 0) 0x7f3e280a07a0 con 0x7f3e280a2570 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient(hunting): my global_id is 4135 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e10003530 con 0x7f3e280a2570 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2739883635 0 0) 0x7f3e10003530 con 0x7f3e280a2570 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 >> v1:172.21.15.6:6790/0 conn(0x7f3e280a2190 legacy=0x7f3e280a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 >> v1:172.21.15.6:6789/0 conn(0x7f3e280a2980 legacy=0x7f3e2813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 monclient: found mon.noname-c 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e280a66e0 con 0x7f3e280a2570 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3e1c001620 con 0x7f3e280a2570 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 5 monclient: authenticate success, global_id 4135 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: dump: 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:37.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e1c001880 con 0x7f3e280a2570 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 1 -- 172.21.15.63:0/4250967424 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3e1c001c50 con 0x7f3e280a2570 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e2f150640 4 set_mon_vals no callback set 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.635+0000 7f3e277fe640 10 monclient: dump: 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: shutdown 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- 172.21.15.63:0/4250967424 >> v1:172.21.15.63:6789/0 conn(0x7f3e280a2570 legacy=0x7f3e2813b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- 172.21.15.63:0/4250967424 shutdown_connections 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- 172.21.15.63:0/4250967424 >> 172.21.15.63:0/4250967424 conn(0x7f3e280a9850 msgr2=0x7f3e280a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- 172.21.15.63:0/4250967424 shutdown_connections 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- 172.21.15.63:0/4250967424 wait complete. 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: monmap: 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:37.649 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 Processor -- start 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- start start 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: init 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient(hunting): picked mon.noname-c con 0x7f3e280a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient(hunting): picked mon.noname-a con 0x7f3e280a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient(hunting): picked mon.noname-b con 0x7f3e280a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e281244f0 con 0x7f3e280a2570 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e280a0100 con 0x7f3e280a2980 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e28123f30 con 0x7f3e280a2190 2024-09-16T05:34:37.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient(hunting): authenticate will time out at 934.417922s 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2d6ed640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f3e280a2980 0x7f3e281d3ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59254/0 (socket says 172.21.15.63:59254) 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2d6ed640 1 -- 172.21.15.63:0/63756941 learned_addr learned my addr 172.21.15.63:0/63756941 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1462380008 0 0) 0x7f3e280a0100 con 0x7f3e280a2980 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): my global_id is 4138 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e14003530 con 0x7f3e280a2980 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4288526432 0 0) 0x7f3e14003530 con 0x7f3e280a2980 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e280a0100 con 0x7f3e280a2980 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3e20001c80 con 0x7f3e280a2980 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:37.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): dump: 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107975938 0 0) 0x7f3e281244f0 con 0x7f3e280a2570 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): my global_id is 4137 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e14003530 con 0x7f3e280a2570 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735568717 0 0) 0x7f3e28123f30 con 0x7f3e280a2190 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient(hunting): my global_id is 4139 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e281244f0 con 0x7f3e280a2190 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3309388660 0 0) 0x7f3e280a0100 con 0x7f3e280a2980 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 >> v1:172.21.15.6:6790/0 conn(0x7f3e280a2190 legacy=0x7f3e28120a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 >> v1:172.21.15.6:6789/0 conn(0x7f3e280a2570 legacy=0x7f3e28121180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 monclient: found mon.b 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e28124210 con 0x7f3e280a2980 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 5 monclient: authenticate success, global_id 4138 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: _renew_subs 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3e28122870 con 0x7f3e280a2980 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: _renew_subs 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3e28121890 con 0x7f3e280a2980 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e20002090 con 0x7f3e280a2980 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3e20003e70 con 0x7f3e280a2980 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 10 monclient: dump: 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e26ffd640 4 set_mon_vals no callback set 2024-09-16T05:34:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3e200043a0 con 0x7f3e280a2980 2024-09-16T05:34:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 2411846704 0 0) 0x7f3e20006e20 con 0x7f3e280a2980 2024-09-16T05:34:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.639+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e08005090 con 0x7f3e280a2980 2024-09-16T05:34:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.643+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f3e20006910 con 0x7f3e280a2980 2024-09-16T05:34:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.643+0000 7f3e257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.643+0000 7f3e257fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:37.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.711+0000 7f3e2f150640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "52fa74b0-2bf6-4c82-acb0-4032eca1f08b", "id": 4}] 2024-09-16T05:34:37.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.711+0000 7f3e2f150640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "52fa74b0-2bf6-4c82-acb0-4032eca1f08b", "id": 4}] 2024-09-16T05:34:37.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.711+0000 7f3e2f150640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.711+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "52fa74b0-2bf6-4c82-acb0-4032eca1f08b", "id": 4} v 0) -- 0x7f3e08005380 con 0x7f3e280a2980 2024-09-16T05:34:37.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e257fa640 1 -- 172.21.15.63:0/63756941 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "52fa74b0-2bf6-4c82-acb0-4032eca1f08b", "id": 4}]=0 v7) ==== 112+0+2 (unknown 2523312138 0 1697012583) 0x7f3e20006af0 con 0x7f3e280a2980 2024-09-16T05:34:37.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "52fa74b0-2bf6-4c82-acb0-4032eca1f08b", "id": 4}] 2024-09-16T05:34:37.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e257fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:37.721 INFO:teuthology.orchestra.run.smithi063.stdout:4 2024-09-16T05:34:37.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e2f150640 10 monclient: shutdown 2024-09-16T05:34:37.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e2f150640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 >> v1:172.21.15.63:6789/0 conn(0x7f3e280a2980 legacy=0x7f3e281d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 shutdown_connections 2024-09-16T05:34:37.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 >> 172.21.15.63:0/63756941 conn(0x7f3e280a9850 msgr2=0x7f3e2812fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 shutdown_connections 2024-09-16T05:34:37.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.719+0000 7f3e2f150640 1 -- 172.21.15.63:0/63756941 wait complete. 2024-09-16T05:34:37.736 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new a4dac512-ff9b-4381-9065-82bf9a711bdc 5 2024-09-16T05:34:37.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient: get_monmap_and_config 2024-09-16T05:34:37.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient: monmap: 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:37.854434+0000 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:37.854434+0000 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:37.853 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 1 Processor -- start 2024-09-16T05:34:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 1 -- start start 2024-09-16T05:34:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient: init 2024-09-16T05:34:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d8134880 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d80a0f20 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient(hunting): picked mon.noname-a con 0x7fb1d813ae70 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d80a3180 con 0x7fb1d813ae70 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d809f2a0 con 0x7fb1d8134880 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d80a0260 con 0x7fb1d80a0f20 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1e062b640 10 monclient(hunting): authenticate will time out at 934.631433s 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1de3c7640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb1d8134880 0x7fb1d8134c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59270/0 (socket says 172.21.15.63:59270) 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1de3c7640 1 -- 172.21.15.63:0/1807326534 learned_addr learned my addr 172.21.15.63:0/1807326534 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656892885 0 0) 0x7fb1d809f2a0 con 0x7fb1d8134880 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1dd3c5640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1dd3c5640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1dd3c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1dd3c5640 10 monclient(hunting): my global_id is 4141 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.851+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1c4003530 con 0x7fb1d8134880 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4161602313 0 0) 0x7fb1c4003530 con 0x7fb1d8134880 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 >> v1:172.21.15.6:6790/0 conn(0x7fb1d80a0f20 legacy=0x7fb1d8139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 >> v1:172.21.15.6:6789/0 conn(0x7fb1d813ae70 legacy=0x7fb1d813d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 1 monclient: found mon.noname-c 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d80a0540 con 0x7fb1d8134880 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d40019a0 con 0x7fb1d8134880 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: dump: 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 5 monclient: authenticate success, global_id 4141 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1d4001c90 con 0x7fb1d8134880 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 1 -- 172.21.15.63:0/1807326534 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d4002020 con 0x7fb1d8134880 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dd3c5640 10 monclient: dump: 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 4 set_mon_vals no callback set 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: shutdown 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- 172.21.15.63:0/1807326534 >> v1:172.21.15.63:6789/0 conn(0x7fb1d8134880 legacy=0x7fb1d8134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- 172.21.15.63:0/1807326534 shutdown_connections 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- 172.21.15.63:0/1807326534 >> 172.21.15.63:0/1807326534 conn(0x7fb1d80a37e0 msgr2=0x7fb1d80a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- 172.21.15.63:0/1807326534 shutdown_connections 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- 172.21.15.63:0/1807326534 wait complete. 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: build_initial_monmap 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: monmap: 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.865 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 Processor -- start 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- start start 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: init 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d80a0f20 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d8134880 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient(hunting): picked mon.noname-a con 0x7fb1d813ae70 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d81cc6e0 con 0x7fb1d813ae70 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d806ebc0 con 0x7fb1d80a0f20 2024-09-16T05:34:37.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d8139c70 con 0x7fb1d8134880 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient(hunting): authenticate will time out at 934.634588s 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1de3c7640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb1d80a0f20 0x7fb1d81cb8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59280/0 (socket says 172.21.15.63:59280) 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1de3c7640 1 -- 172.21.15.63:0/1600874808 learned_addr learned my addr 172.21.15.63:0/1600874808 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 85789789 0 0) 0x7fb1d806ebc0 con 0x7fb1d80a0f20 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient(hunting): my global_id is 4144 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1bc003530 con 0x7fb1d80a0f20 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2013020761 0 0) 0x7fb1bc003530 con 0x7fb1d80a0f20 2024-09-16T05:34:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1d806ebc0 con 0x7fb1d80a0f20 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d4001960 con 0x7fb1d80a0f20 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient(hunting): dump: 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.868 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3591124816 0 0) 0x7fb1d806ebc0 con 0x7fb1d80a0f20 2024-09-16T05:34:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 >> v1:172.21.15.6:6790/0 conn(0x7fb1d8134880 legacy=0x7fb1d81cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 >> v1:172.21.15.6:6789/0 conn(0x7fb1d813ae70 legacy=0x7fb1d81d3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 monclient: found mon.b 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d813a790 con 0x7fb1d80a0f20 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 5 monclient: authenticate success, global_id 4144 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: _renew_subs 2024-09-16T05:34:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d806e280 con 0x7fb1d80a0f20 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: _renew_subs 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1d813a0a0 con 0x7fb1d80a0f20 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1d4001c90 con 0x7fb1d80a0f20 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d4005160 con 0x7fb1d80a0f20 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1dcbc4640 4 set_mon_vals no callback set 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 10 monclient: dump: 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb1d4005690 con 0x7fb1d80a0f20 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.855+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 188946641 0 0) 0x7fb1d4006d90 con 0x7fb1d80a0f20 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.859+0000 7fb1e062b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.859+0000 7fb1e062b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.859+0000 7fb1e062b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.859+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c0005090 con 0x7fb1d80a0f20 2024-09-16T05:34:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.859+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7fb1d4006920 con 0x7fb1d80a0f20 2024-09-16T05:34:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.859+0000 7fb1ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.859+0000 7fb1ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:37.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.927+0000 7fb1e062b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a4dac512-ff9b-4381-9065-82bf9a711bdc", "id": 5}] 2024-09-16T05:34:37.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.927+0000 7fb1e062b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a4dac512-ff9b-4381-9065-82bf9a711bdc", "id": 5}] 2024-09-16T05:34:37.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.927+0000 7fb1e062b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:37.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.927+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a4dac512-ff9b-4381-9065-82bf9a711bdc", "id": 5} v 0) -- 0x7fb1c0005380 con 0x7fb1d80a0f20 2024-09-16T05:34:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.935+0000 7fb1ceffd640 1 -- 172.21.15.63:0/1600874808 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a4dac512-ff9b-4381-9065-82bf9a711bdc", "id": 5}]=0 v8) ==== 112+0+2 (unknown 3784861361 0 1988412176) 0x7fb1d4005a20 con 0x7fb1d80a0f20 2024-09-16T05:34:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.935+0000 7fb1ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a4dac512-ff9b-4381-9065-82bf9a711bdc", "id": 5}] 2024-09-16T05:34:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.935+0000 7fb1ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:37.938 INFO:teuthology.orchestra.run.smithi063.stdout:5 2024-09-16T05:34:37.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.939+0000 7fb1e062b640 10 monclient: shutdown 2024-09-16T05:34:37.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.939+0000 7fb1e062b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:37.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.939+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 >> v1:172.21.15.63:6789/0 conn(0x7fb1d80a0f20 legacy=0x7fb1d81cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:37.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.939+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 shutdown_connections 2024-09-16T05:34:37.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.939+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 >> 172.21.15.63:0/1600874808 conn(0x7fb1d80a37e0 msgr2=0x7fb1d812d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:37.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.939+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 shutdown_connections 2024-09-16T05:34:37.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-16T05:34:37.939+0000 7fb1e062b640 1 -- 172.21.15.63:0/1600874808 wait complete. 2024-09-16T05:34:37.953 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T05:34:37.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T05:34:37.956 INFO:tasks.ceph.osd.0:Started 2024-09-16T05:34:37.956 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T05:34:37.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T05:34:37.959 INFO:tasks.ceph.osd.1:Started 2024-09-16T05:34:37.959 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T05:34:37.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T05:34:37.961 INFO:tasks.ceph.osd.2:Started 2024-09-16T05:34:37.962 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T05:34:37.962 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T05:34:37.963 INFO:tasks.ceph.osd.3:Started 2024-09-16T05:34:37.964 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T05:34:37.964 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T05:34:37.966 INFO:tasks.ceph.osd.4:Started 2024-09-16T05:34:37.966 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T05:34:37.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T05:34:37.968 INFO:tasks.ceph.osd.5:Started 2024-09-16T05:34:37.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:34:37.996 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:37.995+0000 7f650b3ec180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T05:34:38.056 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:38.055+0000 7f650b3ec180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T05:34:38.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.087+0000 7f5ede99b640 10 monclient: get_monmap_and_config 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.087+0000 7f5ede99b640 10 monclient: build_initial_monmap 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.087+0000 7f5ede99b640 10 monclient: monmap: 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:38.093586+0000 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:38.093586+0000 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:38.094 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:38.095 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:38.095 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.087+0000 7f5ede99b640 1 Processor -- start 2024-09-16T05:34:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.087+0000 7f5ede99b640 1 -- start start 2024-09-16T05:34:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.087+0000 7f5ede99b640 10 monclient: init 2024-09-16T05:34:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.087+0000 7f5ede99b640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.087+0000 7f5ede99b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed80a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed80a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed80a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed80a33a0 con 0x7f5ed80a2570 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed80a07a0 con 0x7f5ed80a2980 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed80a2d90 con 0x7f5ed80a2190 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): authenticate will time out at 939.900097s 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ed80a2570 0x7f5ed813b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49534/0 (socket says 172.21.15.6:49534) 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5ed80a2190 0x7f5ed80a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:44134/0 (socket says 172.21.15.6:44134) 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed77fe640 1 -- 172.21.15.6:0/1219177204 learned_addr learned my addr 172.21.15.6:0/1219177204 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995371293 0 0) 0x7f5ed80a33a0 con 0x7f5ed80a2570 2024-09-16T05:34:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): my global_id is 4164 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ec0003530 con 0x7f5ed80a2570 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 201155671 0 0) 0x7f5ed80a2d90 con 0x7f5ed80a2190 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): my global_id is 4166 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed80a33a0 con 0x7f5ed80a2190 2024-09-16T05:34:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2561880284 0 0) 0x7f5ed80a07a0 con 0x7f5ed80a2980 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient(hunting): my global_id is 4165 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed80a2d90 con 0x7f5ed80a2980 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1582504976 0 0) 0x7f5ed80a33a0 con 0x7f5ed80a2190 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 >> v1:172.21.15.63:6789/0 conn(0x7f5ed80a2980 legacy=0x7f5ed813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 >> v1:172.21.15.6:6789/0 conn(0x7f5ed80a2570 legacy=0x7f5ed813b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 monclient: found mon.noname-b 2024-09-16T05:34:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed80a66e0 con 0x7f5ed80a2190 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec8001600 con 0x7f5ed80a2190 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: dump: 2024-09-16T05:34:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 5 monclient: authenticate success, global_id 4166 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ec80018e0 con 0x7f5ed80a2190 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 4 set_mon_vals no callback set 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 1 -- 172.21.15.6:0/1219177204 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec8001cf0 con 0x7f5ed80a2190 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed6ffd640 10 monclient: dump: 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient: shutdown 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- 172.21.15.6:0/1219177204 >> v1:172.21.15.6:6790/0 conn(0x7f5ed80a2190 legacy=0x7f5ed80a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- 172.21.15.6:0/1219177204 shutdown_connections 2024-09-16T05:34:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- 172.21.15.6:0/1219177204 >> 172.21.15.6:0/1219177204 conn(0x7f5ed80a9850 msgr2=0x7f5ed80a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- 172.21.15.6:0/1219177204 shutdown_connections 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- 172.21.15.6:0/1219177204 wait complete. 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient: build_initial_monmap 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient: monmap: 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 Processor -- start 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- start start 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient: init 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed80a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed80a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed80a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed81244c0 con 0x7f5ed80a2570 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed8121600 con 0x7f5ed80a2980 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed8122180 con 0x7f5ed80a2190 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ede99b640 10 monclient(hunting): authenticate will time out at 939.903318s 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ed80a2570 0x7f5ed8120ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49540/0 (socket says 172.21.15.6:49540) 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed77fe640 1 -- 172.21.15.6:0/3361973762 learned_addr learned my addr 172.21.15.6:0/3361973762 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678345778 0 0) 0x7f5ed8122180 con 0x7f5ed80a2190 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): my global_id is 4169 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ec4003530 con 0x7f5ed80a2190 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1011075120 0 0) 0x7f5ed81244c0 con 0x7f5ed80a2570 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): my global_id is 4167 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed8122180 con 0x7f5ed80a2570 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 476186606 0 0) 0x7f5ed8121600 con 0x7f5ed80a2980 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 10 monclient(hunting): my global_id is 4168 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.091+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed81244c0 con 0x7f5ed80a2980 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3470777237 0 0) 0x7f5ec4003530 con 0x7f5ed80a2190 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ed8121600 con 0x7f5ed80a2190 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec8001c30 con 0x7f5ed80a2190 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): dump: 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4001323782 0 0) 0x7f5ed81244c0 con 0x7f5ed80a2980 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ec4003530 con 0x7f5ed80a2980 2024-09-16T05:34:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ed0001c40 con 0x7f5ed80a2980 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): dump: 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2346513046 0 0) 0x7f5ed8122180 con 0x7f5ed80a2570 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ed81244c0 con 0x7f5ed80a2570 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ecc0011f0 con 0x7f5ed80a2570 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient(hunting): dump: 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3226373527 0 0) 0x7f5ed8121600 con 0x7f5ed80a2190 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 >> v1:172.21.15.63:6789/0 conn(0x7f5ed80a2980 legacy=0x7f5ed81d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 >> v1:172.21.15.6:6789/0 conn(0x7f5ed80a2570 legacy=0x7f5ed8120ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 monclient: found mon.c 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed812d640 con 0x7f5ed80a2190 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 5 monclient: authenticate success, global_id 4169 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 10 monclient: _renew_subs 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed80a8050 con 0x7f5ed80a2190 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 10 monclient: _renew_subs 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ed8122840 con 0x7f5ed80a2190 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ec80018c0 con 0x7f5ed80a2190 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec8004d60 con 0x7f5ed80a2190 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed67fc640 4 set_mon_vals no callback set 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: dump: 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5ec80052d0 con 0x7f5ed80a2190 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 3700216273 0 0) 0x7f5ec8006680 con 0x7f5ed80a2190 2024-09-16T05:34:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb8005090 con 0x7f5ed80a2190 2024-09-16T05:34:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f5ec8004d60 con 0x7f5ed80a2190 2024-09-16T05:34:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.095+0000 7f5ed4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:38.124 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:38.111+0000 7f650b3ec180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T05:34:38.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.155+0000 7f5ede99b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:38.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.155+0000 7f5ede99b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:38.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.155+0000 7f5ede99b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:38.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.155+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5eb8005380 con 0x7f5ed80a2190 2024-09-16T05:34:38.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.155+0000 7f5ed4ff9640 1 -- 172.21.15.6:0/3361973762 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 3765814266) 0x7f5ec8006950 con 0x7f5ed80a2190 2024-09-16T05:34:38.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.155+0000 7f5ed4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:38.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.155+0000 7f5ed4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:38.162 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:38.162 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","created":"2024-09-16T05:34:36.231874+0000","modified":"2024-09-16T05:34:37.929266+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T05:34:37.929266+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28573ac1-022a-424e-a024-a8ae14680604","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89377298-ce68-4fd4-bb63-2ffd60eeea3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7c5624a2-d552-4295-b112-d87972274f82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"52fa74b0-2bf6-4c82-acb0-4032eca1f08b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4dac512-ff9b-4381-9065-82bf9a711bdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:34:38.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.159+0000 7f5ede99b640 10 monclient: shutdown 2024-09-16T05:34:38.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.159+0000 7f5ede99b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:38.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.159+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 >> v1:172.21.15.6:6790/0 conn(0x7f5ed80a2190 legacy=0x7f5ed81207e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.159+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 shutdown_connections 2024-09-16T05:34:38.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.159+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 >> 172.21.15.6:0/3361973762 conn(0x7f5ed80a9850 msgr2=0x7f5ed812f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:38.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.159+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 shutdown_connections 2024-09-16T05:34:38.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:38.159+0000 7f5ede99b640 1 -- 172.21.15.6:0/3361973762 wait complete. 2024-09-16T05:34:38.180 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T05:34:38.180 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T05:34:38.193 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:38.191+0000 7f650b3ec180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T05:34:38.247 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:38.247+0000 7f650b3ec180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T05:34:38.330 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:38.327+0000 7f650b3ec180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T05:34:38.385 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:38.383+0000 7f650b3ec180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T05:34:38.544 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:38.543+0000 7f650b3ec180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T05:34:38.608 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:38.607+0000 7f650b3ec180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T05:34:38.784 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T05:34:38.906 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.899+0000 7fa3b0d1a640 10 monclient: get_monmap_and_config 2024-09-16T05:34:38.906 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.899+0000 7fa3b0d1a640 10 monclient: build_initial_monmap 2024-09-16T05:34:38.906 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.899+0000 7fa3b0d1a640 10 monclient: monmap: 2024-09-16T05:34:38.906 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-09-16T05:34:38.906 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.906 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:38.905917+0000 2024-09-16T05:34:38.907 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:38.905917+0000 2024-09-16T05:34:38.907 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:38.907 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.907 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:38.907 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:38.907 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:38.907 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:38.910 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.899+0000 7fa3b0d1a640 1 Processor -- start 2024-09-16T05:34:38.910 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.899+0000 7fa3b0d1a640 1 -- start start 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.899+0000 7fa3b0d1a640 10 monclient: init 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac0a38b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac0a1850 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac0a43a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ac06e690 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ac0a0c90 con 0x7fa3ac0a1850 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ac0a49a0 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient(hunting): authenticate will time out at 940.712550s 2024-09-16T05:34:38.911 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3aad9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3ac0a43a0 0x7fa3ac13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49556/0 (socket says 172.21.15.6:49556) 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3aa59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa3ac0a38b0 0x7fa3ac0a3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:44164/0 (socket says 172.21.15.6:44164) 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3aad9d640 1 -- 172.21.15.6:0/3057259039 learned_addr learned my addr 172.21.15.6:0/3057259039 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651018106 0 0) 0x7fa3ac06e690 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient(hunting): my global_id is 4170 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa398003530 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174317510 0 0) 0x7fa3ac0a49a0 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.912 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient(hunting): my global_id is 4172 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3ac06e690 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 293503229 0 0) 0x7fa398003530 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 >> v1:172.21.15.6:6790/0 conn(0x7fa3ac0a38b0 legacy=0x7fa3ac0a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 >> v1:172.21.15.63:6789/0 conn(0x7fa3ac0a1850 legacy=0x7fa3ac13bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 monclient: found mon.noname-a 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:38.913 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ac0a4c20 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: _finish_auth 0 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3a40019a0 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: dump: 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.914 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 5 monclient: authenticate success, global_id 4170 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3a4001bd0 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 1 -- 172.21.15.6:0/3057259039 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3a4001fe0 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.915 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3a9d9b640 10 monclient: dump: 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:38.916 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 4 set_mon_vals no callback set 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient: shutdown 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/3057259039 >> v1:172.21.15.6:6789/0 conn(0x7fa3ac0a43a0 legacy=0x7fa3ac13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/3057259039 shutdown_connections 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/3057259039 >> 172.21.15.6:0/3057259039 conn(0x7fa3ac0a59b0 msgr2=0x7fa3ac0a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/3057259039 shutdown_connections 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/3057259039 wait complete. 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient: build_initial_monmap 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient: monmap: 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:38.917 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 Processor -- start 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 1 -- start start 2024-09-16T05:34:38.918 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.903+0000 7fa3b0d1a640 10 monclient: init 2024-09-16T05:34:38.922 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:38.922 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:38.922 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ac0a1850 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ac0a38b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ac0a43a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ac137480 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ac136030 con 0x7fa3ac0a1850 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ac1364f0 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient(hunting): authenticate will time out at 940.716141s 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3aad9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3ac0a43a0 0x7fa3ac1d3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49558/0 (socket says 172.21.15.6:49558) 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3aad9d640 1 -- 172.21.15.6:0/1901841822 learned_addr learned my addr 172.21.15.6:0/1901841822 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2493704688 0 0) 0x7fa3ac137480 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.923 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.924 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.924 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): my global_id is 4173 2024-09-16T05:34:38.924 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa38c003530 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.924 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3052361805 0 0) 0x7fa3ac1364f0 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.924 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.925 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.925 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.925 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): my global_id is 4175 2024-09-16T05:34:38.925 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3ac137480 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.925 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2610383174 0 0) 0x7fa3ac136030 con 0x7fa3ac0a1850 2024-09-16T05:34:38.925 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): my global_id is 4174 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3ac1364f0 con 0x7fa3ac0a1850 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 769675646 0 0) 0x7fa38c003530 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3ac136030 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3a4001f60 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): dump: 2024-09-16T05:34:38.926 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:38.927 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:38.928 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1311370785 0 0) 0x7fa3ac137480 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.928 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa38c003530 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.928 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3940018c0 con 0x7fa3ac0a38b0 2024-09-16T05:34:38.928 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:38.928 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:38.928 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient(hunting): dump: 2024-09-16T05:34:38.928 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:38.928 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:38.929 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 608734561 0 0) 0x7fa3ac136030 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.930 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 >> v1:172.21.15.6:6790/0 conn(0x7fa3ac0a38b0 legacy=0x7fa3ac13f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.930 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 >> v1:172.21.15.63:6789/0 conn(0x7fa3ac0a1850 legacy=0x7fa3ac1cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.930 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:38.930 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 monclient: found mon.a 2024-09-16T05:34:38.930 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ac0a8050 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 5 monclient: authenticate success, global_id 4173 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient: _renew_subs 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ac1371a0 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient: _renew_subs 2024-09-16T05:34:38.931 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:38.932 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3ac1d4a90 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.932 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3a4001bd0 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.932 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:38.932 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3a4005120 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.932 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:38.932 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:38.932 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a959a640 4 set_mon_vals no callback set 2024-09-16T05:34:38.933 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 10 monclient: dump: 2024-09-16T05:34:38.933 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:38.933 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:38.933 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.933 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:38.933 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:38.933 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:38.933 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa3a4005650 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 3700216273 0 0) 0x7fa3a4005f30 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:38.934 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.907+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa390005090 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.935 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.919+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7fa3a4006250 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.935 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.919+0000 7fa3a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:38.935 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.919+0000 7fa3a37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:38.983 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.975+0000 7fa3b0d1a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:38.983 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.975+0000 7fa3b0d1a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:38.983 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.975+0000 7fa3b0d1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:38.983 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.975+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa390005380 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.983 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3a37fe640 1 -- 172.21.15.6:0/1901841822 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 3765814266) 0x7fa3a4005940 con 0x7fa3ac0a43a0 2024-09-16T05:34:38.983 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:38.983 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3a37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:38.984 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-16T05:34:38.984 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":8,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","created":"2024-09-16T05:34:36.231874+0000","modified":"2024-09-16T05:34:37.929266+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T05:34:37.929266+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28573ac1-022a-424e-a024-a8ae14680604","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89377298-ce68-4fd4-bb63-2ffd60eeea3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7c5624a2-d552-4295-b112-d87972274f82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"52fa74b0-2bf6-4c82-acb0-4032eca1f08b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4dac512-ff9b-4381-9065-82bf9a711bdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:34:38.984 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3b0d1a640 10 monclient: shutdown 2024-09-16T05:34:38.984 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3b0d1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:38.984 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 >> v1:172.21.15.6:6789/0 conn(0x7fa3ac0a43a0 legacy=0x7fa3ac1d3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:38.985 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 shutdown_connections 2024-09-16T05:34:38.985 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 >> 172.21.15.6:0/1901841822 conn(0x7fa3ac0a59b0 msgr2=0x7fa3ac0a6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:38.985 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 shutdown_connections 2024-09-16T05:34:38.985 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:38.979+0000 7fa3b0d1a640 1 -- 172.21.15.6:0/1901841822 wait complete. 2024-09-16T05:34:38.999 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T05:34:39.071 INFO:tasks.ceph.osd.3.smithi063.stderr:2024-09-16T05:34:39.071+0000 7f15e3f907c0 -1 Falling back to public interface 2024-09-16T05:34:39.113 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-09-16T05:34:39.107+0000 7f372c60f7c0 -1 Falling back to public interface 2024-09-16T05:34:39.124 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-16T05:34:39.119+0000 7f6d76c9f7c0 -1 Falling back to public interface 2024-09-16T05:34:39.184 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-09-16T05:34:39.179+0000 7f7fb98da7c0 -1 Falling back to public interface 2024-09-16T05:34:39.263 INFO:tasks.ceph.mgr.x.smithi063.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T05:34:39.263 INFO:tasks.ceph.mgr.x.smithi063.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T05:34:39.263 INFO:tasks.ceph.mgr.x.smithi063.stderr: from numpy import show_config as show_numpy_config 2024-09-16T05:34:39.336 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:39.335+0000 7f650b3ec180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T05:34:39.340 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-09-16T05:34:39.339+0000 7f43ee7ae7c0 -1 Falling back to public interface 2024-09-16T05:34:39.351 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-09-16T05:34:39.351+0000 7f043d1487c0 -1 Falling back to public interface 2024-09-16T05:34:39.506 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:39.503+0000 7f650b3ec180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:34:39.670 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:39.671+0000 7f650b3ec180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T05:34:39.737 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:39.735+0000 7f650b3ec180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T05:34:39.878 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:39.879+0000 7f650b3ec180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T05:34:40.504 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:40.503+0000 7f650b3ec180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T05:34:40.583 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:40.583+0000 7f650b3ec180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T05:34:40.774 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:40.771+0000 7f650b3ec180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T05:34:40.851 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:40.851+0000 7f650b3ec180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T05:34:41.157 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:41.155+0000 7f650b3ec180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T05:34:41.222 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:41.219+0000 7f650b3ec180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T05:34:42.259 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:42.259+0000 7f64b1f20640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T05:34:42.260 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-16T05:34:42.259+0000 7f64b1f20640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T05:34:43.094 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-09-16T05:34:43.087+0000 7f372c60f7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T05:34:43.109 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-16T05:34:43.103+0000 7f6d76c9f7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T05:34:43.110 INFO:tasks.ceph.osd.3.smithi063.stderr:2024-09-16T05:34:43.107+0000 7f15e3f907c0 -1 osd.3 0 log_to_monitors true 2024-09-16T05:34:43.212 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-09-16T05:34:43.207+0000 7f7fb98da7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T05:34:43.344 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-09-16T05:34:43.343+0000 7f43ee7ae7c0 -1 osd.4 0 log_to_monitors true 2024-09-16T05:34:43.349 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-09-16T05:34:43.347+0000 7f043d1487c0 -1 osd.5 0 log_to_monitors true 2024-09-16T05:34:44.337 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-09-16T05:34:44.331+0000 7f37285b8640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T05:34:44.340 INFO:tasks.ceph.osd.3.smithi063.stderr:2024-09-16T05:34:44.339+0000 7f15dff39640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T05:34:44.345 INFO:tasks.ceph.osd.3.smithi063.stderr:2024-09-16T05:34:44.343+0000 7f15dbd76640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:34:44.348 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-09-16T05:34:44.343+0000 7f37243f5640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:34:45.232 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-16T05:34:45.227+0000 7f6d72c48640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T05:34:45.236 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-16T05:34:45.231+0000 7f6d6aa7d640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:34:45.250 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-09-16T05:34:45.243+0000 7f7fb5883640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T05:34:45.254 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-09-16T05:34:45.247+0000 7f7fad6b8640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:34:45.352 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-09-16T05:34:45.351+0000 7f04390f1640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T05:34:45.359 INFO:tasks.ceph.osd.5.smithi063.stderr:2024-09-16T05:34:45.359+0000 7f0434f2e640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:34:45.603 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T05:34:45.727 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.719+0000 7f6990af9640 10 monclient: get_monmap_and_config 2024-09-16T05:34:45.727 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.719+0000 7f6990af9640 10 monclient: build_initial_monmap 2024-09-16T05:34:45.727 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.719+0000 7f6990af9640 10 monclient: monmap: 2024-09-16T05:34:45.727 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-09-16T05:34:45.727 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:45.727 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:45.726603+0000 2024-09-16T05:34:45.727 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:45.726603+0000 2024-09-16T05:34:45.727 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:45.728 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:45.728 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:45.728 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:45.728 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:45.728 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:45.736 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 Processor -- start 2024-09-16T05:34:45.737 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- start start 2024-09-16T05:34:45.737 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient: init 2024-09-16T05:34:45.737 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:45.737 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient(hunting): picked mon.noname-b con 0x7f698c0a0570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient(hunting): picked mon.noname-c con 0x7f698c0a1060 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient(hunting): picked mon.noname-a con 0x7f698c13af20 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c09e0d0 con 0x7f698c13af20 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c09ea00 con 0x7f698c0a1060 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c09ece0 con 0x7f698c0a0570 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient(hunting): authenticate will time out at 947.533270s 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698a59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f698c0a0570 0x7f698c0a0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34394/0 (socket says 172.21.15.6:34394) 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698a59c640 1 -- 172.21.15.6:0/1295255334 learned_addr learned my addr 172.21.15.6:0/1295255334 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:45.738 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 38562921 0 0) 0x7f698c09e0d0 con 0x7f698c13af20 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): my global_id is 4209 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6974003530 con 0x7f698c13af20 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 326436907 0 0) 0x7f698c09ece0 con 0x7f698c0a0570 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): my global_id is 4211 2024-09-16T05:34:45.739 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f698c09e0d0 con 0x7f698c0a0570 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413567741 0 0) 0x7f698c09ea00 con 0x7f698c0a1060 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient(hunting): my global_id is 4210 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f698c09ece0 con 0x7f698c0a1060 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3585864331 0 0) 0x7f6974003530 con 0x7f698c13af20 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 >> v1:172.21.15.6:6790/0 conn(0x7f698c0a0570 legacy=0x7f698c0a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 >> v1:172.21.15.63:6789/0 conn(0x7f698c0a1060 legacy=0x7f698c139800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: _finish_hunting 0 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 monclient: found mon.noname-a 2024-09-16T05:34:45.740 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c09b7c0 con 0x7f698c13af20 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: _finish_auth 0 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: _check_auth_tickets 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 5 monclient: authenticate success, global_id 4209 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f69840019a0 con 0x7f698c13af20 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:45.741 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: dump: 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:45.742 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6984001c00 con 0x7f698c13af20 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 4 set_mon_vals no callback set 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 1 -- 172.21.15.6:0/1295255334 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6984001fd0 con 0x7f698c13af20 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f698959a640 10 monclient: dump: 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:45.743 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient: shutdown 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- 172.21.15.6:0/1295255334 >> v1:172.21.15.6:6789/0 conn(0x7f698c13af20 legacy=0x7f698c13d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- 172.21.15.6:0/1295255334 shutdown_connections 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- 172.21.15.6:0/1295255334 >> 172.21.15.6:0/1295255334 conn(0x7f698c0a1660 msgr2=0x7f698c0a1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- 172.21.15.6:0/1295255334 shutdown_connections 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 -- 172.21.15.6:0/1295255334 wait complete. 2024-09-16T05:34:45.744 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient: build_initial_monmap 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 10 monclient: monmap: 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.723+0000 7f6990af9640 1 Processor -- start 2024-09-16T05:34:45.745 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 1 -- start start 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: init 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient(hunting): picked mon.noname-c con 0x7f698c0a0570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient(hunting): picked mon.noname-a con 0x7f698c0a1060 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient(hunting): picked mon.noname-b con 0x7f698c13af20 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c121070 con 0x7f698c0a1060 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c13a320 con 0x7f698c13af20 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c139df0 con 0x7f698c0a0570 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:45.746 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient(hunting): authenticate will time out at 947.536556s 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f698a59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f698c0a0570 0x7f698c133690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34408/0 (socket says 172.21.15.6:34408) 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f698a59c640 1 -- 172.21.15.6:0/1466018631 learned_addr learned my addr 172.21.15.6:0/1466018631 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869352432 0 0) 0x7f698c139df0 con 0x7f698c0a0570 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): my global_id is 4214 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f696c003530 con 0x7f698c0a0570 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1550303812 0 0) 0x7f698c121070 con 0x7f698c0a1060 2024-09-16T05:34:45.747 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): my global_id is 4212 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f698c139df0 con 0x7f698c0a1060 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 35410643 0 0) 0x7f698c13a320 con 0x7f698c13af20 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): my global_id is 4213 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f698c121070 con 0x7f698c13af20 2024-09-16T05:34:45.748 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3900046906 0 0) 0x7f696c003530 con 0x7f698c0a0570 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f698c13a320 con 0x7f698c0a0570 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1884196331 0 0) 0x7f698c139df0 con 0x7f698c0a1060 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f696c003530 con 0x7f698c0a1060 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f69800018c0 con 0x7f698c0a0570 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): dump: 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.749 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f697c001840 con 0x7f698c0a1060 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:45.750 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): dump: 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2223954606 0 0) 0x7f698c121070 con 0x7f698c13af20 2024-09-16T05:34:45.751 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f698c139df0 con 0x7f698c13af20 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6984001700 con 0x7f698c13af20 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient(hunting): dump: 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:45.752 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1801558370 0 0) 0x7f696c003530 con 0x7f698c0a1060 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 >> v1:172.21.15.6:6790/0 conn(0x7f698c0a0570 legacy=0x7f698c133690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 >> v1:172.21.15.63:6789/0 conn(0x7f698c13af20 legacy=0x7f698c1d3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 monclient: found mon.a 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c074140 con 0x7f698c0a1060 2024-09-16T05:34:45.753 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient: _finish_auth 0 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 5 monclient: authenticate success, global_id 4212 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: _renew_subs 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f698c120ab0 con 0x7f698c0a1060 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: _renew_subs 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f698c11f660 con 0x7f698c0a1060 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f697c003600 con 0x7f698c0a1060 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f697c003950 con 0x7f698c0a1060 2024-09-16T05:34:45.754 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6988d99640 4 set_mon_vals no callback set 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 10 monclient: dump: 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:45.755 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:45.756 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:45.756 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 4) ==== 43962+0+0 (unknown 3853214890 0 0) 0x7f697c00e520 con 0x7f698c0a1060 2024-09-16T05:34:45.756 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:45.756 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:45.756 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:45.756 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6970005090 con 0x7f698c0a1060 2024-09-16T05:34:45.756 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(11..11 src has 1..11) ==== 2487+0+0 (unknown 1372937698 0 0) 0x7f697c042710 con 0x7f698c0a1060 2024-09-16T05:34:45.756 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.727+0000 7f698a59c640 10 monclient: get_auth_request con 0x7f696c036c40 auth_method 0 2024-09-16T05:34:45.757 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.731+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f697c013eb0 con 0x7f698c0a1060 2024-09-16T05:34:45.757 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.731+0000 7f697affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:45.757 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.731+0000 7f697affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:45.905 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.899+0000 7f6990af9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:45.905 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.899+0000 7f6990af9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:45.905 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.899+0000 7f6990af9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:45.905 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.899+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6970005380 con 0x7f698c0a1060 2024-09-16T05:34:45.906 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.899+0000 7f697affd640 1 -- 172.21.15.6:0/1466018631 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6533 (unknown 238102980 0 4225086262) 0x7f697c0137b0 con 0x7f698c0a1060 2024-09-16T05:34:45.906 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.899+0000 7f697affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:45.906 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.899+0000 7f697affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:45.906 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-16T05:34:45.907 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","created":"2024-09-16T05:34:36.231874+0000","modified":"2024-09-16T05:34:45.331952+0000","last_up_change":"2024-09-16T05:34:45.331952+0000","last_in_change":"2024-09-16T05:34:37.929266+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"28573ac1-022a-424e-a024-a8ae14680604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":3069057739}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":3069057739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":3069057739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":3069057739}]},"public_addr":"172.21.15.6:6804/3069057739","cluster_addr":"172.21.15.6:6805/3069057739","heartbeat_back_addr":"172.21.15.6:6807/3069057739","heartbeat_front_addr":"172.21.15.6:6806/3069057739","state":["exists","up"]},{"osd":1,"uuid":"5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":4098222447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":4098222447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":4098222447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":4098222447}]},"public_addr":"172.21.15.6:6808/4098222447","cluster_addr":"172.21.15.6:6809/4098222447","heartbeat_back_addr":"172.21.15.6:6811/4098222447","heartbeat_front_addr":"172.21.15.6:6810/4098222447","state":["exists","up"]},{"osd":2,"uuid":"89377298-ce68-4fd4-bb63-2ffd60eeea3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":1572197727}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":1572197727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":1572197727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":1572197727}]},"public_addr":"172.21.15.6:6800/1572197727","cluster_addr":"172.21.15.6:6801/1572197727","heartbeat_back_addr":"172.21.15.6:6803/1572197727","heartbeat_front_addr":"172.21.15.6:6802/1572197727","state":["exists","up"]},{"osd":3,"uuid":"7c5624a2-d552-4295-b112-d87972274f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6800","nonce":709309215}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":709309215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":709309215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":709309215}]},"public_addr":"172.21.15.63:6800/709309215","cluster_addr":"172.21.15.63:6801/709309215","heartbeat_back_addr":"172.21.15.63:6803/709309215","heartbeat_front_addr":"172.21.15.63:6802/709309215","state":["exists","up"]},{"osd":4,"uuid":"52fa74b0-2bf6-4c82-acb0-4032eca1f08b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a4dac512-ff9b-4381-9065-82bf9a711bdc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:34:45.909 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.903+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 >> v1:172.21.15.63:6812/3643207211 conn(0x7f696c036c40 legacy=0x7f696c039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:45.909 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.903+0000 7f6990af9640 10 monclient: shutdown 2024-09-16T05:34:45.909 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.903+0000 7f6990af9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:45.909 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.903+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 >> v1:172.21.15.6:6789/0 conn(0x7f698c0a1060 legacy=0x7f698c1d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:45.909 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.903+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 shutdown_connections 2024-09-16T05:34:45.909 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.903+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 >> 172.21.15.6:0/1466018631 conn(0x7f698c0a1660 msgr2=0x7f698c0a38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:45.909 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.903+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 shutdown_connections 2024-09-16T05:34:45.909 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:45.903+0000 7f6990af9640 1 -- 172.21.15.6:0/1466018631 wait complete. 2024-09-16T05:34:45.924 DEBUG:teuthology.misc:4 of 6 OSDs are up 2024-09-16T05:34:46.031 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-09-16T05:34:46.031+0000 7f43eaf69640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T05:34:46.039 INFO:tasks.ceph.osd.4.smithi063.stderr:2024-09-16T05:34:46.039+0000 7f43e258c640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T05:34:52.530 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T05:34:52.654 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.647+0000 7fe6481ff640 10 monclient: get_monmap_and_config 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.647+0000 7fe6481ff640 10 monclient: build_initial_monmap 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.647+0000 7fe6481ff640 10 monclient: monmap: 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:52.654319+0000 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:52.654319+0000 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:52.655 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:52.662 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 Processor -- start 2024-09-16T05:34:52.662 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- start start 2024-09-16T05:34:52.662 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient: init 2024-09-16T05:34:52.662 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:52.662 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:52.662 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient(hunting): picked mon.noname-a con 0x7fe6400a38b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:52.662 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient(hunting): picked mon.noname-b con 0x7fe6400a1850 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:52.662 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient(hunting): picked mon.noname-c con 0x7fe6400a43a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64006e690 con 0x7fe6400a38b0 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6400a0c90 con 0x7fe6400a43a0 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6400a49a0 con 0x7fe6400a1850 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient(hunting): authenticate will time out at 954.461066s 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe64579a640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe6400a1850 0x7fe64013bb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33532/0 (socket says 172.21.15.6:33532) 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe645f9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe6400a38b0 0x7fe6400a3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58436/0 (socket says 172.21.15.6:58436) 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe64579a640 1 -- 172.21.15.6:0/4103546666 learned_addr learned my addr 172.21.15.6:0/4103546666 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998926381 0 0) 0x7fe64006e690 con 0x7fe6400a38b0 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.663 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient(hunting): my global_id is 4224 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe62c003530 con 0x7fe6400a38b0 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359630430 0 0) 0x7fe6400a49a0 con 0x7fe6400a1850 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient(hunting): my global_id is 4226 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64006e690 con 0x7fe6400a1850 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3424063943 0 0) 0x7fe64006e690 con 0x7fe6400a1850 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 >> v1:172.21.15.63:6789/0 conn(0x7fe6400a43a0 legacy=0x7fe64013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 >> v1:172.21.15.6:6789/0 conn(0x7fe6400a38b0 legacy=0x7fe6400a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.664 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: _finish_hunting 0 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 monclient: found mon.noname-b 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6400a4c20 con 0x7fe6400a1850 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: _finish_auth 0 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: _check_auth_tickets 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe630001640 con 0x7fe6400a1850 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: dump: 2024-09-16T05:34:52.665 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 5 monclient: authenticate success, global_id 4226 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6300018a0 con 0x7fe6400a1850 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:52.666 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 1 -- 172.21.15.6:0/4103546666 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe630001cb0 con 0x7fe6400a1850 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 4 set_mon_vals no callback set 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe644f99640 10 monclient: dump: 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.667 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient: shutdown 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- 172.21.15.6:0/4103546666 >> v1:172.21.15.6:6790/0 conn(0x7fe6400a1850 legacy=0x7fe64013bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- 172.21.15.6:0/4103546666 shutdown_connections 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- 172.21.15.6:0/4103546666 >> 172.21.15.6:0/4103546666 conn(0x7fe6400a59b0 msgr2=0x7fe6400a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- 172.21.15.6:0/4103546666 shutdown_connections 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- 172.21.15.6:0/4103546666 wait complete. 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient: build_initial_monmap 2024-09-16T05:34:52.668 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient: monmap: 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 Processor -- start 2024-09-16T05:34:52.669 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 1 -- start start 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.651+0000 7fe6481ff640 10 monclient: init 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient(hunting): picked mon.noname-c con 0x7fe6400a1850 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient(hunting): picked mon.noname-b con 0x7fe6400a38b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient(hunting): picked mon.noname-a con 0x7fe6400a43a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe640135ef0 con 0x7fe6400a43a0 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6400a00a0 con 0x7fe6400a38b0 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe640133860 con 0x7fe6400a1850 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient(hunting): authenticate will time out at 954.464164s 2024-09-16T05:34:52.670 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe645f9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe6400a1850 0x7fe640131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33538/0 (socket says 172.21.15.6:33538) 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe645f9b640 1 -- 172.21.15.6:0/3662895273 learned_addr learned my addr 172.21.15.6:0/3662895273 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006848401 0 0) 0x7fe640133860 con 0x7fe6400a1850 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): my global_id is 4229 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe624003530 con 0x7fe6400a1850 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1202403409 0 0) 0x7fe640135ef0 con 0x7fe6400a43a0 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): my global_id is 4227 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe640133860 con 0x7fe6400a43a0 2024-09-16T05:34:52.671 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3399033923 0 0) 0x7fe624003530 con 0x7fe6400a1850 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe640135ef0 con 0x7fe6400a1850 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe63c001a40 con 0x7fe6400a1850 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): dump: 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.672 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2582719320 0 0) 0x7fe640133860 con 0x7fe6400a43a0 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe624003530 con 0x7fe6400a43a0 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe638001a70 con 0x7fe6400a43a0 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): dump: 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.673 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522519892 0 0) 0x7fe6400a4c20 con 0x7fe6400a38b0 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient(hunting): my global_id is 4228 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe640133860 con 0x7fe6400a38b0 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2709278428 0 0) 0x7fe640135ef0 con 0x7fe6400a1850 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 >> v1:172.21.15.63:6789/0 conn(0x7fe6400a38b0 legacy=0x7fe640132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.674 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 >> v1:172.21.15.6:6789/0 conn(0x7fe6400a43a0 legacy=0x7fe6401d3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient: _finish_hunting 0 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 monclient: found mon.c 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6400a7f30 con 0x7fe6400a1850 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient: _finish_auth 0 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient: _check_auth_tickets 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 5 monclient: authenticate success, global_id 4229 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient: _renew_subs 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe640132d10 con 0x7fe6400a1850 2024-09-16T05:34:52.675 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient: _renew_subs 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe640135300 con 0x7fe6400a1850 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe63c001cb0 con 0x7fe6400a1850 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe63c004c40 con 0x7fe6400a1850 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 10 monclient: dump: 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-09-16T05:34:52.676 INFO:teuthology.misc.health.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.677 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.677 INFO:teuthology.misc.health.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.677 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.677 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.677 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.677 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.677 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr: 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe637fff640 4 set_mon_vals no callback set 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe63c00f9e0 con 0x7fe6400a1850 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe628005090 con 0x7fe6400a1850 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 1629749375 0 0) 0x7fe63c044080 con 0x7fe6400a1850 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.655+0000 7fe64579a640 10 monclient: get_auth_request con 0x7fe624036c40 auth_method 0 2024-09-16T05:34:52.678 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.659+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe63c015560 con 0x7fe6400a1850 2024-09-16T05:34:52.679 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.659+0000 7fe6367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:52.679 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.659+0000 7fe6367fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:52.830 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.823+0000 7fe6481ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:52.830 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.823+0000 7fe6481ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:52.830 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.823+0000 7fe6481ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:52.830 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.823+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe628005380 con 0x7fe6400a1850 2024-09-16T05:34:52.831 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6367fc640 1 -- 172.21.15.6:0/3662895273 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8985 (unknown 4221056075 0 3028220996) 0x7fe63c014e60 con 0x7fe6400a1850 2024-09-16T05:34:52.831 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:52.831 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6367fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:52.832 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-16T05:34:52.832 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":15,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","created":"2024-09-16T05:34:36.231874+0000","modified":"2024-09-16T05:34:49.378734+0000","last_up_change":"2024-09-16T05:34:46.347254+0000","last_in_change":"2024-09-16T05:34:37.929266+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:34:47.340654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28573ac1-022a-424e-a024-a8ae14680604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":3069057739}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":3069057739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":3069057739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":3069057739}]},"public_addr":"172.21.15.6:6804/3069057739","cluster_addr":"172.21.15.6:6805/3069057739","heartbeat_back_addr":"172.21.15.6:6807/3069057739","heartbeat_front_addr":"172.21.15.6:6806/3069057739","state":["exists","up"]},{"osd":1,"uuid":"5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":4098222447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":4098222447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":4098222447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":4098222447}]},"public_addr":"172.21.15.6:6808/4098222447","cluster_addr":"172.21.15.6:6809/4098222447","heartbeat_back_addr":"172.21.15.6:6811/4098222447","heartbeat_front_addr":"172.21.15.6:6810/4098222447","state":["exists","up"]},{"osd":2,"uuid":"89377298-ce68-4fd4-bb63-2ffd60eeea3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":1572197727}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":1572197727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":1572197727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":1572197727}]},"public_addr":"172.21.15.6:6800/1572197727","cluster_addr":"172.21.15.6:6801/1572197727","heartbeat_back_addr":"172.21.15.6:6803/1572197727","heartbeat_front_addr":"172.21.15.6:6802/1572197727","state":["exists","up"]},{"osd":3,"uuid":"7c5624a2-d552-4295-b112-d87972274f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6800","nonce":709309215}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":709309215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":709309215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":709309215}]},"public_addr":"172.21.15.63:6800/709309215","cluster_addr":"172.21.15.63:6801/709309215","heartbeat_back_addr":"172.21.15.63:6803/709309215","heartbeat_front_addr":"172.21.15.63:6802/709309215","state":["exists","up"]},{"osd":4,"uuid":"52fa74b0-2bf6-4c82-acb0-4032eca1f08b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":118134224}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":118134224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":118134224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":118134224}]},"public_addr":"172.21.15.63:6804/118134224","cluster_addr":"172.21.15.63:6805/118134224","heartbeat_back_addr":"172.21.15.63:6807/118134224","heartbeat_front_addr":"172.21.15.63:6806/118134224","state":["exists","up"]},{"osd":5,"uuid":"a4dac512-ff9b-4381-9065-82bf9a711bdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":165284194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":165284194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":165284194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":165284194}]},"public_addr":"172.21.15.63:6808/165284194","cluster_addr":"172.21.15.63:6809/165284194","heartbeat_back_addr":"172.21.15.63:6811/165284194","heartbeat_front_addr":"172.21.15.63:6810/165284194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.065166+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.182281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.088394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.071475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.333358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.397991+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:34:52.834 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe624036c40 legacy=0x7fe624039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.834 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6481ff640 10 monclient: shutdown 2024-09-16T05:34:52.834 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6481ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:52.834 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 >> v1:172.21.15.6:6790/0 conn(0x7fe6400a1850 legacy=0x7fe640131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.834 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 shutdown_connections 2024-09-16T05:34:52.834 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 >> 172.21.15.6:0/3662895273 conn(0x7fe6400a59b0 msgr2=0x7fe6400a6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:52.834 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 shutdown_connections 2024-09-16T05:34:52.835 INFO:teuthology.misc.health.smithi006.stderr:2024-09-16T05:34:52.827+0000 7fe6481ff640 1 -- 172.21.15.6:0/3662895273 wait complete. 2024-09-16T05:34:52.848 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T05:34:52.848 INFO:tasks.ceph:Creating RBD pool 2024-09-16T05:34:52.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T05:34:52.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.967+0000 7f14eb1e8640 10 monclient: get_monmap_and_config 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.967+0000 7f14eb1e8640 10 monclient: build_initial_monmap 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.967+0000 7f14eb1e8640 10 monclient: monmap: 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:52.974393+0000 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:52.974393+0000 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:52.975 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:52.976 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:52.976 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:52.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 Processor -- start 2024-09-16T05:34:52.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- start start 2024-09-16T05:34:52.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient: init 2024-09-16T05:34:52.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:52.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:52.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f14e409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f14e409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f14e40a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e40a1370 con 0x7f14e409e230 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e409ead0 con 0x7f14e40a0da0 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e409edb0 con 0x7f14e409aa30 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient(hunting): authenticate will time out at 954.781163s 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e8f84640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f14e409aa30 0x7f14e409ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33550/0 (socket says 172.21.15.6:33550) 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e8f84640 1 -- 172.21.15.6:0/2346317123 learned_addr learned my addr 172.21.15.6:0/2346317123 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408597511 0 0) 0x7f14e409edb0 con 0x7f14e409aa30 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): my global_id is 4232 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14cc003530 con 0x7f14e409aa30 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3845490402 0 0) 0x7f14e40a1370 con 0x7f14e409e230 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): my global_id is 4230 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14e409edb0 con 0x7f14e409e230 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 429598757 0 0) 0x7f14e409ead0 con 0x7f14e40a0da0 2024-09-16T05:34:52.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient(hunting): my global_id is 4231 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14e40a1370 con 0x7f14e40a0da0 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3327279131 0 0) 0x7f14e409edb0 con 0x7f14e409e230 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 >> v1:172.21.15.6:6790/0 conn(0x7f14e409aa30 legacy=0x7f14e409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 >> v1:172.21.15.63:6789/0 conn(0x7f14e40a0da0 legacy=0x7f14e413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 monclient: found mon.noname-a 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:52.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e409dd50 con 0x7f14e409e230 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14d8001580 con 0x7f14e409e230 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: dump: 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.987 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 5 monclient: authenticate success, global_id 4230 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14d8001870 con 0x7f14e409e230 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 1 -- 172.21.15.6:0/2346317123 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14d8001c40 con 0x7f14e409e230 2024-09-16T05:34:52.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14e37fe640 10 monclient: dump: 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.989 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 4 set_mon_vals no callback set 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient: shutdown 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2346317123 >> v1:172.21.15.6:6789/0 conn(0x7f14e409e230 legacy=0x7f14e41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2346317123 shutdown_connections 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2346317123 >> 172.21.15.6:0/2346317123 conn(0x7f14e40a1650 msgr2=0x7f14e40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2346317123 shutdown_connections 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2346317123 wait complete. 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient: build_initial_monmap 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient: monmap: 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:52.990 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 Processor -- start 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 1 -- start start 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.971+0000 7f14eb1e8640 10 monclient: init 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:52.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f14e409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f14e409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f14e40a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e411e7f0 con 0x7f14e40a0da0 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e41cc4f0 con 0x7f14e409e230 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e413dac0 con 0x7f14e409aa30 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient(hunting): authenticate will time out at 954.784482s 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e8f84640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f14e409aa30 0x7f14e41cb6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33554/0 (socket says 172.21.15.6:33554) 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e9785640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f14e40a0da0 0x7f14e41d3930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58470/0 (socket says 172.21.15.6:58470) 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e8f84640 1 -- 172.21.15.6:0/2496463645 learned_addr learned my addr 172.21.15.6:0/2496463645 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2586800775 0 0) 0x7f14e413dac0 con 0x7f14e409aa30 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): my global_id is 4235 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14d0003530 con 0x7f14e409aa30 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956098429 0 0) 0x7f14e411e7f0 con 0x7f14e40a0da0 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): my global_id is 4233 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14e413dac0 con 0x7f14e40a0da0 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2513388946 0 0) 0x7f14e413dac0 con 0x7f14e40a0da0 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14e411e7f0 con 0x7f14e40a0da0 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14dc001bc0 con 0x7f14e40a0da0 2024-09-16T05:34:52.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): dump: 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:52.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3549598272 0 0) 0x7f14d0003530 con 0x7f14e409aa30 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14e413dac0 con 0x7f14e409aa30 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14d4001860 con 0x7f14e409aa30 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): dump: 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.995 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 193031857 0 0) 0x7f14e41cc4f0 con 0x7f14e409e230 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient(hunting): my global_id is 4234 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14d0003530 con 0x7f14e409e230 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1287439688 0 0) 0x7f14e411e7f0 con 0x7f14e40a0da0 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 >> v1:172.21.15.6:6790/0 conn(0x7f14e409aa30 legacy=0x7f14e41cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 >> v1:172.21.15.63:6789/0 conn(0x7f14e409e230 legacy=0x7f14e41cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:52.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient: _finish_hunting 0 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 monclient: found mon.a 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e406ea40 con 0x7f14e40a0da0 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient: _finish_auth 0 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient: _check_auth_tickets 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 5 monclient: authenticate success, global_id 4233 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient: _renew_subs 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:52.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14e40740c0 con 0x7f14e40a0da0 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient: _renew_subs 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f14e411d0c0 con 0x7f14e40a0da0 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14dc003980 con 0x7f14e40a0da0 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14dc003cd0 con 0x7f14e40a0da0 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e2ffd640 4 set_mon_vals no callback set 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 10 monclient: dump: 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:52.998 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c4005090 con 0x7f14e40a0da0 2024-09-16T05:34:52.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f14dc003ef0 con 0x7f14e40a0da0 2024-09-16T05:34:53.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 1629749375 0 0) 0x7f14dc0442d0 con 0x7f14e40a0da0 2024-09-16T05:34:53.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.975+0000 7f14e8f84640 10 monclient: get_auth_request con 0x7f14d0036c70 auth_method 0 2024-09-16T05:34:53.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.979+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f14dc015750 con 0x7f14e40a0da0 2024-09-16T05:34:53.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.979+0000 7f14e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:53.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:52.979+0000 7f14e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:53.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.147+0000 7f14eb1e8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T05:34:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.147+0000 7f14eb1e8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T05:34:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.147+0000 7f14eb1e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:53.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.147+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f14c4005b90 con 0x7f14e40a0da0 2024-09-16T05:34:53.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.887+0000 7f14e17fa640 1 -- 172.21.15.6:0/2496463645 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (unknown 231814880 0 0) 0x7f14dc015050 con 0x7f14e40a0da0 2024-09-16T05:34:53.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.887+0000 7f14e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T05:34:53.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.887+0000 7f14e17fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T05:34:53.894 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2024-09-16T05:34:53.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.891+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 >> v1:172.21.15.63:6812/3643207211 conn(0x7f14d0036c70 legacy=0x7f14d0039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:53.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.891+0000 7f14eb1e8640 10 monclient: shutdown 2024-09-16T05:34:53.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.891+0000 7f14eb1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:53.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.891+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 >> v1:172.21.15.6:6789/0 conn(0x7f14e40a0da0 legacy=0x7f14e41d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:53.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.891+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 shutdown_connections 2024-09-16T05:34:53.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.891+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 >> 172.21.15.6:0/2496463645 conn(0x7f14e40a1650 msgr2=0x7f14e406ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:53.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.891+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 shutdown_connections 2024-09-16T05:34:53.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.891+0000 7f14eb1e8640 1 -- 172.21.15.6:0/2496463645 wait complete. 2024-09-16T05:34:53.918 DEBUG:teuthology.orchestra.run.smithi006:> rbd --cluster ceph pool init rbd 2024-09-16T05:34:53.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient: get_monmap_and_config 2024-09-16T05:34:53.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient: build_initial_monmap 2024-09-16T05:34:53.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient: monmap: 2024-09-16T05:34:53.972 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:53.971870+0000 2024-09-16T05:34:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:53.971870+0000 2024-09-16T05:34:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:53.974 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 1 Processor -- start 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 1 -- start start 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient: init 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient: _reopen_session rank -1 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient(hunting): picked mon.noname-c con 0x563667e09a60 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient(hunting): picked mon.noname-a con 0x563667e9f240 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient(hunting): picked mon.noname-b con 0x563667ea57c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x563667c8bea0 con 0x563667e9f240 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x563667c72180 con 0x563667e09a60 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x563667c5c990 con 0x563667ea57c0 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient(hunting): _renew_subs 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient(hunting): authenticate will time out at 955.778649s 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cb07640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x563667ea57c0 0x563667ea7bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33570/0 (socket says 172.21.15.6:33570) 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cb07640 1 -- 172.21.15.6:0/2342062015 learned_addr learned my addr 172.21.15.6:0/2342062015 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544692831 0 0) 0x563667c5c990 con 0x563667ea57c0 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient(hunting): _negotiate 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient(hunting): my global_id is 4238 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5058001e20 con 0x563667ea57c0 2024-09-16T05:34:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824183889 0 0) 0x563667c8bea0 con 0x563667e9f240 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient(hunting): _negotiate 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient(hunting): my global_id is 4236 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x563667c5c990 con 0x563667e9f240 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 92319609 0 0) 0x7f5058001e20 con 0x563667ea57c0 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 >> v1:172.21.15.63:6789/0 conn(0x563667e09a60 legacy=0x563667e09e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 >> v1:172.21.15.6:6789/0 conn(0x563667e9f240 legacy=0x563667ea40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: _finish_hunting 0 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 monclient: found mon.noname-b 2024-09-16T05:34:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:53.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:53.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x563667c980d0 con 0x563667ea57c0 2024-09-16T05:34:53.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: _finish_auth 0 2024-09-16T05:34:53.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: _check_auth_tickets 2024-09-16T05:34:53.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:53.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f505c0019a0 con 0x563667ea57c0 2024-09-16T05:34:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 5 monclient: authenticate success, global_id 4238 2024-09-16T05:34:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:53.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: dump: 2024-09-16T05:34:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:53.992 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:53.992 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:53.992 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:53.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:53.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f505c001c40 con 0x563667ea57c0 2024-09-16T05:34:53.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:53.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 1 -- 172.21.15.6:0/2342062015 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f505c002050 con 0x563667ea57c0 2024-09-16T05:34:53.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 10 monclient: get_monmap_and_config success 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506cd91f80 4 set_mon_vals no callback set 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.967+0000 7f506a8a3640 10 monclient: dump: 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:53.993 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: shutdown 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- 172.21.15.6:0/2342062015 >> v1:172.21.15.6:6790/0 conn(0x563667ea57c0 legacy=0x563667ea7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- 172.21.15.6:0/2342062015 shutdown_connections 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- 172.21.15.6:0/2342062015 >> 172.21.15.6:0/2342062015 conn(0x563667dee510 msgr2=0x563667dee8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- 172.21.15.6:0/2342062015 shutdown_connections 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- 172.21.15.6:0/2342062015 wait complete. 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: build_initial_monmap 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: monmap: 2024-09-16T05:34:53.994 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 Processor -- start 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- start start 2024-09-16T05:34:53.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: init 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: _reopen_session rank -1 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient(hunting): picked mon.noname-c con 0x563667e09a60 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient(hunting): picked mon.noname-b con 0x563667e9f240 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient(hunting): picked mon.noname-a con 0x563667ea57c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x563667df0db0 con 0x563667ea57c0 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x563667ea35c0 con 0x563667e9f240 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x563667ea0c90 con 0x563667e09a60 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient(hunting): _renew_subs 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient(hunting): authenticate will time out at 955.782516s 2024-09-16T05:34:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506b8a5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x563667e09a60 0x563667e9fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33586/0 (socket says 172.21.15.6:33586) 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506b8a5640 1 -- 172.21.15.6:0/3973183555 learned_addr learned my addr 172.21.15.6:0/3973183555 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688111754 0 0) 0x563667ea0c90 con 0x563667e09a60 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): my global_id is 4241 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f504c001e20 con 0x563667e09a60 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804811761 0 0) 0x563667c980d0 con 0x563667ea57c0 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:53.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): my global_id is 4239 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x563667ea0c90 con 0x563667ea57c0 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1253632581 0 0) 0x7f504c001e20 con 0x563667e09a60 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x563667c980d0 con 0x563667e09a60 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f50600019d0 con 0x563667e09a60 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): dump: 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:53.998 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422885526 0 0) 0x563667ea35c0 con 0x563667e9f240 2024-09-16T05:34:53.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): my global_id is 4240 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f504c001e20 con 0x563667e9f240 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1631234047 0 0) 0x563667ea0c90 con 0x563667ea57c0 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x563667ea35c0 con 0x563667ea57c0 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f505c001df0 con 0x563667ea57c0 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:54.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient(hunting): dump: 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:54.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 346690647 0 0) 0x563667c980d0 con 0x563667e09a60 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 >> v1:172.21.15.63:6789/0 conn(0x563667e9f240 legacy=0x563667ea0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 >> v1:172.21.15.6:6789/0 conn(0x563667ea57c0 legacy=0x563667eb5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient: _finish_hunting 0 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 monclient: found mon.c 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x563667ea5490 con 0x563667e09a60 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient: _finish_auth 0 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient: _check_auth_tickets 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 5 monclient: authenticate success, global_id 4241 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: _renew_subs 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x563667ea2950 con 0x563667e09a60 2024-09-16T05:34:54.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: _renew_subs 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5060001cb0 con 0x563667e09a60 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x563667e08bd0 con 0x563667e09a60 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5060002360 con 0x563667e09a60 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f5053fff640 10 monclient: dump: 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:54.003 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.971+0000 7f506a0a2640 4 set_mon_vals no callback set 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.975+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5060002580 con 0x563667e09a60 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.975+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 3495+0+0 (unknown 41665978 0 0) 0x7f5060042aa0 con 0x563667e09a60 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.975+0000 7f506cd91f80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.975+0000 7f506cd91f80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.975+0000 7f506cd91f80 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.975+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x563667eb7390 con 0x563667e09a60 2024-09-16T05:34:54.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:53.975+0000 7f506b0a4640 10 monclient: get_auth_request con 0x7f504c0356d0 auth_method 0 2024-09-16T05:34:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.895+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (unknown 1851909647 0 0) 0x7f5060013e30 con 0x563667e09a60 2024-09-16T05:34:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.895+0000 7f5053fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T05:34:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.895+0000 7f5053fff640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T05:34:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.895+0000 7f506cd91f80 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:54.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.895+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f505c001c40 con 0x563667e09a60 2024-09-16T05:34:54.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.907+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f50600130f0 con 0x563667e09a60 2024-09-16T05:34:54.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.907+0000 7f5053fff640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-16T05:34:54.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.907+0000 7f506a0a2640 10 monclient: _renew_subs 2024-09-16T05:34:54.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.907+0000 7f506a0a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:54.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.907+0000 7f506a0a2640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=17}) -- 0x7f5044002af0 con 0x563667e09a60 2024-09-16T05:34:54.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.907+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 11 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (unknown 4189429440 0 0) 0x7f5060041980 con 0x563667e09a60 2024-09-16T05:34:54.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.907+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.63:6808/165284194 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x563667ebc190 con 0x563667eb8680 2024-09-16T05:34:54.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.907+0000 7f506cb07640 10 monclient: get_auth_request con 0x563667eb8680 auth_method 0 2024-09-16T05:34:54.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.915+0000 7f506cb07640 1 -- 172.21.15.6:0/3973183555 <== osd.5 v1:172.21.15.63:6808/165284194 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 1703302865 0 0) 0x7f505c0046d0 con 0x563667eb8680 2024-09-16T05:34:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.915+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.63:6808/165284194 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x563667f0dc00 con 0x563667eb8680 2024-09-16T05:34:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.919+0000 7f506cb07640 1 -- 172.21.15.6:0/3973183555 <== osd.5 v1:172.21.15.63:6808/165284194 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4274215525 0 0) 0x7f505c0046d0 con 0x563667eb8680 2024-09-16T05:34:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.919+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.63:6808/165284194 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x563667f0cd10 con 0x563667eb8680 2024-09-16T05:34:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.919+0000 7f506cb07640 1 -- 172.21.15.6:0/3973183555 <== osd.5 v1:172.21.15.63:6808/165284194 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4287864937 0 0) 0x7f505c0046d0 con 0x563667eb8680 2024-09-16T05:34:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.919+0000 7f50698a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:54.919+0000 7f50698a1640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f5048001f10 con 0x563667e09a60 2024-09-16T05:34:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.915+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 12 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (unknown 3204592382 0 0) 0x7f5060013730 con 0x563667e09a60 2024-09-16T05:34:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.915+0000 7f5053fff640 10 monclient: _renew_subs 2024-09-16T05:34:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.915+0000 7f5053fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.915+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=18}) -- 0x563667c980d0 con 0x563667e09a60 2024-09-16T05:34:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.919+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 13 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (unknown 1978170183 0 0) 0x7f506001be50 con 0x563667e09a60 2024-09-16T05:34:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.919+0000 7f506a0a2640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.63:6808/165284194 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f50440056d0 con 0x563667eb8680 2024-09-16T05:34:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.923+0000 7f506cb07640 1 -- 172.21.15.6:0/3973183555 <== osd.5 v1:172.21.15.63:6808/165284194 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 2910594948 0 0) 0x7f505c0046d0 con 0x563667eb8680 2024-09-16T05:34:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.923+0000 7f50698a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:55.923+0000 7f50698a1640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7f5048003330 con 0x563667e09a60 2024-09-16T05:34:56.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.923+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 14 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (unknown 528410013 0 0) 0x7f506001c2e0 con 0x563667e09a60 2024-09-16T05:34:56.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.923+0000 7f5053fff640 10 monclient: _renew_subs 2024-09-16T05:34:56.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.923+0000 7f5053fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:56.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.923+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=19}) -- 0x563667ea0c90 con 0x563667e09a60 2024-09-16T05:34:56.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.923+0000 7f5053fff640 1 -- 172.21.15.6:0/3973183555 <== mon.2 v1:172.21.15.6:6790/0 15 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (unknown 3251434369 0 0) 0x7f506001c050 con 0x563667e09a60 2024-09-16T05:34:56.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.927+0000 7f506a0a2640 1 -- 172.21.15.6:0/3973183555 --> v1:172.21.15.63:6808/165284194 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f5044005ab0 con 0x563667eb8680 2024-09-16T05:34:56.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.931+0000 7f506cb07640 1 -- 172.21.15.6:0/3973183555 <== osd.5 v1:172.21.15.63:6808/165284194 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 900577457 0 0) 0x7f505c0046d0 con 0x563667eb8680 2024-09-16T05:34:56.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.931+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 >> v1:172.21.15.63:6808/165284194 conn(0x563667eb8680 legacy=0x563667ebaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:56.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.931+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 >> v1:172.21.15.63:6812/3643207211 conn(0x7f504c0356d0 legacy=0x7f504c037b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.931+0000 7f506cd91f80 10 monclient: shutdown 2024-09-16T05:34:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.931+0000 7f506cd91f80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.931+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 >> v1:172.21.15.6:6790/0 conn(0x563667e09a60 legacy=0x563667e9fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.931+0000 7f506cb07640 1 -- 172.21.15.6:0/3973183555 reap_dead start 2024-09-16T05:34:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.935+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 shutdown_connections 2024-09-16T05:34:56.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.935+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 >> 172.21.15.6:0/3973183555 conn(0x563667dee510 msgr2=0x563667e9de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:56.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.935+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 shutdown_connections 2024-09-16T05:34:56.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:56.935+0000 7f506cd91f80 1 -- 172.21.15.6:0/3973183555 wait complete. 2024-09-16T05:34:56.945 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T05:34:56.945 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T05:34:56.946 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T05:34:56.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T05:34:56.946 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T05:34:57.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.071+0000 7fd9f9bb4640 10 monclient: get_monmap_and_config 2024-09-16T05:34:57.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.071+0000 7fd9f9bb4640 10 monclient: build_initial_monmap 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.071+0000 7fd9f9bb4640 10 monclient: monmap: 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:57.078195+0000 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:57.078195+0000 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:57.079 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 Processor -- start 2024-09-16T05:34:57.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- start start 2024-09-16T05:34:57.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient: init 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient(hunting): picked mon.noname-b con 0x7fd9f40a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient(hunting): picked mon.noname-c con 0x7fd9f40a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient(hunting): picked mon.noname-a con 0x7fd9f40a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f406e520 con 0x7fd9f40a1a00 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f40a0780 con 0x7fd9f40a4290 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f40a4c70 con 0x7fd9f40a37a0 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient(hunting): authenticate will time out at 958.884907s 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f37fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd9f40a37a0 0x7fd9f40a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33592/0 (socket says 172.21.15.6:33592) 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f37fe640 1 -- 172.21.15.6:0/3153394947 learned_addr learned my addr 172.21.15.6:0/3153394947 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:57.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234600961 0 0) 0x7fd9f406e520 con 0x7fd9f40a1a00 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient(hunting): my global_id is 4242 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9e0003530 con 0x7fd9f40a1a00 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 863044349 0 0) 0x7fd9f40a4c70 con 0x7fd9f40a37a0 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient(hunting): my global_id is 4244 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9f406e520 con 0x7fd9f40a37a0 2024-09-16T05:34:57.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2992728733 0 0) 0x7fd9f406e520 con 0x7fd9f40a37a0 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 >> v1:172.21.15.63:6789/0 conn(0x7fd9f40a4290 legacy=0x7fd9f413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 >> v1:172.21.15.6:6789/0 conn(0x7fd9f40a1a00 legacy=0x7fd9f413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 monclient: found mon.noname-b 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f40a65f0 con 0x7fd9f40a37a0 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: _finish_auth 0 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd9ec001600 con 0x7fd9f40a37a0 2024-09-16T05:34:57.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 5 monclient: authenticate success, global_id 4244 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: dump: 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.086 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9ec001860 con 0x7fd9f40a37a0 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 1 -- 172.21.15.6:0/3153394947 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd9ec001c30 con 0x7fd9f40a37a0 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f27fc640 10 monclient: dump: 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.087 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 4 set_mon_vals no callback set 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient: shutdown 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:57.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3153394947 >> v1:172.21.15.6:6790/0 conn(0x7fd9f40a37a0 legacy=0x7fd9f40a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3153394947 shutdown_connections 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3153394947 >> 172.21.15.6:0/3153394947 conn(0x7fd9f40a58a0 msgr2=0x7fd9f40a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3153394947 shutdown_connections 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3153394947 wait complete. 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient: build_initial_monmap 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient: monmap: 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:57.090 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 Processor -- start 2024-09-16T05:34:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 1 -- start start 2024-09-16T05:34:57.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.075+0000 7fd9f9bb4640 10 monclient: init 2024-09-16T05:34:57.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient(hunting): picked mon.noname-a con 0x7fd9f40a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient(hunting): picked mon.noname-c con 0x7fd9f40a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient(hunting): picked mon.noname-b con 0x7fd9f40a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f40a0100 con 0x7fd9f40a1a00 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f413efa0 con 0x7fd9f40a4290 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f413dd10 con 0x7fd9f40a37a0 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient(hunting): authenticate will time out at 958.888250s 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd9f40a1a00 0x7fd9f413b550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58518/0 (socket says 172.21.15.6:58518) 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f2ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd9f40a37a0 0x7fd9f413bc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33600/0 (socket says 172.21.15.6:33600) 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f37fe640 1 -- 172.21.15.6:0/3536663917 learned_addr learned my addr 172.21.15.6:0/3536663917 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4267879549 0 0) 0x7fd9f40a65f0 con 0x7fd9f40a1a00 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): my global_id is 4245 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9d4003530 con 0x7fd9f40a1a00 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4186765392 0 0) 0x7fd9f413dd10 con 0x7fd9f40a37a0 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): my global_id is 4247 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9f40a65f0 con 0x7fd9f40a37a0 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1433629035 0 0) 0x7fd9d4003530 con 0x7fd9f40a1a00 2024-09-16T05:34:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9f413dd10 con 0x7fd9f40a1a00 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd9ec001500 con 0x7fd9f40a1a00 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): dump: 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2570001525 0 0) 0x7fd9f40a65f0 con 0x7fd9f40a37a0 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9d4003530 con 0x7fd9f40a37a0 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd9dc001840 con 0x7fd9f40a37a0 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): dump: 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605454664 0 0) 0x7fd9f413efa0 con 0x7fd9f40a4290 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient(hunting): my global_id is 4246 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9f40a65f0 con 0x7fd9f40a4290 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2590192552 0 0) 0x7fd9f413dd10 con 0x7fd9f40a1a00 2024-09-16T05:34:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 >> v1:172.21.15.6:6790/0 conn(0x7fd9f40a37a0 legacy=0x7fd9f413bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 >> v1:172.21.15.63:6789/0 conn(0x7fd9f40a4290 legacy=0x7fd9f41d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient: _finish_hunting 0 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 monclient: found mon.a 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f40a81c0 con 0x7fd9f40a1a00 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient: _finish_auth 0 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient: _check_auth_tickets 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 5 monclient: authenticate success, global_id 4245 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient: _renew_subs 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f413ecc0 con 0x7fd9f40a1a00 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient: _renew_subs 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9f413c370 con 0x7fd9f40a1a00 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9ec004c80 con 0x7fd9f40a1a00 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd9ec005050 con 0x7fd9f40a1a00 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f1ffb640 4 set_mon_vals no callback set 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 10 monclient: dump: 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd9ec0051f0 con 0x7fd9f40a1a00 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d8005090 con 0x7fd9f40a1a00 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7fd9ec044230 con 0x7fd9f40a1a00 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.079+0000 7fd9f2ffd640 10 monclient: get_auth_request con 0x7fd9d4036c40 auth_method 0 2024-09-16T05:34:57.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.083+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd9f40a0100 con 0x7fd9f40a1a00 2024-09-16T05:34:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.083+0000 7fd9ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.083+0000 7fd9ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.287+0000 7fd9f9bb4640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T05:34:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.287+0000 7fd9f9bb4640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T05:34:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.287+0000 7fd9f9bb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.287+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd9d8005380 con 0x7fd9f40a1a00 2024-09-16T05:34:57.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.291+0000 7fd9ebfff640 1 -- 172.21.15.6:0/3536663917 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+84113 (unknown 481731318 0 3069344929) 0x7fd9ec015580 con 0x7fd9f40a1a00 2024-09-16T05:34:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.291+0000 7fd9ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T05:34:57.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.291+0000 7fd9ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:57.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.295+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd9d4036c40 legacy=0x7fd9d4039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.295+0000 7fd9f9bb4640 10 monclient: shutdown 2024-09-16T05:34:57.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.295+0000 7fd9f9bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.295+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 >> v1:172.21.15.6:6789/0 conn(0x7fd9f40a1a00 legacy=0x7fd9f413b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.295+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 shutdown_connections 2024-09-16T05:34:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.295+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 >> 172.21.15.6:0/3536663917 conn(0x7fd9f40a58a0 msgr2=0x7fd9f409d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.295+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 shutdown_connections 2024-09-16T05:34:57.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.295+0000 7fd9f9bb4640 1 -- 172.21.15.6:0/3536663917 wait complete. 2024-09-16T05:34:57.315 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6812","nonce":3643207211}]},"active_addr":"172.21.15.63:6812/3643207211","active_change":"2024-09-16T05:34:41.245218+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":121938566}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3525644892}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":1439578732}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3875622956}]}]} 2024-09-16T05:34:57.317 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T05:34:57.317 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T05:34:57.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:34:57.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.431+0000 7f743ce38640 10 monclient: get_monmap_and_config 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.431+0000 7f743ce38640 10 monclient: build_initial_monmap 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.431+0000 7f743ce38640 10 monclient: monmap: 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:57.438422+0000 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:57.438422+0000 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:57.439 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 Processor -- start 2024-09-16T05:34:57.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- start start 2024-09-16T05:34:57.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient: init 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient(hunting): picked mon.noname-c con 0x7f743809bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient(hunting): picked mon.noname-a con 0x7f743809bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient(hunting): picked mon.noname-b con 0x7f74381397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74380a1350 con 0x7f743809bf20 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74380a10d0 con 0x7f743809bb40 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743809d940 con 0x7f74381397f0 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient(hunting): authenticate will time out at 959.245193s 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7436d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f74381397f0 0x7f743813bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33602/0 (socket says 172.21.15.6:33602) 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f742ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f743809bf20 0x7f743809c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58522/0 (socket says 172.21.15.6:58522) 2024-09-16T05:34:57.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7436d9d640 1 -- 172.21.15.6:0/913668843 learned_addr learned my addr 172.21.15.6:0/913668843 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693932493 0 0) 0x7f74380a1350 con 0x7f743809bf20 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient(hunting): my global_id is 4248 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7424003530 con 0x7f743809bf20 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247965885 0 0) 0x7f743809d940 con 0x7f74381397f0 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient(hunting): my global_id is 4250 2024-09-16T05:34:57.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74380a1350 con 0x7f74381397f0 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1476464972 0 0) 0x7f7424003530 con 0x7f743809bf20 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 >> v1:172.21.15.6:6790/0 conn(0x7f74381397f0 legacy=0x7f743813bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 >> v1:172.21.15.63:6789/0 conn(0x7f743809bb40 legacy=0x7f743809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 monclient: found mon.noname-a 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743809dc20 con 0x7f743809bf20 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: _finish_auth 0 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7420001550 con 0x7f743809bf20 2024-09-16T05:34:57.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: dump: 2024-09-16T05:34:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.446 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 5 monclient: authenticate success, global_id 4248 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74200017b0 con 0x7f743809bf20 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 1 -- 172.21.15.6:0/913668843 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7420001b40 con 0x7f743809bf20 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 4 set_mon_vals no callback set 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f7435d9b640 10 monclient: dump: 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient: shutdown 2024-09-16T05:34:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- 172.21.15.6:0/913668843 >> v1:172.21.15.6:6789/0 conn(0x7f743809bf20 legacy=0x7f743809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- 172.21.15.6:0/913668843 shutdown_connections 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- 172.21.15.6:0/913668843 >> 172.21.15.6:0/913668843 conn(0x7f74380a1650 msgr2=0x7f74380a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- 172.21.15.6:0/913668843 shutdown_connections 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- 172.21.15.6:0/913668843 wait complete. 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient: build_initial_monmap 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient: monmap: 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:57.449 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:57.450 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 Processor -- start 2024-09-16T05:34:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 1 -- start start 2024-09-16T05:34:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.435+0000 7f743ce38640 10 monclient: init 2024-09-16T05:34:57.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:57.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient(hunting): picked mon.noname-a con 0x7f743809bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient(hunting): picked mon.noname-b con 0x7f743809bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient(hunting): picked mon.noname-c con 0x7f74381397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743813c950 con 0x7f743809bb40 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7438071c30 con 0x7f743809bf20 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743813aa20 con 0x7f74381397f0 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient(hunting): authenticate will time out at 959.248400s 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f743809bb40 0x7f7438139c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58538/0 (socket says 172.21.15.6:58538) 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f7436d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f74381397f0 0x7f74381d3600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33614/0 (socket says 172.21.15.6:33614) 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743659c640 1 -- 172.21.15.6:0/2111765810 learned_addr learned my addr 172.21.15.6:0/2111765810 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677126114 0 0) 0x7f74280012d0 con 0x7f743809bb40 2024-09-16T05:34:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): my global_id is 4251 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7418003530 con 0x7f743809bb40 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1527139077 0 0) 0x7f743813aa20 con 0x7f74381397f0 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): my global_id is 4253 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74280012d0 con 0x7f74381397f0 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 50251115 0 0) 0x7f7418003530 con 0x7f743809bb40 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f743813aa20 con 0x7f743809bb40 2024-09-16T05:34:57.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7428001bc0 con 0x7f743809bb40 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): dump: 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.456 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 299674918 0 0) 0x7f74280012d0 con 0x7f74381397f0 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7418003530 con 0x7f74381397f0 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7430001c40 con 0x7f74381397f0 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): dump: 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.457 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343539843 0 0) 0x7f7438071c30 con 0x7f743809bf20 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient(hunting): my global_id is 4252 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74280012d0 con 0x7f743809bf20 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4015423769 0 0) 0x7f743813aa20 con 0x7f743809bb40 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 >> v1:172.21.15.6:6790/0 conn(0x7f74381397f0 legacy=0x7f74381d3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 >> v1:172.21.15.63:6789/0 conn(0x7f743809bf20 legacy=0x7f743813a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 monclient: found mon.a 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74380a3c50 con 0x7f743809bb40 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient: _finish_auth 0 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 5 monclient: authenticate success, global_id 4251 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient: _renew_subs 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f74380721f0 con 0x7f743809bb40 2024-09-16T05:34:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient: _renew_subs 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7438072610 con 0x7f743809bb40 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7428003950 con 0x7f743809bb40 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7428003d20 con 0x7f743809bb40 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743559a640 4 set_mon_vals no callback set 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 10 monclient: dump: 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.460 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f741c005090 con 0x7f743809bb40 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7428003ec0 con 0x7f743809bb40 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f74280444c0 con 0x7f743809bb40 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.439+0000 7f742ffff640 10 monclient: get_auth_request con 0x7f7418036c40 auth_method 0 2024-09-16T05:34:57.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.443+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f743813c950 con 0x7f743809bb40 2024-09-16T05:34:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.443+0000 7f742effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.443+0000 7f742effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:57.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.607+0000 7f743ce38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:57.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.607+0000 7f743ce38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:57.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.607+0000 7f743ce38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.607+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f741c005380 con 0x7f743809bb40 2024-09-16T05:34:57.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f742effd640 1 -- 172.21.15.6:0/2111765810 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10889 (unknown 941590788 0 3514356718) 0x7f74280157b0 con 0x7f743809bb40 2024-09-16T05:34:57.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f742effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:57.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f742effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:57.616 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:57.616 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","created":"2024-09-16T05:34:36.231874+0000","modified":"2024-09-16T05:34:56.913811+0000","last_up_change":"2024-09-16T05:34:46.347254+0000","last_in_change":"2024-09-16T05:34:37.929266+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:34:47.340654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T05:34:53.191455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28573ac1-022a-424e-a024-a8ae14680604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":3069057739}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":3069057739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":3069057739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":3069057739}]},"public_addr":"172.21.15.6:6804/3069057739","cluster_addr":"172.21.15.6:6805/3069057739","heartbeat_back_addr":"172.21.15.6:6807/3069057739","heartbeat_front_addr":"172.21.15.6:6806/3069057739","state":["exists","up"]},{"osd":1,"uuid":"5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":4098222447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":4098222447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":4098222447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":4098222447}]},"public_addr":"172.21.15.6:6808/4098222447","cluster_addr":"172.21.15.6:6809/4098222447","heartbeat_back_addr":"172.21.15.6:6811/4098222447","heartbeat_front_addr":"172.21.15.6:6810/4098222447","state":["exists","up"]},{"osd":2,"uuid":"89377298-ce68-4fd4-bb63-2ffd60eeea3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":1572197727}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":1572197727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":1572197727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":1572197727}]},"public_addr":"172.21.15.6:6800/1572197727","cluster_addr":"172.21.15.6:6801/1572197727","heartbeat_back_addr":"172.21.15.6:6803/1572197727","heartbeat_front_addr":"172.21.15.6:6802/1572197727","state":["exists","up"]},{"osd":3,"uuid":"7c5624a2-d552-4295-b112-d87972274f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6800","nonce":709309215}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":709309215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":709309215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":709309215}]},"public_addr":"172.21.15.63:6800/709309215","cluster_addr":"172.21.15.63:6801/709309215","heartbeat_back_addr":"172.21.15.63:6803/709309215","heartbeat_front_addr":"172.21.15.63:6802/709309215","state":["exists","up"]},{"osd":4,"uuid":"52fa74b0-2bf6-4c82-acb0-4032eca1f08b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":118134224}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":118134224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":118134224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":118134224}]},"public_addr":"172.21.15.63:6804/118134224","cluster_addr":"172.21.15.63:6805/118134224","heartbeat_back_addr":"172.21.15.63:6807/118134224","heartbeat_front_addr":"172.21.15.63:6806/118134224","state":["exists","up"]},{"osd":5,"uuid":"a4dac512-ff9b-4381-9065-82bf9a711bdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":165284194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":165284194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":165284194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":165284194}]},"public_addr":"172.21.15.63:6808/165284194","cluster_addr":"172.21.15.63:6809/165284194","heartbeat_back_addr":"172.21.15.63:6811/165284194","heartbeat_front_addr":"172.21.15.63:6810/165284194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.065166+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.182281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.088394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.071475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.333358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.397991+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:34:57.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7418036c40 legacy=0x7f7418039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f743ce38640 10 monclient: shutdown 2024-09-16T05:34:57.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f743ce38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:57.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 >> v1:172.21.15.6:6789/0 conn(0x7f743809bb40 legacy=0x7f7438139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 shutdown_connections 2024-09-16T05:34:57.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 >> 172.21.15.6:0/2111765810 conn(0x7f74380a1650 msgr2=0x7f7438138b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:57.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 shutdown_connections 2024-09-16T05:34:57.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.611+0000 7f743ce38640 1 -- 172.21.15.6:0/2111765810 wait complete. 2024-09-16T05:34:57.635 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T05:34:57.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:34:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.747+0000 7fdba86b6640 10 monclient: get_monmap_and_config 2024-09-16T05:34:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.747+0000 7fdba86b6640 10 monclient: build_initial_monmap 2024-09-16T05:34:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.747+0000 7fdba86b6640 10 monclient: monmap: 2024-09-16T05:34:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:57.754384+0000 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:57.754384+0000 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 Processor -- start 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- start start 2024-09-16T05:34:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient: init 2024-09-16T05:34:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:57.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient(hunting): picked mon.noname-c con 0x7fdba006cd20 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient(hunting): picked mon.noname-b con 0x7fdba00a3220 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient(hunting): picked mon.noname-a con 0x7fdba013cb80 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba00a2fe0 con 0x7fdba013cb80 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba006ecb0 con 0x7fdba006cd20 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba0073a30 con 0x7fdba00a3220 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient(hunting): authenticate will time out at 959.560992s 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5c51640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdba00a3220 0x7fdba006d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33618/0 (socket says 172.21.15.6:33618) 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5c51640 1 -- 172.21.15.6:0/3826470186 learned_addr learned my addr 172.21.15.6:0/3826470186 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370504263 0 0) 0x7fdba0073a30 con 0x7fdba00a3220 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient(hunting): my global_id is 4256 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb8c003530 con 0x7fdba00a3220 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944522918 0 0) 0x7fdba00a2fe0 con 0x7fdba013cb80 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient(hunting): my global_id is 4254 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba0073a30 con 0x7fdba013cb80 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3131233073 0 0) 0x7fdb8c003530 con 0x7fdba00a3220 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 >> v1:172.21.15.63:6789/0 conn(0x7fdba006cd20 legacy=0x7fdba006d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 >> v1:172.21.15.6:6789/0 conn(0x7fdba013cb80 legacy=0x7fdba013ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: _finish_hunting 0 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 monclient: found mon.noname-b 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba00a29f0 con 0x7fdba00a3220 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: _finish_auth 0 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: _check_auth_tickets 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdb9c0019a0 con 0x7fdba00a3220 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: dump: 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 5 monclient: authenticate success, global_id 4256 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb9c001c00 con 0x7fdba00a3220 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:57.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 1 -- 172.21.15.6:0/3826470186 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdb9c002010 con 0x7fdba00a3220 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba5450640 10 monclient: dump: 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 4 set_mon_vals no callback set 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient: shutdown 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- 172.21.15.6:0/3826470186 >> v1:172.21.15.6:6790/0 conn(0x7fdba00a3220 legacy=0x7fdba006d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- 172.21.15.6:0/3826470186 shutdown_connections 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- 172.21.15.6:0/3826470186 >> 172.21.15.6:0/3826470186 conn(0x7fdba00a3880 msgr2=0x7fdba00a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- 172.21.15.6:0/3826470186 shutdown_connections 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 1 -- 172.21.15.6:0/3826470186 wait complete. 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient: build_initial_monmap 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.751+0000 7fdba86b6640 10 monclient: monmap: 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:57.768 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 1 Processor -- start 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 1 -- start start 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: init 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:57.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient(hunting): picked mon.noname-a con 0x7fdba006cd20 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient(hunting): picked mon.noname-b con 0x7fdba00a3220 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient(hunting): picked mon.noname-c con 0x7fdba013cb80 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba0138ac0 con 0x7fdba006cd20 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba01cbfb0 con 0x7fdba00a3220 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba0137e80 con 0x7fdba013cb80 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient(hunting): authenticate will time out at 959.564766s 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba6452640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdba006cd20 0x7fdba013b2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58556/0 (socket says 172.21.15.6:58556) 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba6452640 1 -- 172.21.15.6:0/1063869328 learned_addr learned my addr 172.21.15.6:0/1063869328 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 643182722 0 0) 0x7fdba0138ac0 con 0x7fdba006cd20 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): my global_id is 4257 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb84003530 con 0x7fdba006cd20 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 408289386 0 0) 0x7fdba0137e80 con 0x7fdba013cb80 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): my global_id is 4259 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba0138ac0 con 0x7fdba013cb80 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2759095217 0 0) 0x7fdba01cbfb0 con 0x7fdba00a3220 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:57.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): my global_id is 4258 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba0137e80 con 0x7fdba00a3220 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1924928899 0 0) 0x7fdb84003530 con 0x7fdba006cd20 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdba01cbfb0 con 0x7fdba006cd20 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdb900019f0 con 0x7fdba006cd20 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): dump: 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.772 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3288677050 0 0) 0x7fdba0138ac0 con 0x7fdba013cb80 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb84003530 con 0x7fdba013cb80 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdb98001730 con 0x7fdba013cb80 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient(hunting): dump: 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.773 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3042993804 0 0) 0x7fdba01cbfb0 con 0x7fdba006cd20 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 >> v1:172.21.15.6:6790/0 conn(0x7fdba013cb80 legacy=0x7fdba01d3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 >> v1:172.21.15.63:6789/0 conn(0x7fdba00a3220 legacy=0x7fdba01cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 monclient: found mon.a 2024-09-16T05:34:57.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba01387e0 con 0x7fdba006cd20 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 5 monclient: authenticate success, global_id 4257 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: _renew_subs 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdba013b9f0 con 0x7fdba006cd20 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: _renew_subs 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdba01cb7d0 con 0x7fdba006cd20 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb90001d10 con 0x7fdba006cd20 2024-09-16T05:34:57.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdb90003a70 con 0x7fdba006cd20 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba4c4f640 4 set_mon_vals no callback set 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 10 monclient: dump: 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:57.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb88005090 con 0x7fdba006cd20 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdb90003c10 con 0x7fdba006cd20 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7fdb90044170 con 0x7fdba006cd20 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.755+0000 7fdba5c51640 10 monclient: get_auth_request con 0x7fdb84036c20 auth_method 0 2024-09-16T05:34:57.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.759+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fdb90015400 con 0x7fdba006cd20 2024-09-16T05:34:57.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.759+0000 7fdb96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:57.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.759+0000 7fdb96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:57.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.923+0000 7fdba86b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:57.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.923+0000 7fdba86b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:57.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.923+0000 7fdba86b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:57.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.923+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb88005380 con 0x7fdba006cd20 2024-09-16T05:34:57.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.927+0000 7fdb96ffd640 1 -- 172.21.15.6:0/1063869328 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10889 (unknown 941590788 0 3514356718) 0x7fdb900190b0 con 0x7fdba006cd20 2024-09-16T05:34:57.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.927+0000 7fdb96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:34:57.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.927+0000 7fdb96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:57.934 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:34:57.934 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","created":"2024-09-16T05:34:36.231874+0000","modified":"2024-09-16T05:34:56.913811+0000","last_up_change":"2024-09-16T05:34:46.347254+0000","last_in_change":"2024-09-16T05:34:37.929266+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:34:47.340654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T05:34:53.191455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28573ac1-022a-424e-a024-a8ae14680604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":3069057739}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":3069057739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":3069057739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":3069057739}]},"public_addr":"172.21.15.6:6804/3069057739","cluster_addr":"172.21.15.6:6805/3069057739","heartbeat_back_addr":"172.21.15.6:6807/3069057739","heartbeat_front_addr":"172.21.15.6:6806/3069057739","state":["exists","up"]},{"osd":1,"uuid":"5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":4098222447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":4098222447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":4098222447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":4098222447}]},"public_addr":"172.21.15.6:6808/4098222447","cluster_addr":"172.21.15.6:6809/4098222447","heartbeat_back_addr":"172.21.15.6:6811/4098222447","heartbeat_front_addr":"172.21.15.6:6810/4098222447","state":["exists","up"]},{"osd":2,"uuid":"89377298-ce68-4fd4-bb63-2ffd60eeea3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":1572197727}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":1572197727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":1572197727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":1572197727}]},"public_addr":"172.21.15.6:6800/1572197727","cluster_addr":"172.21.15.6:6801/1572197727","heartbeat_back_addr":"172.21.15.6:6803/1572197727","heartbeat_front_addr":"172.21.15.6:6802/1572197727","state":["exists","up"]},{"osd":3,"uuid":"7c5624a2-d552-4295-b112-d87972274f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6800","nonce":709309215}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":709309215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":709309215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":709309215}]},"public_addr":"172.21.15.63:6800/709309215","cluster_addr":"172.21.15.63:6801/709309215","heartbeat_back_addr":"172.21.15.63:6803/709309215","heartbeat_front_addr":"172.21.15.63:6802/709309215","state":["exists","up"]},{"osd":4,"uuid":"52fa74b0-2bf6-4c82-acb0-4032eca1f08b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":118134224}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":118134224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":118134224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":118134224}]},"public_addr":"172.21.15.63:6804/118134224","cluster_addr":"172.21.15.63:6805/118134224","heartbeat_back_addr":"172.21.15.63:6807/118134224","heartbeat_front_addr":"172.21.15.63:6806/118134224","state":["exists","up"]},{"osd":5,"uuid":"a4dac512-ff9b-4381-9065-82bf9a711bdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":165284194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":165284194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":165284194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":165284194}]},"public_addr":"172.21.15.63:6808/165284194","cluster_addr":"172.21.15.63:6809/165284194","heartbeat_back_addr":"172.21.15.63:6811/165284194","heartbeat_front_addr":"172.21.15.63:6810/165284194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.065166+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.182281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.088394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.071475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.333358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.397991+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:34:57.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.931+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 >> v1:172.21.15.63:6812/3643207211 conn(0x7fdb84036c20 legacy=0x7fdb840390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.931+0000 7fdba86b6640 10 monclient: shutdown 2024-09-16T05:34:57.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.931+0000 7fdba86b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:57.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.931+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 >> v1:172.21.15.6:6789/0 conn(0x7fdba006cd20 legacy=0x7fdba013b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:57.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.931+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 shutdown_connections 2024-09-16T05:34:57.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.931+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 >> 172.21.15.6:0/1063869328 conn(0x7fdba00a3880 msgr2=0x7fdba013bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:57.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.931+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 shutdown_connections 2024-09-16T05:34:57.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:57.931+0000 7fdba86b6640 1 -- 172.21.15.6:0/1063869328 wait complete. 2024-09-16T05:34:57.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T05:34:57.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T05:34:57.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T05:34:57.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T05:34:57.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T05:34:57.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T05:34:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient: monmap: 2024-09-16T05:34:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.076 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.075681+0000 2024-09-16T05:34:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.075681+0000 2024-09-16T05:34:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.077 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient: monmap: 2024-09-16T05:34:58.078 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.078 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.078 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.076172+0000 2024-09-16T05:34:58.078 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.076172+0000 2024-09-16T05:34:58.078 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.079 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.079 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 1 Processor -- start 2024-09-16T05:34:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 1 -- start start 2024-09-16T05:34:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient: init 2024-09-16T05:34:58.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 1 Processor -- start 2024-09-16T05:34:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 1 -- start start 2024-09-16T05:34:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient: init 2024-09-16T05:34:58.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:58.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient(hunting): picked mon.noname-a con 0x7fbb540a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient(hunting): picked mon.noname-b con 0x7fbb540a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient(hunting): picked mon.noname-c con 0x7fbb540a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb527fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbb540a2570 0x7fbb5413b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33644/0 (socket says 172.21.15.6:33644) 2024-09-16T05:34:58.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb527fc640 1 -- 172.21.15.6:0/1872430765 learned_addr learned my addr 172.21.15.6:0/1872430765 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:58.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient(hunting): picked mon.noname-b con 0x7fa8fc134880 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa90164a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8fc0a0f20 0x7fa8fc139750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58580/0 (socket says 172.21.15.6:58580) 2024-09-16T05:34:58.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa90164a640 1 -- 172.21.15.6:0/1658713073 learned_addr learned my addr 172.21.15.6:0/1658713073 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 1 -- 172.21.15.6:0/1872430765 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb540a33a0 con 0x7fbb540a2190 2024-09-16T05:34:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 1 -- 172.21.15.6:0/1872430765 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb540a07a0 con 0x7fbb540a2980 2024-09-16T05:34:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 1 -- 172.21.15.6:0/1872430765 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb540a2d90 con 0x7fbb540a2570 2024-09-16T05:34:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 10 monclient(hunting): authenticate will time out at 959.883069s 2024-09-16T05:34:58.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient(hunting): picked mon.noname-a con 0x7fa8fc0a0f20 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 901594207 0 0) 0x7fbb540a07a0 con 0x7fbb540a2980 2024-09-16T05:34:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): my global_id is 4261 2024-09-16T05:34:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb3c003530 con 0x7fbb540a2980 2024-09-16T05:34:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient(hunting): picked mon.noname-c con 0x7fa8fc13ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240773481 0 0) 0x7fbb540a2d90 con 0x7fbb540a2570 2024-09-16T05:34:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): my global_id is 4262 2024-09-16T05:34:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb540a07a0 con 0x7fbb540a2570 2024-09-16T05:34:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2007583203 0 0) 0x7fbb540a33a0 con 0x7fbb540a2190 2024-09-16T05:34:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient(hunting): my global_id is 4260 2024-09-16T05:34:58.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb540a2d90 con 0x7fbb540a2190 2024-09-16T05:34:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 1 -- 172.21.15.6:0/1658713073 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8fc0a3180 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 1 -- 172.21.15.6:0/1658713073 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8fc09f2a0 con 0x7fa8fc13ae70 2024-09-16T05:34:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3014614040 0 0) 0x7fbb540a07a0 con 0x7fbb540a2570 2024-09-16T05:34:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 >> v1:172.21.15.63:6789/0 conn(0x7fbb540a2980 legacy=0x7fbb5413f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 1 -- 172.21.15.6:0/1658713073 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8fc0a0260 con 0x7fa8fc134880 2024-09-16T05:34:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 >> v1:172.21.15.6:6789/0 conn(0x7fbb540a2190 legacy=0x7fbb540a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 monclient: found mon.noname-b 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb540a66e0 con 0x7fbb540a2570 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb48001600 con 0x7fbb540a2570 2024-09-16T05:34:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb51ffb640 10 monclient: dump: 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 5 monclient: authenticate success, global_id 4262 2024-09-16T05:34:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fbb5935a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa9040af640 10 monclient(hunting): authenticate will time out at 959.883830s 2024-09-16T05:34:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1032904698 0 0) 0x7fa8ec001120 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa900e49640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa900e49640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa900e49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa900e49640 10 monclient(hunting): my global_id is 4263 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.071+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8e8003530 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3403153371 0 0) 0x7fa8fc09f2a0 con 0x7fa8fc13ae70 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient(hunting): my global_id is 4264 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8ec001120 con 0x7fa8fc13ae70 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1172675493 0 0) 0x7fa8fc0a0260 con 0x7fa8fc134880 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient(hunting): my global_id is 4265 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8fc09f2a0 con 0x7fa8fc134880 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb48001860 con 0x7fbb540a2570 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb51ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb51ffb640 1 -- 172.21.15.6:0/1872430765 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb48001c70 con 0x7fbb540a2570 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb51ffb640 10 monclient: dump: 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 4 set_mon_vals no callback set 2024-09-16T05:34:58.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2838458353 0 0) 0x7fa8e8003530 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 >> v1:172.21.15.6:6790/0 conn(0x7fa8fc134880 legacy=0x7fa8fc134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 >> v1:172.21.15.63:6789/0 conn(0x7fa8fc13ae70 legacy=0x7fa8fc13d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 monclient: found mon.noname-a 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8fc0a0540 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa8ec0017f0 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: dump: 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 5 monclient: authenticate success, global_id 4263 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8ec001a50 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 1 -- 172.21.15.6:0/1658713073 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa8ec0048b0 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa900e49640 10 monclient: dump: 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.102 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.103 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.103 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 4 set_mon_vals no callback set 2024-09-16T05:34:58.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient: shutdown 2024-09-16T05:34:58.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 1 -- 172.21.15.6:0/1872430765 >> v1:172.21.15.6:6790/0 conn(0x7fbb540a2570 legacy=0x7fbb5413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 1 -- 172.21.15.6:0/1872430765 shutdown_connections 2024-09-16T05:34:58.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 1 -- 172.21.15.6:0/1872430765 >> 172.21.15.6:0/1872430765 conn(0x7fbb540a9850 msgr2=0x7fbb540a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 1 -- 172.21.15.6:0/1872430765 shutdown_connections 2024-09-16T05:34:58.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient: shutdown 2024-09-16T05:34:58.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 1 -- 172.21.15.6:0/1658713073 >> v1:172.21.15.6:6789/0 conn(0x7fa8fc0a0f20 legacy=0x7fa8fc139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 1 -- 172.21.15.6:0/1872430765 wait complete. 2024-09-16T05:34:58.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 1 -- 172.21.15.6:0/1658713073 shutdown_connections 2024-09-16T05:34:58.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 1 -- 172.21.15.6:0/1658713073 >> 172.21.15.6:0/1658713073 conn(0x7fa8fc0a37e0 msgr2=0x7fa8fc0a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 1 -- 172.21.15.6:0/1658713073 shutdown_connections 2024-09-16T05:34:58.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 1 -- 172.21.15.6:0/1658713073 wait complete. 2024-09-16T05:34:58.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient: monmap: 2024-09-16T05:34:58.106 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.106 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient: monmap: 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.108 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 1 Processor -- start 2024-09-16T05:34:58.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 1 -- start start 2024-09-16T05:34:58.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient: init 2024-09-16T05:34:58.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 1 Processor -- start 2024-09-16T05:34:58.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 1 -- start start 2024-09-16T05:34:58.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient: init 2024-09-16T05:34:58.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:58.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient(hunting): picked mon.noname-c con 0x7fbb540a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:58.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient(hunting): picked mon.noname-c con 0x7fa8fc0a0f20 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient(hunting): picked mon.noname-b con 0x7fa8fc134880 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient(hunting): picked mon.noname-a con 0x7fbb540a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 10 monclient(hunting): picked mon.noname-a con 0x7fa8fc13ae70 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 10 monclient(hunting): picked mon.noname-b con 0x7fbb540a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa901e4b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa8fc0a0f20 0x7fa8fc13b250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33678/0 (socket says 172.21.15.6:33678) 2024-09-16T05:34:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa901e4b640 1 -- 172.21.15.6:0/1224390420 learned_addr learned my addr 172.21.15.6:0/1224390420 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb5935a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb54123e50 con 0x7fbb540a2570 2024-09-16T05:34:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb527fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbb540a2570 0x7fbb54120cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58586/0 (socket says 172.21.15.6:58586) 2024-09-16T05:34:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fbb527fc640 1 -- 172.21.15.6:0/448612724 learned_addr learned my addr 172.21.15.6:0/448612724 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.075+0000 7fa9040af640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8fc13df00 con 0x7fa8fc13ae70 2024-09-16T05:34:58.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb54124130 con 0x7fbb540a2980 2024-09-16T05:34:58.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8fc13c070 con 0x7fa8fc134880 2024-09-16T05:34:58.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8fc0742c0 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 10 monclient(hunting): authenticate will time out at 959.888151s 2024-09-16T05:34:58.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb541213d0 con 0x7fbb540a2190 2024-09-16T05:34:58.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 10 monclient(hunting): authenticate will time out at 959.888227s 2024-09-16T05:34:58.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3464689706 0 0) 0x7fa8fc13c070 con 0x7fa8fc134880 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): my global_id is 4267 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8e0003530 con 0x7fa8fc134880 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 921194626 0 0) 0x7fa8fc0742c0 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): my global_id is 4268 2024-09-16T05:34:58.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8fc13c070 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4082638043 0 0) 0x7fbb54124130 con 0x7fbb540a2980 2024-09-16T05:34:58.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): my global_id is 4270 2024-09-16T05:34:58.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb40003530 con 0x7fbb540a2980 2024-09-16T05:34:58.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3164429934 0 0) 0x7fbb54123e50 con 0x7fbb540a2570 2024-09-16T05:34:58.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): my global_id is 4269 2024-09-16T05:34:58.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb54124130 con 0x7fbb540a2570 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 15831716 0 0) 0x7fa8ec001120 con 0x7fa8fc13ae70 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): my global_id is 4266 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8fc0742c0 con 0x7fa8fc13ae70 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1244570262 0 0) 0x7fa8fc13c070 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8ec001120 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa8f8001980 con 0x7fa8fc0a0f20 2024-09-16T05:34:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:58.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): dump: 2024-09-16T05:34:58.126 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.126 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.126 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.126 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.126 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2551091654 0 0) 0x7fa8e0003530 con 0x7fa8fc134880 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8fc13c070 con 0x7fa8fc134880 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa8ec0047f0 con 0x7fa8fc134880 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient(hunting): dump: 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.128 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2247760546 0 0) 0x7fbb40003530 con 0x7fbb540a2980 2024-09-16T05:34:58.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb54123e50 con 0x7fbb540a2980 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb4c001c60 con 0x7fbb540a2980 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): dump: 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.130 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.131 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.131 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.131 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.131 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3327575624 0 0) 0x7fbb54124130 con 0x7fbb540a2570 2024-09-16T05:34:58.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1698375088 0 0) 0x7fa8fc13c070 con 0x7fa8fc134880 2024-09-16T05:34:58.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 >> v1:172.21.15.6:6790/0 conn(0x7fa8fc0a0f20 legacy=0x7fa8fc13b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 >> v1:172.21.15.6:6789/0 conn(0x7fa8fc13ae70 legacy=0x7fa8fc1d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 monclient: found mon.b 2024-09-16T05:34:58.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8fc13e1e0 con 0x7fa8fc134880 2024-09-16T05:34:58.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 5 monclient: authenticate success, global_id 4267 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 10 monclient: _renew_subs 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8fc13d350 con 0x7fa8fc134880 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 10 monclient: _renew_subs 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa9040af640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8fc09d440 con 0x7fa8fc134880 2024-09-16T05:34:58.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb40003530 con 0x7fbb540a2570 2024-09-16T05:34:58.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb48001bf0 con 0x7fbb540a2570 2024-09-16T05:34:58.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient(hunting): dump: 2024-09-16T05:34:58.134 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 347579556 0 0) 0x7fbb54123e50 con 0x7fbb540a2980 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 >> v1:172.21.15.6:6790/0 conn(0x7fbb540a2190 legacy=0x7fbb541205b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 >> v1:172.21.15.6:6789/0 conn(0x7fbb540a2570 legacy=0x7fbb54120cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 monclient: found mon.b 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb54122c80 con 0x7fbb540a2980 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 5 monclient: authenticate success, global_id 4270 2024-09-16T05:34:58.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 10 monclient: _renew_subs 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb541225c0 con 0x7fbb540a2980 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 10 monclient: _renew_subs 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb5935a640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb5412d300 con 0x7fbb540a2980 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb4c0039a0 con 0x7fbb540a2980 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb4c003cf0 con 0x7fbb540a2980 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: dump: 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.137 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb517fa640 4 set_mon_vals no callback set 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbb2c000f90 con 0x7fbb540a2980 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8ec004e60 con 0x7fa8fc134880 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa8ec0051b0 con 0x7fa8fc134880 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: dump: 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.139 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f3fff640 4 set_mon_vals no callback set 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8dffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fa8d8000f90 con 0x7fa8fc134880 2024-09-16T05:34:58.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbb4c00ea90 con 0x7fbb540a2980 2024-09-16T05:34:58.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa8ec00ff50 con 0x7fa8fc134880 2024-09-16T05:34:58.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7fa8ec0441a0 con 0x7fa8fc134880 2024-09-16T05:34:58.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7fbb4c043170 con 0x7fbb540a2980 2024-09-16T05:34:58.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6808/165284194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8e0041480 con 0x7fa8e003d7d0 2024-09-16T05:34:58.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fa8ec044580 con 0x7fa8fc134880 2024-09-16T05:34:58.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fa8f27fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T05:34:58.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6800/709309215 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb400412e0 con 0x7fbb4003d630 2024-09-16T05:34:58.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fbb4c043550 con 0x7fbb540a2980 2024-09-16T05:34:58.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.079+0000 7fbb3b7fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T05:34:58.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.083+0000 7fbb52ffd640 10 monclient: get_auth_request con 0x7fbb40036af0 auth_method 0 2024-09-16T05:34:58.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.083+0000 7fa901e4b640 10 monclient: get_auth_request con 0x7fa8e0036c90 auth_method 0 2024-09-16T05:34:58.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.083+0000 7fa90264c640 10 monclient: get_auth_request con 0x7fa8e003d7d0 auth_method 0 2024-09-16T05:34:58.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.083+0000 7fbb527fc640 10 monclient: get_auth_request con 0x7fbb4003d630 auth_method 0 2024-09-16T05:34:58.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.083+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== osd.5 v1:172.21.15.63:6808/165284194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa8e0041480 con 0x7fa8e003d7d0 2024-09-16T05:34:58.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.083+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== osd.3 v1:172.21.15.63:6800/709309215 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbb400412e0 con 0x7fbb4003d630 2024-09-16T05:34:58.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 --> v1:172.21.15.63:6808/165284194 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa8d8002c30 con 0x7fa8e003d7d0 2024-09-16T05:34:58.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8f27fc640 1 -- 172.21.15.6:0/1224390420 <== osd.5 v1:172.21.15.63:6808/165284194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3543570501) 0x7fa8d8002c30 con 0x7fa8e003d7d0 2024-09-16T05:34:58.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 >> v1:172.21.15.63:6808/165284194 conn(0x7fa8e003d7d0 legacy=0x7fa8e003fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa8e0036c90 legacy=0x7fa8e0039150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 10 monclient: shutdown 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 >> v1:172.21.15.63:6789/0 conn(0x7fa8fc134880 legacy=0x7fa8fc13b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa90264c640 1 -- 172.21.15.6:0/1224390420 reap_dead start 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 shutdown_connections 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 >> 172.21.15.6:0/1224390420 conn(0x7fa8fc0a37e0 msgr2=0x7fa8fc12d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 shutdown_connections 2024-09-16T05:34:58.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.099+0000 7fa8dffff640 1 -- 172.21.15.6:0/1224390420 wait complete. 2024-09-16T05:34:58.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 --> v1:172.21.15.63:6800/709309215 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbb2c002c30 con 0x7fbb4003d630 2024-09-16T05:34:58.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb3b7fe640 1 -- 172.21.15.6:0/448612724 <== osd.3 v1:172.21.15.63:6800/709309215 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7fbb2c002c30 con 0x7fbb4003d630 2024-09-16T05:34:58.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7f6256f50640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7f6256f50640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 >> v1:172.21.15.63:6800/709309215 conn(0x7fbb4003d630 legacy=0x7fbb4003fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7f6256f50640 10 monclient: monmap: 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.121332+0000 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.121332+0000 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.149 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.150 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.150 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbb40036af0 legacy=0x7fbb40038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 10 monclient: shutdown 2024-09-16T05:34:58.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 >> v1:172.21.15.63:6789/0 conn(0x7fbb540a2980 legacy=0x7fbb541d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb537fe640 1 -- 172.21.15.6:0/448612724 reap_dead start 2024-09-16T05:34:58.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 shutdown_connections 2024-09-16T05:34:58.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 >> 172.21.15.6:0/448612724 conn(0x7fbb540a9850 msgr2=0x7fbb5406ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 shutdown_connections 2024-09-16T05:34:58.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7fbb397fa640 1 -- 172.21.15.6:0/448612724 wait complete. 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7f6256f50640 1 Processor -- start 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7f6256f50640 1 -- start start 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.115+0000 7f6256f50640 10 monclient: init 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): picked mon.noname-c con 0x7f62500a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): picked mon.noname-b con 0x7f62500a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): picked mon.noname-a con 0x7f62500a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62500a33a0 con 0x7f62500a2980 2024-09-16T05:34:58.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62500a07a0 con 0x7f62500a2190 2024-09-16T05:34:58.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62500a2d90 con 0x7f62500a2570 2024-09-16T05:34:58.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): authenticate will time out at 959.928194s 2024-09-16T05:34:58.153 INFO:teuthology.orchestra.run.smithi006.stdout:51539607557 2024-09-16T05:34:58.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f62554ed640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62500a2980 0x7f625013f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58608/0 (socket says 172.21.15.6:58608) 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f62554ed640 1 -- 172.21.15.6:0/1864680690 learned_addr learned my addr 172.21.15.6:0/1864680690 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3032745533 0 0) 0x7f62500a33a0 con 0x7f62500a2980 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient(hunting): my global_id is 4272 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6238003530 con 0x7f62500a2980 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1391855737 0 0) 0x7f62500a07a0 con 0x7f62500a2190 2024-09-16T05:34:58.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient(hunting): my global_id is 4273 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62500a33a0 con 0x7f62500a2190 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2608710194 0 0) 0x7f6238003530 con 0x7f62500a2980 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 >> v1:172.21.15.6:6790/0 conn(0x7f62500a2570 legacy=0x7f625013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 >> v1:172.21.15.63:6789/0 conn(0x7f62500a2190 legacy=0x7f62500a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 monclient: found mon.noname-a 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62500a66e0 con 0x7f62500a2980 2024-09-16T05:34:58.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6248001620 con 0x7f62500a2980 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 5 monclient: authenticate success, global_id 4272 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: dump: 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.161 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6248001880 con 0x7f62500a2980 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 1 -- 172.21.15.6:0/1864680690 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6248001c50 con 0x7f62500a2980 2024-09-16T05:34:58.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 4 set_mon_vals no callback set 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f624f7fe640 10 monclient: dump: 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.163 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: shutdown 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- 172.21.15.6:0/1864680690 >> v1:172.21.15.6:6789/0 conn(0x7f62500a2980 legacy=0x7f625013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- 172.21.15.6:0/1864680690 shutdown_connections 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- 172.21.15.6:0/1864680690 >> 172.21.15.6:0/1864680690 conn(0x7f62500a9850 msgr2=0x7f62500a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- 172.21.15.6:0/1864680690 shutdown_connections 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- 172.21.15.6:0/1864680690 wait complete. 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: monmap: 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 Processor -- start 2024-09-16T05:34:58.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- start start 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: init 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): picked mon.noname-c con 0x7f62500a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): picked mon.noname-b con 0x7f62500a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): picked mon.noname-a con 0x7f62500a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62501c60f0 con 0x7f62500a2980 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f625009e4d0 con 0x7f62500a2570 2024-09-16T05:34:58.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62500a0100 con 0x7f62500a2190 2024-09-16T05:34:58.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.119+0000 7f6256f50640 10 monclient(hunting): authenticate will time out at 959.932000s 2024-09-16T05:34:58.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f62554ed640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62500a2980 0x7f625013f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58614/0 (socket says 172.21.15.6:58614) 2024-09-16T05:34:58.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f62554ed640 1 -- 172.21.15.6:0/4251657633 learned_addr learned my addr 172.21.15.6:0/4251657633 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270012452 0 0) 0x7f62501c60f0 con 0x7f62500a2980 2024-09-16T05:34:58.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): my global_id is 4275 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f623c003530 con 0x7f62500a2980 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353584828 0 0) 0x7f625009e4d0 con 0x7f62500a2570 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): my global_id is 4276 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62501c60f0 con 0x7f62500a2570 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 254849601 0 0) 0x7f623c003530 con 0x7f62500a2980 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f625009e4d0 con 0x7f62500a2980 2024-09-16T05:34:58.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6248001bd0 con 0x7f62500a2980 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): dump: 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3352300229 0 0) 0x7f625009e4d0 con 0x7f62500a2980 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 >> v1:172.21.15.6:6790/0 conn(0x7f62500a2190 legacy=0x7f625009c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 >> v1:172.21.15.63:6789/0 conn(0x7f62500a2570 legacy=0x7f625009cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 monclient: found mon.a 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62500a8460 con 0x7f62500a2980 2024-09-16T05:34:58.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 5 monclient: authenticate success, global_id 4275 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 10 monclient: _renew_subs 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f62501c5e10 con 0x7f62500a2980 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 10 monclient: _renew_subs 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f625009d4a0 con 0x7f62500a2980 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6248001880 con 0x7f62500a2980 2024-09-16T05:34:58.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624effd640 4 set_mon_vals no callback set 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.6:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6230000f90 con 0x7f62500a2980 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6248001a30 con 0x7f62500a2980 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: dump: 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.172 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f624800f800 con 0x7f62500a2980 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f6248042900 con 0x7f62500a2980 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.6:6800/1572197727 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f623c042420 con 0x7f623c03e710 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f6254cec640 10 monclient: get_auth_request con 0x7f623c037220 auth_method 0 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624ffff640 10 monclient: get_auth_request con 0x7f623c03e710 auth_method 0 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f62480112b0 con 0x7f62500a2980 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.123+0000 7f624d7fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T05:34:58.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.127+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== osd.2 v1:172.21.15.6:6800/1572197727 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f623c042420 con 0x7f623c03e710 2024-09-16T05:34:58.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.127+0000 7f987ffff640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.127+0000 7f987ffff640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.127+0000 7f987ffff640 10 monclient: monmap: 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.134357+0000 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.134357+0000 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 Processor -- start 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- start start 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient: init 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f98780297a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f9878027740 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f9878021ba0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9878024630 con 0x7f9878027740 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9878022260 con 0x7f98780297a0 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9878023d20 con 0x7f9878021ba0 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient(hunting): authenticate will time out at 959.940974s 2024-09-16T05:34:58.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987effd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f98780297a0 0x7f9878027030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.6:43464/0 (socket says 172.21.15.6:43464) 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987effd640 1 -- 172.21.15.6:0/987677803 learned_addr learned my addr 172.21.15.6:0/987677803 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2489639079 0 0) 0x7f9874001120 con 0x7f9878027740 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): my global_id is 4278 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9868003530 con 0x7f9878027740 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741562010 0 0) 0x7f9878022260 con 0x7f98780297a0 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): my global_id is 4279 2024-09-16T05:34:58.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9874001120 con 0x7f98780297a0 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 619428505 0 0) 0x7f9878023d20 con 0x7f9878021ba0 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient(hunting): my global_id is 4274 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9878022260 con 0x7f9878021ba0 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3265347515 0 0) 0x7f9868003530 con 0x7f9878027740 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 >> v1:172.21.15.6:6790/0 conn(0x7f9878021ba0 legacy=0x7f98780c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 >> v1:172.21.15.63:6789/0 conn(0x7f98780297a0 legacy=0x7f9878027030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 monclient: found mon.noname-a 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9878024000 con 0x7f9878027740 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98740017f0 con 0x7f9878027740 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: dump: 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.181 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 5 monclient: authenticate success, global_id 4278 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9874001a50 con 0x7f9878027740 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 1 -- 172.21.15.6:0/987677803 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9874003600 con 0x7f9878027740 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987dffb640 10 monclient: dump: 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.183 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.184 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 4 set_mon_vals no callback set 2024-09-16T05:34:58.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient: monmap: 2024-09-16T05:34:58.185 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.185 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.137360+0000 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.137360+0000 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 1 Processor -- start 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 1 -- start start 2024-09-16T05:34:58.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient: init 2024-09-16T05:34:58.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 10 monclient: shutdown 2024-09-16T05:34:58.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- 172.21.15.6:0/987677803 >> v1:172.21.15.6:6789/0 conn(0x7f9878027740 legacy=0x7f98780c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- 172.21.15.6:0/987677803 shutdown_connections 2024-09-16T05:34:58.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- 172.21.15.6:0/987677803 >> 172.21.15.6:0/987677803 conn(0x7f9878006660 msgr2=0x7f9878028080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f3a18c4c640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f3a18c4c640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f3a18c4c640 10 monclient: monmap: 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.138377+0000 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.138377+0000 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.188 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.189 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- 172.21.15.6:0/987677803 shutdown_connections 2024-09-16T05:34:58.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f987ffff640 1 -- 172.21.15.6:0/987677803 wait complete. 2024-09-16T05:34:58.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0a0c06bc10 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient(hunting): picked mon.noname-c con 0x7f0a0c06bff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0a0c13cb10 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a0c09fe30 con 0x7f0a0c13cb10 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a1156c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0a0c06bc10 0x7f0a0c136a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33716/0 (socket says 172.21.15.6:33716) 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a1156c640 1 -- 172.21.15.6:0/845175239 learned_addr learned my addr 172.21.15.6:0/845175239 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 1 -- 172.21.15.6:0/845175239 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a0c06ef70 con 0x7f0a0c06bff0 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 1 -- 172.21.15.6:0/845175239 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a0c0a16f0 con 0x7f0a0c06bc10 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.131+0000 7f0a137d0640 10 monclient(hunting): authenticate will time out at 959.943999s 2024-09-16T05:34:58.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133589662 0 0) 0x7f0a0c09fe30 con 0x7f0a0c13cb10 2024-09-16T05:34:58.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): my global_id is 4281 2024-09-16T05:34:58.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09f8003530 con 0x7f0a0c13cb10 2024-09-16T05:34:58.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2533233376 0 0) 0x7f0a0c0a16f0 con 0x7f0a0c06bc10 2024-09-16T05:34:58.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): my global_id is 4277 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a0c09fe30 con 0x7f0a0c06bc10 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446463987 0 0) 0x7f0a0c06ef70 con 0x7f0a0c06bff0 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient(hunting): my global_id is 4282 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a0c0a16f0 con 0x7f0a0c06bff0 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: monmap: 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.194 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3879760222 0 0) 0x7f09f8003530 con 0x7f0a0c13cb10 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 >> v1:172.21.15.6:6790/0 conn(0x7f0a0c06bc10 legacy=0x7f0a0c136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 >> v1:172.21.15.63:6789/0 conn(0x7f0a0c06bff0 legacy=0x7f0a0c06c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 monclient: found mon.noname-a 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0c143650 con 0x7f0a0c13cb10 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a040019a0 con 0x7f0a0c13cb10 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: dump: 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.197 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 5 monclient: authenticate success, global_id 4281 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a04001c00 con 0x7f0a0c13cb10 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 1 -- 172.21.15.6:0/845175239 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a04001f50 con 0x7f0a0c13cb10 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 4 set_mon_vals no callback set 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a03fff640 10 monclient: dump: 2024-09-16T05:34:58.198 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 1 Processor -- start 2024-09-16T05:34:58.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 1 Processor -- start 2024-09-16T05:34:58.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 1 -- start start 2024-09-16T05:34:58.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: init 2024-09-16T05:34:58.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 1 -- start start 2024-09-16T05:34:58.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 10 monclient: init 2024-09-16T05:34:58.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:34:58.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f9878021ba0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f9878027740 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f98780297a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987802bc00 con 0x7f9878021ba0 2024-09-16T05:34:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98780beb20 con 0x7f9878027740 2024-09-16T05:34:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98780c0e80 con 0x7f98780297a0 2024-09-16T05:34:58.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:58.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient(hunting): authenticate will time out at 959.945671s 2024-09-16T05:34:58.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9878021ba0 0x7f98780ae140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58650/0 (socket says 172.21.15.6:58650) 2024-09-16T05:34:58.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987effd640 1 -- 172.21.15.6:0/9973631 learned_addr learned my addr 172.21.15.6:0/9973631 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 10 monclient(hunting): picked mon.noname-c con 0x7f3a1409f860 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 10 monclient(hunting): picked mon.noname-a con 0x7f3a140a0350 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 10 monclient: shutdown 2024-09-16T05:34:58.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 1 -- 172.21.15.6:0/845175239 >> v1:172.21.15.6:6789/0 conn(0x7f0a0c13cb10 legacy=0x7f0a0c13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 10 monclient(hunting): picked mon.noname-b con 0x7f3a1413b1b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a14072db0 con 0x7f3a140a0350 2024-09-16T05:34:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a12d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3a1413b1b0 0x7f3a1413d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33722/0 (socket says 172.21.15.6:33722) 2024-09-16T05:34:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a12d9d640 1 -- 172.21.15.6:0/190378813 learned_addr learned my addr 172.21.15.6:0/190378813 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 1 -- 172.21.15.6:0/845175239 shutdown_connections 2024-09-16T05:34:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 1 -- 172.21.15.6:0/845175239 >> 172.21.15.6:0/845175239 conn(0x7f0a0c0a2580 msgr2=0x7f0a0c0a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 1 -- 172.21.15.6:0/845175239 shutdown_connections 2024-09-16T05:34:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 1 -- 172.21.15.6:0/845175239 wait complete. 2024-09-16T05:34:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3155319378 0 0) 0x7f98780c0e80 con 0x7f98780297a0 2024-09-16T05:34:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): my global_id is 4280 2024-09-16T05:34:58.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f986c003530 con 0x7f98780297a0 2024-09-16T05:34:58.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 1 -- 172.21.15.6:0/190378813 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a140a1430 con 0x7f3a1409f860 2024-09-16T05:34:58.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 1 -- 172.21.15.6:0/190378813 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a140a2780 con 0x7f3a1413b1b0 2024-09-16T05:34:58.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 10 monclient(hunting): authenticate will time out at 959.946448s 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2220519065 0 0) 0x7f98780beb20 con 0x7f9878027740 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): my global_id is 4285 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98780c0e80 con 0x7f9878027740 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2134039480 0 0) 0x7f987802bc00 con 0x7f9878021ba0 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): my global_id is 4284 2024-09-16T05:34:58.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98780beb20 con 0x7f9878021ba0 2024-09-16T05:34:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904713877 0 0) 0x7f3a140a2780 con 0x7f3a1413b1b0 2024-09-16T05:34:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient(hunting): my global_id is 4283 2024-09-16T05:34:58.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39fc003530 con 0x7f3a1413b1b0 2024-09-16T05:34:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4169721828 0 0) 0x7f98780beb20 con 0x7f9878021ba0 2024-09-16T05:34:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f987802bc00 con 0x7f9878021ba0 2024-09-16T05:34:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9870001920 con 0x7f9878021ba0 2024-09-16T05:34:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): dump: 2024-09-16T05:34:58.210 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.211 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.211 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.211 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.211 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.211 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.212 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2428490493 0 0) 0x7f986c003530 con 0x7f98780297a0 2024-09-16T05:34:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98780beb20 con 0x7f98780297a0 2024-09-16T05:34:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9880069850 con 0x7f98780297a0 2024-09-16T05:34:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): dump: 2024-09-16T05:34:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.214 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.215 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.215 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.215 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.215 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.215 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.215 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.215 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 10 monclient: monmap: 2024-09-16T05:34:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.216 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.217 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.217 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.217 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1246802829 0 0) 0x7f98780c0e80 con 0x7f9878027740 2024-09-16T05:34:58.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f986c003530 con 0x7f9878027740 2024-09-16T05:34:58.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9874001c00 con 0x7f9878027740 2024-09-16T05:34:58.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient(hunting): dump: 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.218 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3616364350 0 0) 0x7f3a140a1430 con 0x7f3a1409f860 2024-09-16T05:34:58.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient(hunting): my global_id is 4288 2024-09-16T05:34:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a140a2780 con 0x7f3a1409f860 2024-09-16T05:34:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3292196760 0 0) 0x7f39fc003530 con 0x7f3a1413b1b0 2024-09-16T05:34:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 >> v1:172.21.15.63:6789/0 conn(0x7f3a1409f860 legacy=0x7f3a1409fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 >> v1:172.21.15.6:6789/0 conn(0x7f3a140a0350 legacy=0x7f3a14139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2505393176 0 0) 0x7f98780beb20 con 0x7f98780297a0 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 >> v1:172.21.15.63:6789/0 conn(0x7f9878027740 legacy=0x7f98781559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 >> v1:172.21.15.6:6789/0 conn(0x7f9878021ba0 legacy=0x7f98780ae140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 monclient: found mon.c 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98780bf200 con 0x7f98780297a0 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f9867fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 5 monclient: authenticate success, global_id 4280 2024-09-16T05:34:58.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: _renew_subs 2024-09-16T05:34:58.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f98780c0290 con 0x7f98780297a0 2024-09-16T05:34:58.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: _renew_subs 2024-09-16T05:34:58.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f98780be1d0 con 0x7f98780297a0 2024-09-16T05:34:58.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 monclient: found mon.noname-b 2024-09-16T05:34:58.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1409f0f0 con 0x7f3a1413b1b0 2024-09-16T05:34:58.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 1 Processor -- start 2024-09-16T05:34:58.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 1 -- start start 2024-09-16T05:34:58.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 5 monclient: authenticate success, global_id 4283 2024-09-16T05:34:58.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3a0c0018d0 con 0x7f3a1413b1b0 2024-09-16T05:34:58.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:58.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a1159a640 10 monclient: dump: 2024-09-16T05:34:58.226 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.226 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.226 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.227 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f3a18c4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.135+0000 7f0a137d0640 10 monclient: init 2024-09-16T05:34:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient(hunting): picked mon.noname-c con 0x7f0a0c06bc10 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0a0c06bff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0a0c13cb10 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a0c140410 con 0x7f0a0c13cb10 2024-09-16T05:34:58.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a0c13a8a0 con 0x7f0a0c06bff0 2024-09-16T05:34:58.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a0c070a80 con 0x7f0a0c06bc10 2024-09-16T05:34:58.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient(hunting): authenticate will time out at 959.948450s 2024-09-16T05:34:58.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f987ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f98780c53c0 con 0x7f98780297a0 2024-09-16T05:34:58.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3a0c001b30 con 0x7f3a1413b1b0 2024-09-16T05:34:58.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a1159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a1159a640 1 -- 172.21.15.6:0/190378813 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3a0c001f00 con 0x7f3a1413b1b0 2024-09-16T05:34:58.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a1159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a1159a640 10 monclient: dump: 2024-09-16T05:34:58.230 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.231 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.231 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.231 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.231 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.231 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.231 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.232 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.232 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.232 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 4 set_mon_vals no callback set 2024-09-16T05:34:58.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a1156c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0a0c06bc10 0x7f0a0c138c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33726/0 (socket says 172.21.15.6:33726) 2024-09-16T05:34:58.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a11d6d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0a0c13cb10 0x7f0a0c1d8600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58676/0 (socket says 172.21.15.6:58676) 2024-09-16T05:34:58.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a1156c640 1 -- 172.21.15.6:0/2190978552 learned_addr learned my addr 172.21.15.6:0/2190978552 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9880067a00 con 0x7f98780297a0 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9880068cb0 con 0x7f98780297a0 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 10 monclient: dump: 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.234 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.235 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.235 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.235 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.235 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.235 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9880068e90 con 0x7f98780297a0 2024-09-16T05:34:58.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294194283 0 0) 0x7f0a0c140410 con 0x7f0a0c13cb10 2024-09-16T05:34:58.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): my global_id is 4290 2024-09-16T05:34:58.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09f0003530 con 0x7f0a0c13cb10 2024-09-16T05:34:58.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547524675 0 0) 0x7f0a0c070a80 con 0x7f0a0c06bc10 2024-09-16T05:34:58.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): my global_id is 4286 2024-09-16T05:34:58.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a0c140410 con 0x7f0a0c06bc10 2024-09-16T05:34:58.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f98800be3a0 con 0x7f98780297a0 2024-09-16T05:34:58.240 INFO:teuthology.orchestra.run.smithi006.stdout:47244640261 2024-09-16T05:34:58.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T05:34:58.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931474791 0 0) 0x7f0a0c13a8a0 con 0x7f0a0c06bff0 2024-09-16T05:34:58.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): my global_id is 4291 2024-09-16T05:34:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a0c070a80 con 0x7f0a0c06bff0 2024-09-16T05:34:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4186364721 0 0) 0x7f0a0c140410 con 0x7f0a0c06bc10 2024-09-16T05:34:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a0c13a8a0 con 0x7f0a0c06bc10 2024-09-16T05:34:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a08001980 con 0x7f0a0c06bc10 2024-09-16T05:34:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): dump: 2024-09-16T05:34:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.245 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.63:6804/118134224 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f986c041530 con 0x7f986c03d880 2024-09-16T05:34:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f98800be6d0 con 0x7f98780297a0 2024-09-16T05:34:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f9867fff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T05:34:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f987d7fa640 4 set_mon_vals no callback set 2024-09-16T05:34:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3525805812 0 0) 0x7f09f0003530 con 0x7f0a0c13cb10 2024-09-16T05:34:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a0c140410 con 0x7f0a0c13cb10 2024-09-16T05:34:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a04001700 con 0x7f0a0c13cb10 2024-09-16T05:34:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient(hunting): dump: 2024-09-16T05:34:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.249 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f987effd640 10 monclient: get_auth_request con 0x7f986c036d40 auth_method 0 2024-09-16T05:34:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient: shutdown 2024-09-16T05:34:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- 172.21.15.6:0/190378813 >> v1:172.21.15.6:6790/0 conn(0x7f3a1413b1b0 legacy=0x7f3a1413d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3273866980 0 0) 0x7f0a0c13a8a0 con 0x7f0a0c06bc10 2024-09-16T05:34:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 >> v1:172.21.15.63:6789/0 conn(0x7f0a0c06bff0 legacy=0x7f0a0c139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 >> v1:172.21.15.6:6789/0 conn(0x7f0a0c13cb10 legacy=0x7f0a0c1d8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 monclient: found mon.c 2024-09-16T05:34:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0c0701e0 con 0x7f0a0c06bc10 2024-09-16T05:34:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- 172.21.15.6:0/190378813 shutdown_connections 2024-09-16T05:34:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- 172.21.15.6:0/190378813 >> 172.21.15.6:0/190378813 conn(0x7f3a140a5540 msgr2=0x7f3a140a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f987e7fc640 10 monclient: get_auth_request con 0x7f986c03d880 auth_method 0 2024-09-16T05:34:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 5 monclient: authenticate success, global_id 4286 2024-09-16T05:34:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient: _renew_subs 2024-09-16T05:34:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0c13b620 con 0x7f0a0c06bc10 2024-09-16T05:34:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient: _renew_subs 2024-09-16T05:34:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0a0c139a20 con 0x7f0a0c06bc10 2024-09-16T05:34:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- 172.21.15.6:0/190378813 shutdown_connections 2024-09-16T05:34:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- 172.21.15.6:0/190378813 wait complete. 2024-09-16T05:34:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a08001c80 con 0x7f0a0c06bc10 2024-09-16T05:34:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a08004c40 con 0x7f0a0c06bc10 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a037fe640 4 set_mon_vals no callback set 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: dump: 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.258 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f09ec000f90 con 0x7f0a0c06bc10 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient: monmap: 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 Processor -- start 2024-09-16T05:34:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- start start 2024-09-16T05:34:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient: init 2024-09-16T05:34:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0a0800f9e0 con 0x7f0a0c06bc10 2024-09-16T05:34:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f0a08044500 con 0x7f0a0c06bc10 2024-09-16T05:34:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6804/3069057739 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09f0041530 con 0x7f09f003d880 2024-09-16T05:34:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f0a08044700 con 0x7f0a0c06bc10 2024-09-16T05:34:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a01ffb640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T05:34:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient(hunting): picked mon.noname-a con 0x7f3a1409f860 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient(hunting): picked mon.noname-b con 0x7f3a140a0350 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient(hunting): picked mon.noname-c con 0x7f3a1413b1b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1411fbe0 con 0x7f3a1409f860 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1411fec0 con 0x7f3a140a0350 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1406e630 con 0x7f3a1413b1b0 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a18c4c640 10 monclient(hunting): authenticate will time out at 959.951620s 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a11d6d640 10 monclient: get_auth_request con 0x7f09f003d880 auth_method 0 2024-09-16T05:34:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f0a10d6b640 10 monclient: get_auth_request con 0x7f09f0036cc0 auth_method 0 2024-09-16T05:34:58.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a1259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3a1409f860 0x7f3a141c96e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58690/0 (socket says 172.21.15.6:58690) 2024-09-16T05:34:58.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.139+0000 7f3a1259c640 1 -- 172.21.15.6:0/3822348590 learned_addr learned my addr 172.21.15.6:0/3822348590 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 484175181 0 0) 0x7f3a1406e630 con 0x7f3a1413b1b0 2024-09-16T05:34:58.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): my global_id is 4289 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39f4003530 con 0x7f3a1413b1b0 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 103758586 0 0) 0x7f3a1411fbe0 con 0x7f3a1409f860 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): my global_id is 4293 2024-09-16T05:34:58.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a1406e630 con 0x7f3a1409f860 2024-09-16T05:34:58.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== osd.4 v1:172.21.15.63:6804/118134224 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f986c041530 con 0x7f986c03d880 2024-09-16T05:34:58.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 354172722 0 0) 0x7f39f4003530 con 0x7f3a1413b1b0 2024-09-16T05:34:58.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a1411fbe0 con 0x7f3a1413b1b0 2024-09-16T05:34:58.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3a0c0017d0 con 0x7f3a1413b1b0 2024-09-16T05:34:58.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:58.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): dump: 2024-09-16T05:34:58.267 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.268 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.268 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.268 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.268 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.268 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.268 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.268 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.268 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1663761090 0 0) 0x7f3a1406e630 con 0x7f3a1409f860 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39f4003530 con 0x7f3a1409f860 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3a04001940 con 0x7f3a1409f860 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient(hunting): dump: 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.269 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3761807012 0 0) 0x7f3a1411fbe0 con 0x7f3a1413b1b0 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 >> v1:172.21.15.63:6789/0 conn(0x7f3a140a0350 legacy=0x7f3a141ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 >> v1:172.21.15.6:6789/0 conn(0x7f3a1409f860 legacy=0x7f3a141c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 monclient: found mon.c 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1413abe0 con 0x7f3a1413b1b0 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a18c4c640 5 monclient: authenticate success, global_id 4289 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a18c4c640 10 monclient: _renew_subs 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a18c4c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1411f900 con 0x7f3a1413b1b0 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a18c4c640 10 monclient: _renew_subs 2024-09-16T05:34:58.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a18c4c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3a1411cf40 con 0x7f3a1413b1b0 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3a0c004e20 con 0x7f3a1413b1b0 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3a0c005170 con 0x7f3a1413b1b0 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 10 monclient: dump: 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.272 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a10d99640 4 set_mon_vals no callback set 2024-09-16T05:34:58.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3a0c00ff10 con 0x7f3a1413b1b0 2024-09-16T05:34:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f3a0c044730 con 0x7f3a1413b1b0 2024-09-16T05:34:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6808/4098222447 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a1409f330 con 0x7f3a140713f0 2024-09-16T05:34:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a11d9b640 10 monclient: get_auth_request con 0x7f3a140713f0 auth_method 0 2024-09-16T05:34:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a1259c640 10 monclient: get_auth_request con 0x7f39f4036a70 auth_method 0 2024-09-16T05:34:58.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.143+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== osd.1 v1:172.21.15.6:6808/4098222447 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3a1409f330 con 0x7f3a140713f0 2024-09-16T05:34:58.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.147+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== osd.0 v1:172.21.15.6:6804/3069057739 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f09f0041530 con 0x7f09f003d880 2024-09-16T05:34:58.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.147+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 --> v1:172.21.15.6:6800/1572197727 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6230002c60 con 0x7f623c03e710 2024-09-16T05:34:58.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.147+0000 7f624d7fa640 1 -- 172.21.15.6:0/4251657633 <== osd.2 v1:172.21.15.6:6800/1572197727 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7f6230002c60 con 0x7f623c03e710 2024-09-16T05:34:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 >> v1:172.21.15.6:6800/1572197727 conn(0x7f623c03e710 legacy=0x7f623c040bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 >> v1:172.21.15.63:6812/3643207211 conn(0x7f623c037220 legacy=0x7f623c0396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 10 monclient: shutdown 2024-09-16T05:34:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 >> v1:172.21.15.6:6789/0 conn(0x7f62500a2980 legacy=0x7f625013f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f62554ed640 1 -- 172.21.15.6:0/4251657633 reap_dead start 2024-09-16T05:34:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 shutdown_connections 2024-09-16T05:34:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 >> 172.21.15.6:0/4251657633 conn(0x7f62500a9850 msgr2=0x7f625006ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 shutdown_connections 2024-09-16T05:34:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.151+0000 7f6256f50640 1 -- 172.21.15.6:0/4251657633 wait complete. 2024-09-16T05:34:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.155+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 --> v1:172.21.15.63:6804/118134224 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f98780ae850 con 0x7f986c03d880 2024-09-16T05:34:58.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f9867fff640 1 -- 172.21.15.6:0/9973631 <== osd.4 v1:172.21.15.63:6804/118134224 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3543570501) 0x7f98780ae850 con 0x7f986c03d880 2024-09-16T05:34:58.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 >> v1:172.21.15.63:6804/118134224 conn(0x7f986c03d880 legacy=0x7f986c03fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 >> v1:172.21.15.63:6812/3643207211 conn(0x7f986c036d40 legacy=0x7f986c039200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 10 monclient: shutdown 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 >> v1:172.21.15.6:6790/0 conn(0x7f98780297a0 legacy=0x7f98781590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987f7fe640 1 -- 172.21.15.6:0/9973631 reap_dead start 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 shutdown_connections 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 >> 172.21.15.6:0/9973631 conn(0x7f9878006660 msgr2=0x7f98780bb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 shutdown_connections 2024-09-16T05:34:58.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.159+0000 7f987ffff640 1 -- 172.21.15.6:0/9973631 wait complete. 2024-09-16T05:34:58.280 INFO:teuthology.orchestra.run.smithi006.stdout:47244640261 2024-09-16T05:34:58.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T05:34:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 --> v1:172.21.15.6:6808/4098222447 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3a1411d1d0 con 0x7f3a140713f0 2024-09-16T05:34:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a02ffd640 1 -- 172.21.15.6:0/3822348590 <== osd.1 v1:172.21.15.6:6808/4098222447 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7f3a1411d1d0 con 0x7f3a140713f0 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 >> v1:172.21.15.6:6808/4098222447 conn(0x7f3a140713f0 legacy=0x7f3a140626e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 >> v1:172.21.15.63:6812/3643207211 conn(0x7f39f4036a70 legacy=0x7f39f4038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 10 monclient: shutdown 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 >> v1:172.21.15.6:6790/0 conn(0x7f3a1413b1b0 legacy=0x7f3a141d1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a12d9d640 1 -- 172.21.15.6:0/3822348590 reap_dead start 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 shutdown_connections 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 >> 172.21.15.6:0/3822348590 conn(0x7f3a140a5540 msgr2=0x7f3a140a2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 shutdown_connections 2024-09-16T05:34:58.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.171+0000 7f3a18c4c640 1 -- 172.21.15.6:0/3822348590 wait complete. 2024-09-16T05:34:58.283 INFO:teuthology.orchestra.run.smithi006.stdout:51539607557 2024-09-16T05:34:58.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T05:34:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.175+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 --> v1:172.21.15.6:6804/3069057739 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f09ec002c60 con 0x7f09f003d880 2024-09-16T05:34:58.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.175+0000 7f0a01ffb640 1 -- 172.21.15.6:0/2190978552 <== osd.0 v1:172.21.15.6:6804/3069057739 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7f09ec002c60 con 0x7f09f003d880 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.175+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 >> v1:172.21.15.6:6804/3069057739 conn(0x7f09f003d880 legacy=0x7f09f003fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 >> v1:172.21.15.63:6812/3643207211 conn(0x7f09f0036cc0 legacy=0x7f09f0039180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a137d0640 10 monclient: shutdown 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a137d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 >> v1:172.21.15.6:6790/0 conn(0x7f0a0c06bc10 legacy=0x7f0a0c138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a11d6d640 1 -- 172.21.15.6:0/2190978552 reap_dead start 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 shutdown_connections 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 >> 172.21.15.6:0/2190978552 conn(0x7f0a0c0a2580 msgr2=0x7f0a0c1358b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 shutdown_connections 2024-09-16T05:34:58.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.179+0000 7f0a137d0640 1 -- 172.21.15.6:0/2190978552 wait complete. 2024-09-16T05:34:58.286 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2024-09-16T05:34:58.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T05:34:58.287 INFO:teuthology.orchestra.run.smithi006.stdout:47244640261 2024-09-16T05:34:58.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T05:34:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient: monmap: 2024-09-16T05:34:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.404591+0000 2024-09-16T05:34:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.404591+0000 2024-09-16T05:34:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 1 Processor -- start 2024-09-16T05:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 1 -- start start 2024-09-16T05:34:58.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient: init 2024-09-16T05:34:58.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:34:58.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8240a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient(hunting): picked mon.noname-c con 0x7fa8240a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient(hunting): picked mon.noname-b con 0x7fa8240a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8240a33a0 con 0x7fa8240a2190 2024-09-16T05:34:58.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8240a07a0 con 0x7fa8240a2570 2024-09-16T05:34:58.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8240a2d90 con 0x7fa8240a2980 2024-09-16T05:34:58.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa828c3f640 10 monclient(hunting): authenticate will time out at 960.210924s 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8240a2190 0x7fa8240a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58698/0 (socket says 172.21.15.6:58698) 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa822d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa8240a2980 0x7fa82413f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33736/0 (socket says 172.21.15.6:33736) 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82259c640 1 -- 172.21.15.6:0/3643669023 learned_addr learned my addr 172.21.15.6:0/3643669023 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 791608811 0 0) 0x7fa8240a2d90 con 0x7fa8240a2980 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): my global_id is 4292 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa80c003530 con 0x7fa8240a2980 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 671912122 0 0) 0x7fa8240a33a0 con 0x7fa8240a2190 2024-09-16T05:34:58.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): my global_id is 4296 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8240a2d90 con 0x7fa8240a2190 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125859558 0 0) 0x7fa8240a07a0 con 0x7fa8240a2570 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient(hunting): my global_id is 4297 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8240a33a0 con 0x7fa8240a2570 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 340328001 0 0) 0x7fa8240a2d90 con 0x7fa8240a2190 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 >> v1:172.21.15.6:6790/0 conn(0x7fa8240a2980 legacy=0x7fa82413f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 >> v1:172.21.15.63:6789/0 conn(0x7fa8240a2570 legacy=0x7fa82413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.399+0000 7fa82159a640 1 monclient: found mon.noname-a 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8240a66e0 con 0x7fa8240a2190 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa814001530 con 0x7fa8240a2190 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 5 monclient: authenticate success, global_id 4296 2024-09-16T05:34:58.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: dump: 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.417 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.418 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.418 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa814001800 con 0x7fa8240a2190 2024-09-16T05:34:58.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 1 -- 172.21.15.6:0/3643669023 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa814001bd0 con 0x7fa8240a2190 2024-09-16T05:34:58.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82159a640 10 monclient: dump: 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 4 set_mon_vals no callback set 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient: shutdown 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- 172.21.15.6:0/3643669023 >> v1:172.21.15.6:6789/0 conn(0x7fa8240a2190 legacy=0x7fa8240a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- 172.21.15.6:0/3643669023 shutdown_connections 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- 172.21.15.6:0/3643669023 >> 172.21.15.6:0/3643669023 conn(0x7fa8240a9850 msgr2=0x7fa8240a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- 172.21.15.6:0/3643669023 shutdown_connections 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- 172.21.15.6:0/3643669023 wait complete. 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient: monmap: 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.420 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 Processor -- start 2024-09-16T05:34:58.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- start start 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient: init 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8240a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient(hunting): picked mon.noname-b con 0x7fa8240a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient(hunting): picked mon.noname-c con 0x7fa8240a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8240a2190 0x7fa8241209b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58706/0 (socket says 172.21.15.6:58706) 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa82259c640 1 -- 172.21.15.6:0/2257293844 learned_addr learned my addr 172.21.15.6:0/2257293844 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa824122520 con 0x7fa8240a2190 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa824123f50 con 0x7fa8240a2570 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa824124510 con 0x7fa8240a2980 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 511386067 0 0) 0x7fa824122520 con 0x7fa8240a2190 2024-09-16T05:34:58.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa828c3f640 10 monclient(hunting): authenticate will time out at 960.215638s 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): my global_id is 4299 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa804003530 con 0x7fa8240a2190 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153875094 0 0) 0x7fa824124510 con 0x7fa8240a2980 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): my global_id is 4295 2024-09-16T05:34:58.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa824122520 con 0x7fa8240a2980 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881814907 0 0) 0x7fa824123f50 con 0x7fa8240a2570 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 10 monclient(hunting): my global_id is 4300 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.403+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa824124510 con 0x7fa8240a2570 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1553531949 0 0) 0x7fa824122520 con 0x7fa8240a2980 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa824123f50 con 0x7fa8240a2980 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa81c0014a0 con 0x7fa8240a2980 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient(hunting): dump: 2024-09-16T05:34:58.424 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1544581108 0 0) 0x7fa804003530 con 0x7fa8240a2190 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa824122520 con 0x7fa8240a2190 2024-09-16T05:34:58.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa8140047f0 con 0x7fa8240a2190 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient(hunting): dump: 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.426 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2792251241 0 0) 0x7fa824123f50 con 0x7fa8240a2980 2024-09-16T05:34:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 >> v1:172.21.15.63:6789/0 conn(0x7fa8240a2570 legacy=0x7fa8241210c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 >> v1:172.21.15.6:6789/0 conn(0x7fa8240a2190 legacy=0x7fa8241209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 monclient: found mon.c 2024-09-16T05:34:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa824124230 con 0x7fa8240a2980 2024-09-16T05:34:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 5 monclient: authenticate success, global_id 4295 2024-09-16T05:34:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa81c004b70 con 0x7fa8240a2980 2024-09-16T05:34:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa820d99640 4 set_mon_vals no callback set 2024-09-16T05:34:58.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 10 monclient: _renew_subs 2024-09-16T05:34:58.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa81c004ec0 con 0x7fa8240a2980 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa82412d5a0 con 0x7fa8240a2980 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 10 monclient: dump: 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.430 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 10 monclient: _renew_subs 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa824140450 con 0x7fa8240a2980 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa81c00fc60 con 0x7fa8240a2980 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7fa81c0442d0 con 0x7fa8240a2980 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8240008e0 con 0x7fa8240a2980 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.407+0000 7fa82259c640 10 monclient: get_auth_request con 0x7fa804036b80 auth_method 0 2024-09-16T05:34:58.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.415+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa81c015560 con 0x7fa8240a2980 2024-09-16T05:34:58.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.415+0000 7fa812ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.415+0000 7fa812ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient: monmap: 2024-09-16T05:34:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.444132+0000 2024-09-16T05:34:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.444132+0000 2024-09-16T05:34:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.445 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.446 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.446 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.446 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 1 Processor -- start 2024-09-16T05:34:58.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 1 -- start start 2024-09-16T05:34:58.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient: init 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient(hunting): picked mon.noname-c con 0x7f9d400a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient(hunting): picked mon.noname-a con 0x7f9d400a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient(hunting): picked mon.noname-b con 0x7f9d400a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d400a33a0 con 0x7f9d400a2570 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d400a07a0 con 0x7f9d400a2190 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d400a2d90 con 0x7f9d400a2980 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 10 monclient(hunting): authenticate will time out at 960.250437s 2024-09-16T05:34:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9d400a2570 0x7f9d4013b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58718/0 (socket says 172.21.15.6:58718) 2024-09-16T05:34:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3dd9b640 1 -- 172.21.15.6:0/2618082481 learned_addr learned my addr 172.21.15.6:0/2618082481 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1648287652 0 0) 0x7f9d400a33a0 con 0x7f9d400a2570 2024-09-16T05:34:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): my global_id is 4302 2024-09-16T05:34:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d28003530 con 0x7f9d400a2570 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522895090 0 0) 0x7f9d400a2d90 con 0x7f9d400a2980 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): my global_id is 4298 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d400a33a0 con 0x7f9d400a2980 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346977125 0 0) 0x7f9d400a07a0 con 0x7f9d400a2190 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient(hunting): my global_id is 4303 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d400a2d90 con 0x7f9d400a2190 2024-09-16T05:34:58.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 87062829 0 0) 0x7f9d28003530 con 0x7f9d400a2570 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 >> v1:172.21.15.6:6790/0 conn(0x7f9d400a2980 legacy=0x7f9d4013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 >> v1:172.21.15.63:6789/0 conn(0x7f9d400a2190 legacy=0x7f9d400a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 monclient: found mon.noname-a 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d400a66e0 con 0x7f9d400a2570 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9d34001620 con 0x7f9d400a2570 2024-09-16T05:34:58.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 10 monclient: dump: 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 5 monclient: authenticate success, global_id 4302 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d44b42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.439+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d34001940 con 0x7f9d400a2570 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d3d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d3d59a640 1 -- 172.21.15.6:0/2618082481 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9d34001d10 con 0x7f9d400a2570 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d3d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d3d59a640 10 monclient: dump: 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.453 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 4 set_mon_vals no callback set 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient: shutdown 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- 172.21.15.6:0/2618082481 >> v1:172.21.15.6:6789/0 conn(0x7f9d400a2570 legacy=0x7f9d4013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- 172.21.15.6:0/2618082481 shutdown_connections 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- 172.21.15.6:0/2618082481 >> 172.21.15.6:0/2618082481 conn(0x7f9d400a9850 msgr2=0x7f9d400a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- 172.21.15.6:0/2618082481 shutdown_connections 2024-09-16T05:34:58.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- 172.21.15.6:0/2618082481 wait complete. 2024-09-16T05:34:58.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient: monmap: 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 Processor -- start 2024-09-16T05:34:58.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- start start 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient: init 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient(hunting): picked mon.noname-a con 0x7f9d400a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient(hunting): picked mon.noname-b con 0x7f9d400a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient(hunting): picked mon.noname-c con 0x7f9d400a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d401c6310 con 0x7f9d400a2190 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d401c5c10 con 0x7f9d400a2570 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d401c5ef0 con 0x7f9d400a2980 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d44b42640 10 monclient(hunting): authenticate will time out at 960.254464s 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d3e59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9d400a2190 0x7f9d4009c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58726/0 (socket says 172.21.15.6:58726) 2024-09-16T05:34:58.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d3e59c640 1 -- 172.21.15.6:0/119282032 learned_addr learned my addr 172.21.15.6:0/119282032 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263680233 0 0) 0x7f9d400a2d90 con 0x7f9d400a2190 2024-09-16T05:34:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): my global_id is 4305 2024-09-16T05:34:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d20003530 con 0x7f9d400a2190 2024-09-16T05:34:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412373273 0 0) 0x7f9d401c5ef0 con 0x7f9d400a2980 2024-09-16T05:34:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): my global_id is 4301 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d400a2d90 con 0x7f9d400a2980 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1626207683 0 0) 0x7f9d20003530 con 0x7f9d400a2190 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d401c5ef0 con 0x7f9d400a2190 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9d300018c0 con 0x7f9d400a2190 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): dump: 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.460 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1129644251 0 0) 0x7f9d400a2d90 con 0x7f9d400a2980 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d20003530 con 0x7f9d400a2980 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9d38001bc0 con 0x7f9d400a2980 2024-09-16T05:34:58.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): dump: 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3966116213 0 0) 0x7f9d401c5c10 con 0x7f9d400a2570 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient(hunting): my global_id is 4306 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d400a2d90 con 0x7f9d400a2570 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 186003912 0 0) 0x7f9d20003530 con 0x7f9d400a2980 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 >> v1:172.21.15.63:6789/0 conn(0x7f9d400a2570 legacy=0x7f9d4009c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 >> v1:172.21.15.6:6789/0 conn(0x7f9d400a2190 legacy=0x7f9d4009c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 monclient: found mon.c 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4009d0e0 con 0x7f9d400a2980 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.443+0000 7f9d2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d44b42640 5 monclient: authenticate success, global_id 4301 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d44b42640 10 monclient: _renew_subs 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d44b42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4009e2c0 con 0x7f9d400a2980 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d44b42640 10 monclient: _renew_subs 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d44b42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9d400a0100 con 0x7f9d400a2980 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d38003980 con 0x7f9d400a2980 2024-09-16T05:34:58.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d2effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9d38003cd0 con 0x7f9d400a2980 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d2effd640 10 monclient: dump: 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.465 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d3cd99640 4 set_mon_vals no callback set 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9d3800ea70 con 0x7f9d400a2980 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f9d380430e0 con 0x7f9d400a2980 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.447+0000 7f9d3e59c640 10 monclient: get_auth_request con 0x7f9d20036c50 auth_method 0 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.451+0000 7f9d44b42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.451+0000 7f9d44b42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.451+0000 7f9d44b42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.451+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d24005090 con 0x7f9d400a2980 2024-09-16T05:34:58.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.459+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9d380143d0 con 0x7f9d400a2980 2024-09-16T05:34:58.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.459+0000 7f9d2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.459+0000 7f9d2effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 10 monclient: monmap: 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.493239+0000 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.493239+0000 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.495 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 1 Processor -- start 2024-09-16T05:34:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 1 -- start start 2024-09-16T05:34:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 10 monclient: init 2024-09-16T05:34:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 10 monclient(hunting): picked mon.noname-b con 0x7f63a00a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f63a5093640 10 monclient(hunting): picked mon.noname-c con 0x7f63a00a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7fd125ffa640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7fd125ffa640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f639ed9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f63a00a2190 0x7f63a00a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33782/0 (socket says 172.21.15.6:33782) 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7fd125ffa640 10 monclient: monmap: 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.494642+0000 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.494642+0000 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.498 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.487+0000 7f639f59e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f63a00a2980 0x7f63a013f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58730/0 (socket says 172.21.15.6:58730) 2024-09-16T05:34:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient(hunting): picked mon.noname-a con 0x7f63a00a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639ed9d640 1 -- 172.21.15.6:0/2826246327 learned_addr learned my addr 172.21.15.6:0/2826246327 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a33a0 con 0x7f63a00a2980 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465531730 0 0) 0x7f63a00a33a0 con 0x7f63a00a2980 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- 172.21.15.6:0/2826246327 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a07a0 con 0x7f63a00a2570 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- 172.21.15.6:0/2826246327 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a2d90 con 0x7f63a00a2190 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient(hunting): authenticate will time out at 960.300488s 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): my global_id is 4308 2024-09-16T05:34:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f638c003530 con 0x7f63a00a2980 2024-09-16T05:34:58.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 1 Processor -- start 2024-09-16T05:34:58.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 1 -- start start 2024-09-16T05:34:58.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient: init 2024-09-16T05:34:58.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 697835817 0 0) 0x7f63a00a2d90 con 0x7f63a00a2190 2024-09-16T05:34:58.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): my global_id is 4304 2024-09-16T05:34:58.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63a00a33a0 con 0x7f63a00a2190 2024-09-16T05:34:58.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967290783 0 0) 0x7f63a00a07a0 con 0x7f63a00a2570 2024-09-16T05:34:58.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient(hunting): my global_id is 4309 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63a00a2d90 con 0x7f63a00a2570 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1802199722 0 0) 0x7f638c003530 con 0x7f63a00a2980 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 >> v1:172.21.15.6:6790/0 conn(0x7f63a00a2190 legacy=0x7f63a00a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 >> v1:172.21.15.63:6789/0 conn(0x7f63a00a2570 legacy=0x7f63a013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 monclient: found mon.noname-a 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a00a66e0 con 0x7f63a00a2980 2024-09-16T05:34:58.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f63980019a0 con 0x7f63a00a2980 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 5 monclient: authenticate success, global_id 4308 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: dump: 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.503 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6398001c00 con 0x7f63a00a2980 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 1 -- 172.21.15.6:0/2826246327 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6398001fd0 con 0x7f63a00a2980 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f639e59c640 10 monclient: dump: 2024-09-16T05:34:58.504 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 4 set_mon_vals no callback set 2024-09-16T05:34:58.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:58.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient(hunting): picked mon.noname-a con 0x7fd1200a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient(hunting): picked mon.noname-b con 0x7fd1200a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient: shutdown 2024-09-16T05:34:58.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- 172.21.15.6:0/2826246327 >> v1:172.21.15.6:6789/0 conn(0x7f63a00a2980 legacy=0x7f63a013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- 172.21.15.6:0/2826246327 shutdown_connections 2024-09-16T05:34:58.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- 172.21.15.6:0/2826246327 >> 172.21.15.6:0/2826246327 conn(0x7f63a00a9850 msgr2=0x7f63a00a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd1200a2570 0x7fd12013b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33798/0 (socket says 172.21.15.6:33798) 2024-09-16T05:34:58.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd1200a2190 0x7fd1200a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58744/0 (socket says 172.21.15.6:58744) 2024-09-16T05:34:58.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11effd640 1 -- 172.21.15.6:0/2285271893 learned_addr learned my addr 172.21.15.6:0/2285271893 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- 172.21.15.6:0/2826246327 shutdown_connections 2024-09-16T05:34:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- 172.21.15.6:0/2826246327 wait complete. 2024-09-16T05:34:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient(hunting): picked mon.noname-c con 0x7fd1200a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 1 -- 172.21.15.6:0/2285271893 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1200a33a0 con 0x7fd1200a2190 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 1 -- 172.21.15.6:0/2285271893 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1200a07a0 con 0x7fd1200a2980 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 1 -- 172.21.15.6:0/2285271893 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1200a2d90 con 0x7fd1200a2570 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient(hunting): authenticate will time out at 960.302157s 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247967075 0 0) 0x7fd1200a33a0 con 0x7fd1200a2190 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient(hunting): my global_id is 4311 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd10c003530 con 0x7fd1200a2190 2024-09-16T05:34:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750966231 0 0) 0x7fd1200a2d90 con 0x7fd1200a2570 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient(hunting): my global_id is 4307 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1200a33a0 con 0x7fd1200a2570 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2506103507 0 0) 0x7fd1200a33a0 con 0x7fd1200a2570 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 >> v1:172.21.15.63:6789/0 conn(0x7fd1200a2980 legacy=0x7fd12013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 >> v1:172.21.15.6:6789/0 conn(0x7fd1200a2190 legacy=0x7fd1200a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 monclient: found mon.noname-b 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1200a66e0 con 0x7fd1200a2570 2024-09-16T05:34:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd110001620 con 0x7fd1200a2570 2024-09-16T05:34:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient: monmap: 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 Processor -- start 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: dump: 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 5 monclient: authenticate success, global_id 4307 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd110001880 con 0x7fd1200a2570 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 1 -- 172.21.15.6:0/2285271893 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd110001c50 con 0x7fd1200a2570 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd11e7fc640 10 monclient: dump: 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7fd125ffa640 4 set_mon_vals no callback set 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- start start 2024-09-16T05:34:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient: init 2024-09-16T05:34:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f319926a640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f319926a640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f319926a640 10 monclient: monmap: 2024-09-16T05:34:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.498392+0000 2024-09-16T05:34:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.498392+0000 2024-09-16T05:34:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.517 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient(hunting): picked mon.noname-b con 0x7f63a00a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient(hunting): picked mon.noname-a con 0x7f63a00a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient(hunting): picked mon.noname-c con 0x7f63a00a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a011f7b0 con 0x7f63a00a2570 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a009ea20 con 0x7f63a00a2190 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a011fa90 con 0x7f63a00a2980 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f63a5093640 10 monclient(hunting): authenticate will time out at 960.303627s 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f6397fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f63a00a2570 0x7f63a009cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58756/0 (socket says 172.21.15.6:58756) 2024-09-16T05:34:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.491+0000 7f6397fff640 1 -- 172.21.15.6:0/2156403424 learned_addr learned my addr 172.21.15.6:0/2156403424 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient: shutdown 2024-09-16T05:34:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- 172.21.15.6:0/2285271893 >> v1:172.21.15.6:6790/0 conn(0x7fd1200a2570 legacy=0x7fd12013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- 172.21.15.6:0/2285271893 shutdown_connections 2024-09-16T05:34:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- 172.21.15.6:0/2285271893 >> 172.21.15.6:0/2285271893 conn(0x7fd1200a9850 msgr2=0x7fd1200a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923683309 0 0) 0x7f63a011f7b0 con 0x7f63a00a2570 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): my global_id is 4314 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6380003530 con 0x7f63a00a2570 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633968081 0 0) 0x7f63a011fa90 con 0x7f63a00a2980 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): my global_id is 4310 2024-09-16T05:34:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63a011f7b0 con 0x7f63a00a2980 2024-09-16T05:34:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- 172.21.15.6:0/2285271893 shutdown_connections 2024-09-16T05:34:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- 172.21.15.6:0/2285271893 wait complete. 2024-09-16T05:34:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 1 Processor -- start 2024-09-16T05:34:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 1 -- start start 2024-09-16T05:34:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient: init 2024-09-16T05:34:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2153035613 0 0) 0x7f63a011f7b0 con 0x7f63a00a2980 2024-09-16T05:34:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63a011fa90 con 0x7f63a00a2980 2024-09-16T05:34:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6398001f50 con 0x7f63a00a2980 2024-09-16T05:34:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): dump: 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1348915753 0 0) 0x7f6380003530 con 0x7f63a00a2570 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63a011f7b0 con 0x7f63a00a2570 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f63880018c0 con 0x7f63a00a2570 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient(hunting): dump: 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.525 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.525 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.525 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.525 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1640132575 0 0) 0x7f63a011f7b0 con 0x7f63a00a2570 2024-09-16T05:34:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 >> v1:172.21.15.6:6790/0 conn(0x7f63a00a2980 legacy=0x7f63a01cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 >> v1:172.21.15.63:6789/0 conn(0x7f63a00a2190 legacy=0x7f63a009c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 monclient: found mon.a 2024-09-16T05:34:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a0072540 con 0x7f63a00a2570 2024-09-16T05:34:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 5 monclient: authenticate success, global_id 4314 2024-09-16T05:34:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 10 monclient: _renew_subs 2024-09-16T05:34:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63a00a80a0 con 0x7f63a00a2570 2024-09-16T05:34:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient: monmap: 2024-09-16T05:34:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 10 monclient: _renew_subs 2024-09-16T05:34:58.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63a009e7a0 con 0x7f63a00a2570 2024-09-16T05:34:58.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:58.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient(hunting): picked mon.noname-b con 0x7f31940a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 Processor -- start 2024-09-16T05:34:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient(hunting): picked mon.noname-a con 0x7f31940a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient(hunting): picked mon.noname-c con 0x7f31940a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940a33a0 con 0x7f31940a2570 2024-09-16T05:34:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- start start 2024-09-16T05:34:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient: init 2024-09-16T05:34:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940a07a0 con 0x7f31940a2980 2024-09-16T05:34:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940a2d90 con 0x7f31940a2190 2024-09-16T05:34:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient(hunting): authenticate will time out at 960.305612s 2024-09-16T05:34:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f31927fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f31940a2570 0x7f319413b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58764/0 (socket says 172.21.15.6:58764) 2024-09-16T05:34:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f31927fc640 1 -- 172.21.15.6:0/3257766365 learned_addr learned my addr 172.21.15.6:0/3257766365 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1336080594 0 0) 0x7f31940a33a0 con 0x7f31940a2570 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient(hunting): my global_id is 4317 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f317c003530 con 0x7f31940a2570 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1095413419 0 0) 0x7f317c003530 con 0x7f31940a2570 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 >> v1:172.21.15.6:6790/0 conn(0x7f31940a2190 legacy=0x7f31940a1a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 >> v1:172.21.15.63:6789/0 conn(0x7f31940a2980 legacy=0x7f319413f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 monclient: found mon.noname-a 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31940a66e0 con 0x7f31940a2570 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3188001620 con 0x7f31940a2570 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: dump: 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.534 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.535 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.535 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.535 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.535 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.535 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 5 monclient: authenticate success, global_id 4317 2024-09-16T05:34:58.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63a00008e0 con 0x7f63a00a2570 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6388003620 con 0x7f63a00a2570 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f63880039f0 con 0x7f63a00a2570 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 10 monclient: dump: 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.537 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6388003b90 con 0x7f63a00a2570 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient(hunting): picked mon.noname-b con 0x7fd1200a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient(hunting): picked mon.noname-c con 0x7fd1200a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient(hunting): picked mon.noname-a con 0x7fd1201213d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd120123e70 con 0x7fd1201213d0 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1201225e0 con 0x7fd1200a2190 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd120122160 con 0x7fd1200a2980 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd125ffa640 10 monclient(hunting): authenticate will time out at 960.306632s 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3188001880 con 0x7f31940a2570 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 1 -- 172.21.15.6:0/3257766365 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3188001c90 con 0x7f31940a2570 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f3191ffb640 10 monclient: dump: 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 4 set_mon_vals no callback set 2024-09-16T05:34:58.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd11ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd1201213d0 0x7fd1201d3af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58768/0 (socket says 172.21.15.6:58768) 2024-09-16T05:34:58.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7fd11ffff640 1 -- 172.21.15.6:0/1555382139 learned_addr learned my addr 172.21.15.6:0/1555382139 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f639dd9b640 4 set_mon_vals no callback set 2024-09-16T05:34:58.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f639ed9d640 10 monclient: get_auth_request con 0x7f6380036ba0 auth_method 0 2024-09-16T05:34:58.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 10 monclient: shutdown 2024-09-16T05:34:58.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.495+0000 7f319926a640 1 -- 172.21.15.6:0/3257766365 >> v1:172.21.15.6:6789/0 conn(0x7f31940a2570 legacy=0x7f319413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 428632107 0 0) 0x7fd1201225e0 con 0x7fd1200a2190 2024-09-16T05:34:58.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): my global_id is 4321 2024-09-16T05:34:58.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd100003530 con 0x7fd1200a2190 2024-09-16T05:34:58.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3457187146 0 0) 0x7fd120122160 con 0x7fd1200a2980 2024-09-16T05:34:58.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): my global_id is 4313 2024-09-16T05:34:58.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1201225e0 con 0x7fd1200a2980 2024-09-16T05:34:58.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f63880114d0 con 0x7f63a00a2570 2024-09-16T05:34:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2855159878 0 0) 0x7fd1201225e0 con 0x7fd1200a2980 2024-09-16T05:34:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd120122160 con 0x7fd1200a2980 2024-09-16T05:34:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd110001a30 con 0x7fd1200a2980 2024-09-16T05:34:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): dump: 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 160255439 0 0) 0x7fd100003530 con 0x7fd1200a2190 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1201225e0 con 0x7fd1200a2190 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd108001810 con 0x7fd1200a2190 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient(hunting): dump: 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1631326794 0 0) 0x7fd120122160 con 0x7fd1200a2980 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 >> v1:172.21.15.63:6789/0 conn(0x7fd1200a2190 legacy=0x7fd1201205b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 >> v1:172.21.15.6:6789/0 conn(0x7fd1201213d0 legacy=0x7fd1201d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 monclient: found mon.c 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd120124430 con 0x7fd1200a2980 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 5 monclient: authenticate success, global_id 4313 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd110001d30 con 0x7fd1200a2980 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 10 monclient: _renew_subs 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd120123b90 con 0x7fd1200a2980 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd11dffb640 4 set_mon_vals no callback set 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 10 monclient: _renew_subs 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd120121810 con 0x7fd1200a2980 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd110004da0 con 0x7fd1200a2980 2024-09-16T05:34:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 10 monclient: dump: 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd11000fbc0 con 0x7fd1200a2980 2024-09-16T05:34:58.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7fd110044490 con 0x7fd1200a2980 2024-09-16T05:34:58.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd11f7fe640 10 monclient: get_auth_request con 0x7fd100036d50 auth_method 0 2024-09-16T05:34:58.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1200a82c0 con 0x7fd1200a2980 2024-09-16T05:34:58.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- 172.21.15.6:0/3257766365 shutdown_connections 2024-09-16T05:34:58.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- 172.21.15.6:0/3257766365 >> 172.21.15.6:0/3257766365 conn(0x7f31940a9850 msgr2=0x7f31940a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- 172.21.15.6:0/3257766365 shutdown_connections 2024-09-16T05:34:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6388015550 con 0x7f63a00a2570 2024-09-16T05:34:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f63977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f63977fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:58.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- 172.21.15.6:0/3257766365 wait complete. 2024-09-16T05:34:58.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: monmap: 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.556 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 Processor -- start 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- start start 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: init 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient(hunting): picked mon.noname-b con 0x7f31940a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient(hunting): picked mon.noname-a con 0x7f31940a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient(hunting): picked mon.noname-c con 0x7f31940a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31941c6260 con 0x7f31940a2570 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31941c5ba0 con 0x7f31940a2190 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940a0100 con 0x7f31940a2980 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient(hunting): authenticate will time out at 960.313734s 2024-09-16T05:34:58.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f31937fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f31940a2980 0x7f319409d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33816/0 (socket says 172.21.15.6:33816) 2024-09-16T05:34:58.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f31937fe640 1 -- 172.21.15.6:0/3902048275 learned_addr learned my addr 172.21.15.6:0/3902048275 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2004875721 0 0) 0x7f31940a66e0 con 0x7f31940a2570 2024-09-16T05:34:58.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): my global_id is 4320 2024-09-16T05:34:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3180003530 con 0x7f31940a2570 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3405035623 0 0) 0x7f31941c5ba0 con 0x7f31940a2190 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): my global_id is 4324 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31940a66e0 con 0x7f31940a2190 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2070506278 0 0) 0x7f3180003530 con 0x7f31940a2570 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31941c5ba0 con 0x7f31940a2570 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3188001bd0 con 0x7f31940a2570 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): dump: 2024-09-16T05:34:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3312796868 0 0) 0x7f31941c5ba0 con 0x7f31940a2570 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 >> v1:172.21.15.6:6790/0 conn(0x7f31940a2980 legacy=0x7f319409d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 >> v1:172.21.15.63:6789/0 conn(0x7f31940a2190 legacy=0x7f319409c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 monclient: found mon.a 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319409f830 con 0x7f31940a2570 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 5 monclient: authenticate success, global_id 4320 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: _renew_subs 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f319409ddb0 con 0x7f31940a2570 2024-09-16T05:34:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: _renew_subs 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f31941cd7f0 con 0x7f31940a2570 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3188004c00 con 0x7f31940a2570 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3188004f50 con 0x7f31940a2570 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f317b7fe640 10 monclient: dump: 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.565 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f319926a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.503+0000 7f31917fa640 4 set_mon_vals no callback set 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient: get_monmap_and_config 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient: monmap: 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:58.504498+0000 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:58.504498+0000 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 1 Processor -- start 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 1 -- start start 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient: init 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient(hunting): picked mon.noname-b con 0x7f8e6c09e830 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient(hunting): picked mon.noname-a con 0x7f8e6c09ec10 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient(hunting): picked mon.noname-c con 0x7f8e6c09b850 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6c0a1350 con 0x7f8e6c09ec10 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6c09cdf0 con 0x7f8e6c09b850 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6c08b7a0 con 0x7f8e6c09e830 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e71765640 10 monclient(hunting): authenticate will time out at 960.311645s 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e6affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8e6c09e830 0x7f8e6c09aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33806/0 (socket says 172.21.15.6:33806) 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.499+0000 7f8e6affd640 1 -- 172.21.15.6:0/950681839 learned_addr learned my addr 172.21.15.6:0/950681839 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.507+0000 7f319926a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.507+0000 7f319926a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.507+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31940008e0 con 0x7f31940a2570 2024-09-16T05:34:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.507+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd1200a82c0 con 0x7fd1200a2980 2024-09-16T05:34:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.507+0000 7fd117fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.507+0000 7fd117fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:58.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1621401587 0 0) 0x7f8e6c09cdf0 con 0x7f8e6c09b850 2024-09-16T05:34:58.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient(hunting): my global_id is 4327 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e54003510 con 0x7f8e6c09b850 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526213521 0 0) 0x7f8e6c08b7a0 con 0x7f8e6c09e830 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient(hunting): my global_id is 4316 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e6c09cdf0 con 0x7f8e6c09e830 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 316960065 0 0) 0x7f8e54003510 con 0x7f8e6c09b850 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 >> v1:172.21.15.6:6790/0 conn(0x7f8e6c09e830 legacy=0x7f8e6c09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 >> v1:172.21.15.6:6789/0 conn(0x7f8e6c09ec10 legacy=0x7f8e6c09b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 monclient: found mon.noname-c 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6c0a10d0 con 0x7f8e6c09b850 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8e640019a0 con 0x7f8e6c09b850 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: dump: 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 5 monclient: authenticate success, global_id 4327 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e64001c00 con 0x7f8e6c09b850 2024-09-16T05:34:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f318800fcf0 con 0x7f31940a2570 2024-09-16T05:34:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 1 -- 172.21.15.6:0/950681839 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8e64001f50 con 0x7f8e6c09b850 2024-09-16T05:34:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e69ffb640 10 monclient: dump: 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f3188043f10 con 0x7f31940a2570 2024-09-16T05:34:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 4 set_mon_vals no callback set 2024-09-16T05:34:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f31941c6260 con 0x7f31940a2570 2024-09-16T05:34:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f317b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f317b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:58.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f3192ffd640 10 monclient: get_auth_request con 0x7f3180036bd0 auth_method 0 2024-09-16T05:34:58.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 10 monclient: shutdown 2024-09-16T05:34:58.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 1 -- 172.21.15.6:0/950681839 >> v1:172.21.15.63:6789/0 conn(0x7f8e6c09b850 legacy=0x7f8e6c14c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 1 -- 172.21.15.6:0/950681839 shutdown_connections 2024-09-16T05:34:58.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.511+0000 7f8e71765640 1 -- 172.21.15.6:0/950681839 >> 172.21.15.6:0/950681839 conn(0x7f8e6c0a1650 msgr2=0x7f8e6c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 1 -- 172.21.15.6:0/950681839 shutdown_connections 2024-09-16T05:34:58.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 1 -- 172.21.15.6:0/950681839 wait complete. 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient: build_initial_monmap 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient: monmap: 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 1 Processor -- start 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 1 -- start start 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient: init 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient(hunting): picked mon.noname-c con 0x7f8e6c09e830 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient(hunting): picked mon.noname-b con 0x7f8e6c09ec10 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient(hunting): picked mon.noname-a con 0x7f8e6c08ad10 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6c0a34a0 con 0x7f8e6c08ad10 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6c082ab0 con 0x7f8e6c09ec10 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6c082d90 con 0x7f8e6c09e830 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e71765640 10 monclient(hunting): authenticate will time out at 960.326327s 2024-09-16T05:34:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e6affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8e6c09e830 0x7f8e6c08a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33826/0 (socket says 172.21.15.6:33826) 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e6affd640 1 -- 172.21.15.6:0/1772419641 learned_addr learned my addr 172.21.15.6:0/1772419641 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1295578704 0 0) 0x7f8e6c082d90 con 0x7f8e6c09e830 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): my global_id is 4319 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e58003530 con 0x7f8e6c09e830 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 508954518 0 0) 0x7f8e6c082ab0 con 0x7f8e6c09ec10 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:58.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): my global_id is 4330 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e6c082d90 con 0x7f8e6c09ec10 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1917376976 0 0) 0x7f8e6c082d90 con 0x7f8e6c09ec10 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e6c082ab0 con 0x7f8e6c09ec10 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8e600017f0 con 0x7f8e6c09ec10 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): dump: 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.515+0000 7f8e537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4274535133 0 0) 0x7f8e58003530 con 0x7f8e6c09e830 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e6c082d90 con 0x7f8e6c09e830 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8e5c001920 con 0x7f8e6c09e830 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient(hunting): dump: 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.586 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2652092169 0 0) 0x7f8e6c082ab0 con 0x7f8e6c09ec10 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 >> v1:172.21.15.6:6790/0 conn(0x7f8e6c09e830 legacy=0x7f8e6c08a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 >> v1:172.21.15.6:6789/0 conn(0x7f8e6c08ad10 legacy=0x7f8e6c1e0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:58.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 monclient: found mon.b 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6c081660 con 0x7f8e6c09ec10 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 5 monclient: authenticate success, global_id 4330 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 10 monclient: _renew_subs 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e6c081b00 con 0x7f8e6c09ec10 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 10 monclient: _renew_subs 2024-09-16T05:34:58.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e6c1d8530 con 0x7f8e6c09ec10 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e60003620 con 0x7f8e6c09ec10 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8e60003970 con 0x7f8e6c09ec10 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 10 monclient: dump: 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:58.589 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e697fa640 4 set_mon_vals no callback set 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e4c005090 con 0x7f8e6c09ec10 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8e6000e790 con 0x7f8e6c09ec10 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f8e60042f00 con 0x7f8e6c09ec10 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.519+0000 7f8e6affd640 10 monclient: get_auth_request con 0x7f8e580369b0 auth_method 0 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.531+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8e6c0a1350 con 0x7f8e6c09ec10 2024-09-16T05:34:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.535+0000 7f8e537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:58.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.535+0000 7f8e537fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:34:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:34:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa8241d4ac0 con 0x7fa8240a2980 2024-09-16T05:34:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa812ffd640 1 -- 172.21.15.6:0/2257293844 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 956180462) 0x7fa81c019210 con 0x7fa8240a2980 2024-09-16T05:34:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa812ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:34:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa812ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:58.608 INFO:teuthology.orchestra.run.smithi006.stdout:51539607555 2024-09-16T05:34:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa804036b80 legacy=0x7fa804039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 10 monclient: shutdown 2024-09-16T05:34:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 >> v1:172.21.15.6:6790/0 conn(0x7fa8240a2980 legacy=0x7fa8241217d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 shutdown_connections 2024-09-16T05:34:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 >> 172.21.15.6:0/2257293844 conn(0x7fa8240a9850 msgr2=0x7fa82406ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 shutdown_connections 2024-09-16T05:34:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.603+0000 7fa828c3f640 1 -- 172.21.15.6:0/2257293844 wait complete. 2024-09-16T05:34:58.625 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.5 2024-09-16T05:34:58.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.671+0000 7f63a5093640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:34:58.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.671+0000 7f63a5093640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:34:58.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.671+0000 7f63a5093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.671+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f63a006f180 con 0x7f63a00a2570 2024-09-16T05:34:58.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.671+0000 7f63977fe640 1 -- 172.21.15.6:0/2156403424 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 1444633209) 0x7f6388014e50 con 0x7f63a00a2570 2024-09-16T05:34:58.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.671+0000 7f63977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:34:58.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.671+0000 7f63977fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:58.679 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2024-09-16T05:34:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.675+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6380036ba0 legacy=0x7f6380039060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.675+0000 7f63a5093640 10 monclient: shutdown 2024-09-16T05:34:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.675+0000 7f63a5093640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.675+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 >> v1:172.21.15.6:6789/0 conn(0x7f63a00a2570 legacy=0x7f63a009cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.675+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 shutdown_connections 2024-09-16T05:34:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.675+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 >> 172.21.15.6:0/2156403424 conn(0x7f63a00a9850 msgr2=0x7f63a0136da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.675+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 shutdown_connections 2024-09-16T05:34:58.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.675+0000 7f63a5093640 1 -- 172.21.15.6:0/2156403424 wait complete. 2024-09-16T05:34:58.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.2 2024-09-16T05:34:58.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.695+0000 7f9d44b42640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:34:58.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.695+0000 7f9d44b42640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:34:58.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.695+0000 7f9d44b42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.695+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9d24005380 con 0x7f9d400a2980 2024-09-16T05:34:58.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.695+0000 7f9d2effd640 1 -- 172.21.15.6:0/119282032 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 1444633209) 0x7f9d38013cd0 con 0x7f9d400a2980 2024-09-16T05:34:58.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.695+0000 7f9d2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:34:58.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.695+0000 7f9d2effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:58.703 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2024-09-16T05:34:58.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.699+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9d20036c50 legacy=0x7f9d20039110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.699+0000 7f9d44b42640 10 monclient: shutdown 2024-09-16T05:34:58.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.699+0000 7f9d44b42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.699+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 >> v1:172.21.15.6:6790/0 conn(0x7f9d400a2980 legacy=0x7f9d4013fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.699+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 shutdown_connections 2024-09-16T05:34:58.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.699+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 >> 172.21.15.6:0/119282032 conn(0x7f9d400a9850 msgr2=0x7f9d4012f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.699+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 shutdown_connections 2024-09-16T05:34:58.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.699+0000 7f9d44b42640 1 -- 172.21.15.6:0/119282032 wait complete. 2024-09-16T05:34:58.719 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.3 2024-09-16T05:34:58.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:34:58.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:34:58.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:34:58.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8e4c005380 con 0x7f8e6c09ec10 2024-09-16T05:34:58.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e537fe640 1 -- 172.21.15.6:0/1772419641 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1444633209) 0x7f8e60014130 con 0x7f8e6c09ec10 2024-09-16T05:34:58.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:34:58.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e537fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:58.756 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2024-09-16T05:34:58.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8e580369b0 legacy=0x7f8e58038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 10 monclient: shutdown 2024-09-16T05:34:58.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 >> v1:172.21.15.63:6789/0 conn(0x7f8e6c09ec10 legacy=0x7f8e6c1dd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 shutdown_connections 2024-09-16T05:34:58.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 >> 172.21.15.6:0/1772419641 conn(0x7f8e6c0a1650 msgr2=0x7f8e6c09f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 shutdown_connections 2024-09-16T05:34:58.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.751+0000 7f8e71765640 1 -- 172.21.15.6:0/1772419641 wait complete. 2024-09-16T05:34:58.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.0 2024-09-16T05:34:58.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:34:58.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:34:58.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:58.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f319413fb90 con 0x7f31940a2570 2024-09-16T05:34:58.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f317b7fe640 1 -- 172.21.15.6:0/3902048275 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 956180462) 0x7f31880152c0 con 0x7f31940a2570 2024-09-16T05:34:58.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f317b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:34:58.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f317b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:58.788 INFO:teuthology.orchestra.run.smithi006.stdout:51539607555 2024-09-16T05:34:58.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7fd125ffa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:34:58.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7fd125ffa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:34:58.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7fd125ffa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:58.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd120075290 con 0x7fd1200a2980 2024-09-16T05:34:58.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7fd117fff640 1 -- 172.21.15.6:0/1555382139 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 847495621) 0x7fd110015640 con 0x7fd1200a2980 2024-09-16T05:34:58.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7fd117fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:34:58.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7fd117fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:34:58.790 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2024-09-16T05:34:58.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3180036bd0 legacy=0x7f3180039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 10 monclient: shutdown 2024-09-16T05:34:58.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 >> v1:172.21.15.6:6789/0 conn(0x7f31940a2570 legacy=0x7f319409cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 shutdown_connections 2024-09-16T05:34:58.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 >> 172.21.15.6:0/3902048275 conn(0x7f31940a9850 msgr2=0x7f319412f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 shutdown_connections 2024-09-16T05:34:58.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.783+0000 7f319926a640 1 -- 172.21.15.6:0/3902048275 wait complete. 2024-09-16T05:34:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.787+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd100036d50 legacy=0x7fd100039210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.787+0000 7fd125ffa640 10 monclient: shutdown 2024-09-16T05:34:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.787+0000 7fd125ffa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.787+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 >> v1:172.21.15.6:6790/0 conn(0x7fd1200a2980 legacy=0x7fd120120cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.787+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 shutdown_connections 2024-09-16T05:34:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.787+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 >> 172.21.15.6:0/1555382139 conn(0x7fd1200a9850 msgr2=0x7fd12012f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.787+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 shutdown_connections 2024-09-16T05:34:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:58.787+0000 7fd125ffa640 1 -- 172.21.15.6:0/1555382139 wait complete. 2024-09-16T05:34:58.806 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.4 2024-09-16T05:34:58.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.1 2024-09-16T05:34:59.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T05:34:59.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T05:34:59.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.755+0000 7fc99966c640 10 monclient: get_monmap_and_config 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.755+0000 7fc99966c640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.755+0000 7fc99966c640 10 monclient: monmap: 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:59.761964+0000 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:59.761964+0000 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.762 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.763 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.763 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:59.763 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:59.763 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.755+0000 7fc99966c640 1 Processor -- start 2024-09-16T05:34:59.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.755+0000 7fc99966c640 1 -- start start 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.755+0000 7fc99966c640 10 monclient: init 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient(hunting): picked mon.noname-b con 0x7fc9940a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient(hunting): picked mon.noname-a con 0x7fc9940a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient(hunting): picked mon.noname-c con 0x7fc9940a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940a33a0 con 0x7fc9940a2570 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940a07a0 con 0x7fc9940a2980 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940a2d90 con 0x7fc9940a2190 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient(hunting): authenticate will time out at 961.568577s 2024-09-16T05:34:59.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc9927fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc9940a2570 0x7fc99413b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34182/0 (socket says 172.21.15.6:34182) 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc992ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc9940a2190 0x7fc9940a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42092/0 (socket says 172.21.15.6:42092) 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc9927fc640 1 -- 172.21.15.6:0/3145148899 learned_addr learned my addr 172.21.15.6:0/3145148899 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1827909213 0 0) 0x7fc9940a2d90 con 0x7fc9940a2190 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient(hunting): my global_id is 4322 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc97c003530 con 0x7fc9940a2190 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940639428 0 0) 0x7fc9940a33a0 con 0x7fc9940a2570 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient(hunting): my global_id is 4323 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9940a2d90 con 0x7fc9940a2570 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1257554960 0 0) 0x7fc97c003530 con 0x7fc9940a2190 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 >> v1:172.21.15.63:6789/0 conn(0x7fc9940a2980 legacy=0x7fc99413f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 >> v1:172.21.15.6:6789/0 conn(0x7fc9940a2570 legacy=0x7fc99413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 monclient: found mon.noname-b 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9940a66e0 con 0x7fc9940a2190 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc984001620 con 0x7fc9940a2190 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: dump: 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.769 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 5 monclient: authenticate success, global_id 4322 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9840018c0 con 0x7fc9940a2190 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 4 set_mon_vals no callback set 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 1 -- 172.21.15.6:0/3145148899 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc984001cd0 con 0x7fc9940a2190 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc991ffb640 10 monclient: dump: 2024-09-16T05:34:59.770 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient: shutdown 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- 172.21.15.6:0/3145148899 >> v1:172.21.15.6:6790/0 conn(0x7fc9940a2190 legacy=0x7fc9940a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- 172.21.15.6:0/3145148899 shutdown_connections 2024-09-16T05:34:59.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- 172.21.15.6:0/3145148899 >> 172.21.15.6:0/3145148899 conn(0x7fc9940a9850 msgr2=0x7fc9940a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:59.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- 172.21.15.6:0/3145148899 shutdown_connections 2024-09-16T05:34:59.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- 172.21.15.6:0/3145148899 wait complete. 2024-09-16T05:34:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient: monmap: 2024-09-16T05:34:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 Processor -- start 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 1 -- start start 2024-09-16T05:34:59.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.759+0000 7fc99966c640 10 monclient: init 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient(hunting): picked mon.noname-b con 0x7fc9940a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient(hunting): picked mon.noname-c con 0x7fc9940a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient(hunting): picked mon.noname-a con 0x7fc9940a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc994124740 con 0x7fc9940a2980 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9941237b0 con 0x7fc9940a2190 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940a0100 con 0x7fc9940a2570 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient(hunting): authenticate will time out at 961.572392s 2024-09-16T05:34:59.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc9927fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc9940a2570 0x7fc9941210c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42096/0 (socket says 172.21.15.6:42096) 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc9927fc640 1 -- 172.21.15.6:0/83938017 learned_addr learned my addr 172.21.15.6:0/83938017 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783720225 0 0) 0x7fc9940a0100 con 0x7fc9940a2570 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): my global_id is 4325 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc980003530 con 0x7fc9940a2570 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 581212327 0 0) 0x7fc994124740 con 0x7fc9940a2980 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): my global_id is 4326 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9940a0100 con 0x7fc9940a2980 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755604127 0 0) 0x7fc9941237b0 con 0x7fc9940a2190 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): my global_id is 4336 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc994124740 con 0x7fc9940a2190 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1995773830 0 0) 0x7fc9940a0100 con 0x7fc9940a2980 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9941237b0 con 0x7fc9940a2980 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc98c001c40 con 0x7fc9940a2980 2024-09-16T05:34:59.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): dump: 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1116927720 0 0) 0x7fc980003530 con 0x7fc9940a2570 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9940a0100 con 0x7fc9940a2570 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc988001840 con 0x7fc9940a2570 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient(hunting): dump: 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.781 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 63380381 0 0) 0x7fc9941237b0 con 0x7fc9940a2980 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 >> v1:172.21.15.6:6790/0 conn(0x7fc9940a2570 legacy=0x7fc9941210c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 >> v1:172.21.15.63:6789/0 conn(0x7fc9940a2190 legacy=0x7fc9941209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 monclient: found mon.a 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9940a7f10 con 0x7fc9940a2980 2024-09-16T05:34:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 5 monclient: authenticate success, global_id 4326 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient: _renew_subs 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc994122450 con 0x7fc9940a2980 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient: _renew_subs 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9941d4ac0 con 0x7fc9940a2980 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc98c0039a0 con 0x7fc9940a2980 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc98c003cf0 con 0x7fc9940a2980 2024-09-16T05:34:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc9917fa640 4 set_mon_vals no callback set 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 10 monclient: dump: 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.784 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc974005090 con 0x7fc9940a2980 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc98c00ea90 con 0x7fc9940a2980 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7fc98c043340 con 0x7fc9940a2980 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.763+0000 7fc992ffd640 10 monclient: get_auth_request con 0x7fc980036af0 auth_method 0 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.767+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc98c014630 con 0x7fc9940a2980 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.767+0000 7fc97b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.767+0000 7fc97b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:59.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T05:34:59.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T05:34:59.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T05:34:59.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.807+0000 7f4242e15640 10 monclient: get_monmap_and_config 2024-09-16T05:34:59.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.807+0000 7f4242e15640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.807+0000 7f4242e15640 10 monclient: monmap: 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:59.814425+0000 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:59.814425+0000 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:59.815 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 Processor -- start 2024-09-16T05:34:59.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- start start 2024-09-16T05:34:59.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient: init 2024-09-16T05:34:59.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:59.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient(hunting): picked mon.noname-a con 0x7f423c0a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient(hunting): picked mon.noname-b con 0x7f423c0a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient(hunting): picked mon.noname-c con 0x7f423c0a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f423c0a33a0 con 0x7f423c0a2190 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f423c0a07a0 con 0x7f423c0a2980 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f423c0a2d90 con 0x7f423c0a2570 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient(hunting): authenticate will time out at 961.620703s 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4240bb1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f423c0a2190 0x7f423c0a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34208/0 (socket says 172.21.15.6:34208) 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4240bb1640 1 -- 172.21.15.6:0/52279704 learned_addr learned my addr 172.21.15.6:0/52279704 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130445481 0 0) 0x7f423c0a33a0 con 0x7f423c0a2190 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient(hunting): my global_id is 4329 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f422c003530 con 0x7f423c0a2190 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346896558 0 0) 0x7f423c0a2d90 con 0x7f423c0a2570 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient(hunting): my global_id is 4328 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f423c0a33a0 con 0x7f423c0a2570 2024-09-16T05:34:59.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4250043199 0 0) 0x7f422c003530 con 0x7f423c0a2190 2024-09-16T05:34:59.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 >> v1:172.21.15.6:6790/0 conn(0x7f423c0a2570 legacy=0x7f423c13b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 >> v1:172.21.15.63:6789/0 conn(0x7f423c0a2980 legacy=0x7f423c13f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 monclient: found mon.noname-a 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423c0a66e0 con 0x7f423c0a2190 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4230001620 con 0x7f423c0a2190 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: dump: 2024-09-16T05:34:59.825 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 5 monclient: authenticate success, global_id 4329 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4230001880 con 0x7f423c0a2190 2024-09-16T05:34:59.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 1 -- 172.21.15.6:0/52279704 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4230001c50 con 0x7f423c0a2190 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 4 set_mon_vals no callback set 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f423b7fe640 10 monclient: dump: 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.827 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient: shutdown 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- 172.21.15.6:0/52279704 >> v1:172.21.15.6:6789/0 conn(0x7f423c0a2190 legacy=0x7f423c0a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- 172.21.15.6:0/52279704 shutdown_connections 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- 172.21.15.6:0/52279704 >> 172.21.15.6:0/52279704 conn(0x7f423c0a9850 msgr2=0x7f423c0a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- 172.21.15.6:0/52279704 shutdown_connections 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 1 -- 172.21.15.6:0/52279704 wait complete. 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.811+0000 7f4242e15640 10 monclient: monmap: 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 1 Processor -- start 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 1 -- start start 2024-09-16T05:34:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 10 monclient: init 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 10 monclient(hunting): picked mon.noname-b con 0x7f423c0a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f423bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f423c0a2570 0x7f423c121150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34218/0 (socket says 172.21.15.6:34218) 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f423bfff640 1 -- 172.21.15.6:0/3880834435 learned_addr learned my addr 172.21.15.6:0/3880834435 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 10 monclient(hunting): picked mon.noname-a con 0x7f423c0a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 10 monclient(hunting): picked mon.noname-c con 0x7f423c0a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f423c0a0100 con 0x7f423c0a2570 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f423c1245a0 con 0x7f423c0a2190 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f423c12d6d0 con 0x7f423c0a2980 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f4242e15640 10 monclient(hunting): authenticate will time out at 961.626648s 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724953799 0 0) 0x7f423c0a33a0 con 0x7f423c0a2570 2024-09-16T05:34:59.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): my global_id is 4332 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4220003530 con 0x7f423c0a2570 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324971444 0 0) 0x7f423c12d6d0 con 0x7f423c0a2980 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): my global_id is 4331 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f423c0a33a0 con 0x7f423c0a2980 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3497788851 0 0) 0x7f423c1245a0 con 0x7f423c0a2190 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): my global_id is 4342 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f423c12d6d0 con 0x7f423c0a2190 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1243849842 0 0) 0x7f4220003530 con 0x7f423c0a2570 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f423c1245a0 con 0x7f423c0a2570 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3950924321 0 0) 0x7f423c0a33a0 con 0x7f423c0a2980 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4220003530 con 0x7f423c0a2980 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4228001920 con 0x7f423c0a2570 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): dump: 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.832 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4234001b00 con 0x7f423c0a2980 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient(hunting): dump: 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 198510829 0 0) 0x7f4220003530 con 0x7f423c0a2980 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 >> v1:172.21.15.63:6789/0 conn(0x7f423c0a2190 legacy=0x7f423c120a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 >> v1:172.21.15.6:6789/0 conn(0x7f423c0a2570 legacy=0x7f423c121150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 monclient: found mon.c 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.815+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423c0a82c0 con 0x7f423c0a2980 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4234001ca0 con 0x7f423c0a2980 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f423affd640 4 set_mon_vals no callback set 2024-09-16T05:34:59.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4234003b40 con 0x7f423c0a2980 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 5 monclient: authenticate success, global_id 4331 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 10 monclient: _renew_subs 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f423c1226f0 con 0x7f423c0a2980 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 10 monclient: dump: 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.837 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4234003ce0 con 0x7f423c0a2980 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 10 monclient: _renew_subs 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f423c1404a0 con 0x7f423c0a2980 2024-09-16T05:34:59.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f4234043160 con 0x7f423c0a2980 2024-09-16T05:34:59.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4240bb1640 10 monclient: get_auth_request con 0x7f4220036c80 auth_method 0 2024-09-16T05:34:59.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.819+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f423c0008e0 con 0x7f423c0a2980 2024-09-16T05:34:59.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.827+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4234014360 con 0x7f423c0a2980 2024-09-16T05:34:59.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.827+0000 7f42397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.827+0000 7f42397fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.851+0000 7f9daf6bb640 10 monclient: get_monmap_and_config 2024-09-16T05:34:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.851+0000 7f9daf6bb640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.851+0000 7f9daf6bb640 10 monclient: monmap: 2024-09-16T05:34:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:59.858008+0000 2024-09-16T05:34:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:59.858008+0000 2024-09-16T05:34:59.858 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:59.859 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.851+0000 7f9daf6bb640 1 Processor -- start 2024-09-16T05:34:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.851+0000 7f9daf6bb640 1 -- start start 2024-09-16T05:34:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.851+0000 7f9daf6bb640 10 monclient: init 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient(hunting): picked mon.noname-b con 0x7f9da80a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient(hunting): picked mon.noname-c con 0x7f9da80a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9dad457640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9da80a37a0 0x7f9da80a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42126/0 (socket says 172.21.15.6:42126) 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient(hunting): picked mon.noname-a con 0x7f9da80a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9dad457640 1 -- 172.21.15.6:0/3744307139 learned_addr learned my addr 172.21.15.6:0/3744307139 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3744307139 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da806e520 con 0x7f9da80a1a00 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3744307139 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da80a0780 con 0x7f9da80a4290 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3744307139 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da80a4c70 con 0x7f9da80a37a0 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient(hunting): authenticate will time out at 961.665057s 2024-09-16T05:34:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2952468937 0 0) 0x7f9da80a4c70 con 0x7f9da80a37a0 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient(hunting): my global_id is 4334 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d94003530 con 0x7f9da80a37a0 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341282430 0 0) 0x7f9da80a0780 con 0x7f9da80a4290 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient(hunting): my global_id is 4345 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9da80a4c70 con 0x7f9da80a4290 2024-09-16T05:34:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 283972600 0 0) 0x7f9d94003530 con 0x7f9da80a37a0 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 >> v1:172.21.15.63:6789/0 conn(0x7f9da80a4290 legacy=0x7f9da813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 >> v1:172.21.15.6:6789/0 conn(0x7f9da80a1a00 legacy=0x7f9da813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 monclient: found mon.noname-b 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da80a65f0 con 0x7f9da80a37a0 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9da4001620 con 0x7f9da80a37a0 2024-09-16T05:34:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: dump: 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.865 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 5 monclient: authenticate success, global_id 4334 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9da4001880 con 0x7f9da80a37a0 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 1 -- 172.21.15.6:0/3744307139 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9da4001c90 con 0x7f9da80a37a0 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9d9ffff640 10 monclient: dump: 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.867 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 4 set_mon_vals no callback set 2024-09-16T05:34:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 10 monclient: shutdown 2024-09-16T05:34:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3744307139 >> v1:172.21.15.6:6790/0 conn(0x7f9da80a37a0 legacy=0x7f9da80a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3744307139 shutdown_connections 2024-09-16T05:34:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3744307139 >> 172.21.15.6:0/3744307139 conn(0x7f9da80a58a0 msgr2=0x7f9da80a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3744307139 shutdown_connections 2024-09-16T05:34:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.855+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3744307139 wait complete. 2024-09-16T05:34:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: monmap: 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 1 Processor -- start 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 1 -- start start 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: init 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient(hunting): picked mon.noname-a con 0x7f9da80a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient(hunting): picked mon.noname-b con 0x7f9da80a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient(hunting): picked mon.noname-c con 0x7f9da80a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da8135860 con 0x7f9da80a1a00 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da8132d50 con 0x7f9da80a37a0 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da8135b40 con 0x7f9da80a4290 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9dadc58640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9da80a4290 0x7f9da81d3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42142/0 (socket says 172.21.15.6:42142) 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9dadc58640 1 -- 172.21.15.6:0/3733019013 learned_addr learned my addr 172.21.15.6:0/3733019013 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient(hunting): authenticate will time out at 961.669925s 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035113205 0 0) 0x7f9da8135b40 con 0x7f9da80a4290 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): my global_id is 4337 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d8c003560 con 0x7f9da80a4290 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 562764677 0 0) 0x7f9da8132d50 con 0x7f9da80a37a0 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): my global_id is 4348 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9da8135b40 con 0x7f9da80a37a0 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2323136627 0 0) 0x7f9da8135860 con 0x7f9da80a1a00 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): my global_id is 4338 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9da8132d50 con 0x7f9da80a1a00 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1583621394 0 0) 0x7f9d8c003560 con 0x7f9da80a4290 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9da8135860 con 0x7f9da80a4290 2024-09-16T05:34:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9da0001cf0 con 0x7f9da80a4290 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): dump: 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.879 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1033911487 0 0) 0x7f9da8135b40 con 0x7f9da80a37a0 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d8c003560 con 0x7f9da80a37a0 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9d98001770 con 0x7f9da80a37a0 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient(hunting): dump: 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.880 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3414306335 0 0) 0x7f9da8135860 con 0x7f9da80a4290 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 >> v1:172.21.15.63:6789/0 conn(0x7f9da80a37a0 legacy=0x7f9da8132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 >> v1:172.21.15.6:6789/0 conn(0x7f9da80a1a00 legacy=0x7f9da8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 monclient: found mon.c 2024-09-16T05:34:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da80a8020 con 0x7f9da80a4290 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9d9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 5 monclient: authenticate success, global_id 4337 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: _renew_subs 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9da80a0100 con 0x7f9da80a4290 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: _renew_subs 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.859+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9da8133ec0 con 0x7f9da80a4290 2024-09-16T05:34:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.863+0000 7f9daf6bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.863+0000 7f9daf6bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.863+0000 7f9daf6bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.863+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da81366f0 con 0x7f9da80a4290 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9da0002010 con 0x7f9da80a4290 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9d9dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9d9f7fe640 4 set_mon_vals no callback set 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9da0003df0 con 0x7f9da80a4290 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9d9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9d9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9d9dffb640 10 monclient: dump: 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.883 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9da0004010 con 0x7f9da80a4290 2024-09-16T05:34:59.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.867+0000 7f9dad457640 10 monclient: get_auth_request con 0x7f9d8c036b10 auth_method 0 2024-09-16T05:34:59.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.871+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f9da0003df0 con 0x7f9da80a4290 2024-09-16T05:34:59.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.871+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9da00158f0 con 0x7f9da80a4290 2024-09-16T05:34:59.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.871+0000 7f9d9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.871+0000 7f9d9dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient: get_monmap_and_config 2024-09-16T05:34:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient: monmap: 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:59.940204+0000 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:59.940204+0000 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:59.941 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 1 Processor -- start 2024-09-16T05:34:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 1 -- start start 2024-09-16T05:34:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient: init 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient(hunting): picked mon.noname-b con 0x7f03200a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient(hunting): picked mon.noname-c con 0x7f03200a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient(hunting): picked mon.noname-a con 0x7f03200a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03200a33a0 con 0x7f03200a2980 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03200a07a0 con 0x7f03200a2570 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03200a2d90 con 0x7f03200a2190 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f03254ed640 10 monclient(hunting): authenticate will time out at 961.747041s 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f031f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03200a2980 0x7f032013f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34244/0 (socket says 172.21.15.6:34244) 2024-09-16T05:34:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.935+0000 7f031f7fe640 1 -- 172.21.15.6:0/2800240381 learned_addr learned my addr 172.21.15.6:0/2800240381 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1531015658 0 0) 0x7f03200a33a0 con 0x7f03200a2980 2024-09-16T05:34:59.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): my global_id is 4341 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0308003530 con 0x7f03200a2980 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181118821 0 0) 0x7f03200a07a0 con 0x7f03200a2570 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): my global_id is 4351 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03200a33a0 con 0x7f03200a2570 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2184292452 0 0) 0x7f03200a2d90 con 0x7f03200a2190 2024-09-16T05:34:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 10 monclient(hunting): my global_id is 4340 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03200a07a0 con 0x7f03200a2190 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1024632135 0 0) 0x7f0308003530 con 0x7f03200a2980 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 >> v1:172.21.15.6:6790/0 conn(0x7f03200a2190 legacy=0x7f03200a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.939+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 >> v1:172.21.15.63:6789/0 conn(0x7f03200a2570 legacy=0x7f032013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 1 monclient: found mon.noname-a 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03200a66e0 con 0x7f03200a2980 2024-09-16T05:34:59.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0318001920 con 0x7f03200a2980 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 5 monclient: authenticate success, global_id 4341 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: dump: 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.951 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0318001bf0 con 0x7f03200a2980 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 4 set_mon_vals no callback set 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 1 -- 172.21.15.6:0/2800240381 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0318001f40 con 0x7f03200a2980 2024-09-16T05:34:59.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f031dffb640 10 monclient: dump: 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.953 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 10 monclient: shutdown 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 1 -- 172.21.15.6:0/2800240381 >> v1:172.21.15.6:6789/0 conn(0x7f03200a2980 legacy=0x7f032013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 1 -- 172.21.15.6:0/2800240381 shutdown_connections 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 1 -- 172.21.15.6:0/2800240381 >> 172.21.15.6:0/2800240381 conn(0x7f03200a9850 msgr2=0x7f03200a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 1 -- 172.21.15.6:0/2800240381 shutdown_connections 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 1 -- 172.21.15.6:0/2800240381 wait complete. 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 10 monclient: monmap: 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 1 Processor -- start 2024-09-16T05:34:59.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 1 -- start start 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.943+0000 7f03254ed640 10 monclient: init 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient(hunting): picked mon.noname-b con 0x7f03200a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient(hunting): picked mon.noname-c con 0x7f03200a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient(hunting): picked mon.noname-a con 0x7f03200a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0320128a60 con 0x7f03200a2980 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03200a0100 con 0x7f03200a2190 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f031e7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f03200a2570 0x7f03201d0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42162/0 (socket says 172.21.15.6:42162) 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f031e7fc640 1 -- 172.21.15.6:0/3323655771 learned_addr learned my addr 172.21.15.6:0/3323655771 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03200a8060 con 0x7f03200a2570 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient(hunting): authenticate will time out at 961.757163s 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2272924338 0 0) 0x7f0320128a60 con 0x7f03200a2980 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): my global_id is 4344 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f030c003560 con 0x7f03200a2980 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3820717697 0 0) 0x7f03200a0100 con 0x7f03200a2190 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): my global_id is 4354 2024-09-16T05:34:59.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0320128a60 con 0x7f03200a2190 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2261558304 0 0) 0x7f030c003560 con 0x7f03200a2980 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03200a0100 con 0x7f03200a2980 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f03180018a0 con 0x7f03200a2980 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): dump: 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.962 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907332570 0 0) 0x7f03200a8060 con 0x7f03200a2570 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient(hunting): my global_id is 4343 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f030c003560 con 0x7f03200a2570 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3602077575 0 0) 0x7f03200a0100 con 0x7f03200a2980 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 >> v1:172.21.15.6:6790/0 conn(0x7f03200a2570 legacy=0x7f03201d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 >> v1:172.21.15.63:6789/0 conn(0x7f03200a2190 legacy=0x7f03201cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 monclient: found mon.a 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0320128780 con 0x7f03200a2980 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 5 monclient: authenticate success, global_id 4344 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient: _renew_subs 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0320071f90 con 0x7f03200a2980 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient: _renew_subs 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.947+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0320070fc0 con 0x7f03200a2980 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0318004e20 con 0x7f03200a2980 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0318005170 con 0x7f03200a2980 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f031d7fa640 4 set_mon_vals no callback set 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03077fe640 10 monclient: dump: 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.965 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f031800ff10 con 0x7f03200a2980 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f0318044280 con 0x7f03200a2980 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f031effd640 10 monclient: get_auth_request con 0x7f030c036aa0 auth_method 0 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03254ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03254ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03254ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.951+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0300005090 con 0x7f03200a2980 2024-09-16T05:34:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.959+0000 7fa4f08ba640 10 monclient: get_monmap_and_config 2024-09-16T05:34:59.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.959+0000 7fa4f08ba640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.959+0000 7fa4f08ba640 10 monclient: monmap: 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:59.966380+0000 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:59.966380+0000 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 1 Processor -- start 2024-09-16T05:34:59.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 1 -- start start 2024-09-16T05:34:59.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 10 monclient: init 2024-09-16T05:34:59.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f03180156d0 con 0x7f03200a2980 2024-09-16T05:34:59.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7f03077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7f03077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:59.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:34:59.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa4ec0a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa4ec0a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa4ec0a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ec0a33a0 con 0x7fa4ec0a2190 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ec0a07a0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ec0a2d90 con 0x7fa4ec0a2570 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4f08ba640 10 monclient(hunting): authenticate will time out at 961.773045s 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4ea59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa4ec0a2190 0x7fa4ec0a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34272/0 (socket says 172.21.15.6:34272) 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.963+0000 7fa4ea59c640 1 -- 172.21.15.6:0/634590136 learned_addr learned my addr 172.21.15.6:0/634590136 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270096388 0 0) 0x7fa4ec0a33a0 con 0x7fa4ec0a2190 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient(hunting): my global_id is 4347 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4d4003530 con 0x7fa4ec0a2190 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591858283 0 0) 0x7fa4ec0a2d90 con 0x7fa4ec0a2570 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient(hunting): my global_id is 4346 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4ec0a33a0 con 0x7fa4ec0a2570 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3513277176 0 0) 0x7fa4d4003530 con 0x7fa4ec0a2190 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 >> v1:172.21.15.6:6790/0 conn(0x7fa4ec0a2570 legacy=0x7fa4ec13b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 >> v1:172.21.15.63:6789/0 conn(0x7fa4ec0a2980 legacy=0x7fa4ec13f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 monclient: found mon.noname-a 2024-09-16T05:34:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ec0a66e0 con 0x7fa4ec0a2190 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 5 monclient: authenticate success, global_id 4347 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa4dc001620 con 0x7fa4ec0a2190 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient: dump: 2024-09-16T05:34:59.977 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:34:59.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 1 -- 172.21.15.6:0/634590136 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4dc0018f0 con 0x7fa4ec0a2190 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4e959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 10 monclient: get_monmap_and_config success 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 4 set_mon_vals no callback set 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 10 monclient: shutdown 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 1 -- 172.21.15.6:0/634590136 >> v1:172.21.15.6:6789/0 conn(0x7fa4ec0a2190 legacy=0x7fa4ec0a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 1 -- 172.21.15.6:0/634590136 shutdown_connections 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 1 -- 172.21.15.6:0/634590136 >> 172.21.15.6:0/634590136 conn(0x7fa4ec0a9850 msgr2=0x7fa4ec0a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 1 -- 172.21.15.6:0/634590136 shutdown_connections 2024-09-16T05:34:59.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 1 -- 172.21.15.6:0/634590136 wait complete. 2024-09-16T05:34:59.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.967+0000 7fa4f08ba640 10 monclient: monmap: 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 1 Processor -- start 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 1 -- start start 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: init 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa4ec0a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa4ec0a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa4ec0a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ec0a8100 con 0x7fa4ec0a2980 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ec0a0100 con 0x7fa4ec0a2570 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ec0a7bb0 con 0x7fa4ec0a2190 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient(hunting): authenticate will time out at 961.781052s 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4ea59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa4ec0a2190 0x7fa4ec13f9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42176/0 (socket says 172.21.15.6:42176) 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4ea59c640 1 -- 172.21.15.6:0/3951348711 learned_addr learned my addr 172.21.15.6:0/3951348711 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830266017 0 0) 0x7fa4ec0a7bb0 con 0x7fa4ec0a2190 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): my global_id is 4349 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4cc003530 con 0x7fa4ec0a2190 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2475911822 0 0) 0x7fa4e40015c0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): my global_id is 4350 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4ec0a7bb0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1174628850 0 0) 0x7fa4cc003530 con 0x7fa4ec0a2190 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4e40015c0 con 0x7fa4ec0a2190 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa4dc004950 con 0x7fa4ec0a2190 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): dump: 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.985 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163674819 0 0) 0x7fa4e00010f0 con 0x7fa4ec0a2570 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _negotiate 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): my global_id is 4360 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4cc003530 con 0x7fa4ec0a2570 2024-09-16T05:34:59.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3866854497 0 0) 0x7fa4ec0a7bb0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4e00010f0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa4e4001d90 con 0x7fa4ec0a2980 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient(hunting): dump: 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.987 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3781222915 0 0) 0x7fa4e00010f0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 >> v1:172.21.15.6:6790/0 conn(0x7fa4ec0a2190 legacy=0x7fa4ec13f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 >> v1:172.21.15.63:6789/0 conn(0x7fa4ec0a2570 legacy=0x7fa4ec140100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient: _finish_hunting 0 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 monclient: found mon.a 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ec12b9e0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient: _finish_auth 0 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient: _check_auth_tickets 2024-09-16T05:34:59.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4e4002050 con 0x7fa4ec0a2980 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 5 monclient: authenticate success, global_id 4350 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: _renew_subs 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ec12bcc0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa4e4003df0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4daffd640 10 monclient: dump: 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.989 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: _renew_subs 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa4ec128fb0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4e8d99640 4 set_mon_vals no callback set 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:34:59.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.971+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ec0008e0 con 0x7fa4ec0a2980 2024-09-16T05:34:59.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.975+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa4e400eb90 con 0x7fa4ec0a2980 2024-09-16T05:34:59.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.975+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7fa4e4043380 con 0x7fa4ec0a2980 2024-09-16T05:34:59.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.975+0000 7fa4ea59c640 10 monclient: get_auth_request con 0x7fa4cc036c70 auth_method 0 2024-09-16T05:34:59.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.979+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa4ec0a8100 con 0x7fa4ec0a2980 2024-09-16T05:34:59.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.979+0000 7fa4daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:34:59.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.979+0000 7fa4daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:34:59.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient: get_monmap_and_config 2024-09-16T05:34:59.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient: build_initial_monmap 2024-09-16T05:34:59.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient: monmap: 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:59.989214+0000 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:59.989214+0000 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 1 Processor -- start 2024-09-16T05:34:59.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 1 -- start start 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient: init 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient: _reopen_session rank -1 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient(hunting): picked mon.noname-b con 0x7efeec0a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient(hunting): picked mon.noname-a con 0x7efeec0a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 10 monclient(hunting): picked mon.noname-c con 0x7efeec0a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec0a33a0 con 0x7efeec0a2570 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.983+0000 7efef1ce3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec0a07a0 con 0x7efeec0a2980 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efef1ce3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec0a2d90 con 0x7efeec0a2190 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efef1ce3640 10 monclient(hunting): _renew_subs 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efef1ce3640 10 monclient(hunting): authenticate will time out at 961.796022s 2024-09-16T05:34:59.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efee3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efeec0a2570 0x7efeec13b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34282/0 (socket says 172.21.15.6:34282) 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efee3fff640 1 -- 172.21.15.6:0/827484311 learned_addr learned my addr 172.21.15.6:0/827484311 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079481744 0 0) 0x7efeec0a2d90 con 0x7efeec0a2190 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): my global_id is 4352 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed8003530 con 0x7efeec0a2190 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042428924 0 0) 0x7efeec0a33a0 con 0x7efeec0a2570 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): my global_id is 4353 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec0a2d90 con 0x7efeec0a2570 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367804919 0 0) 0x7efeec0a07a0 con 0x7efeec0a2980 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient(hunting): my global_id is 4363 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec0a33a0 con 0x7efeec0a2980 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 699893889 0 0) 0x7efed8003530 con 0x7efeec0a2190 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 >> v1:172.21.15.63:6789/0 conn(0x7efeec0a2980 legacy=0x7efeec13f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 >> v1:172.21.15.6:6789/0 conn(0x7efeec0a2570 legacy=0x7efeec13b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 monclient: found mon.noname-b 2024-09-16T05:35:00.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec0a66e0 con 0x7efeec0a2190 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: _finish_auth 0 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efef1ce3640 5 monclient: authenticate success, global_id 4352 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efef1ce3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efedc0015a0 con 0x7efeec0a2190 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: dump: 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.002 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efef1ce3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efedc001870 con 0x7efeec0a2190 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:00.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efef1ce3640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efef1ce3640 4 set_mon_vals no callback set 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 1 -- 172.21.15.6:0/827484311 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efedc001bc0 con 0x7efeec0a2190 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.987+0000 7efeeaffd640 10 monclient: dump: 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.004 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient: shutdown 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- 172.21.15.6:0/827484311 >> v1:172.21.15.6:6790/0 conn(0x7efeec0a2190 legacy=0x7efeec0a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- 172.21.15.6:0/827484311 shutdown_connections 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- 172.21.15.6:0/827484311 >> 172.21.15.6:0/827484311 conn(0x7efeec0a9850 msgr2=0x7efeec0a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- 172.21.15.6:0/827484311 shutdown_connections 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- 172.21.15.6:0/827484311 wait complete. 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient: build_initial_monmap 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient: monmap: 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:00.005 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 Processor -- start 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- start start 2024-09-16T05:35:00.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient: init 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient(hunting): picked mon.noname-c con 0x7efeec0a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient(hunting): picked mon.noname-a con 0x7efeec0a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efeeb7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7efeec0a2190 0x7efeec1cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42202/0 (socket says 172.21.15.6:42202) 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efeeb7fe640 1 -- 172.21.15.6:0/4012437608 learned_addr learned my addr 172.21.15.6:0/4012437608 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient(hunting): picked mon.noname-b con 0x7efeec0a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec1225f0 con 0x7efeec0a2570 2024-09-16T05:35:00.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec1248e0 con 0x7efeec0a2980 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec0a7fe0 con 0x7efeec0a2190 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efef1ce3640 10 monclient(hunting): authenticate will time out at 961.803644s 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789696853 0 0) 0x7efeec0a7fe0 con 0x7efeec0a2190 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efee8ff9640 10 monclient(hunting): my global_id is 4355 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.991+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efecc003530 con 0x7efeec0a2190 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4060094048 0 0) 0x7efeec1225f0 con 0x7efeec0a2570 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): my global_id is 4356 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec0a7fe0 con 0x7efeec0a2570 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 344350982 0 0) 0x7efeec1248e0 con 0x7efeec0a2980 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): my global_id is 4366 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec1225f0 con 0x7efeec0a2980 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 117288602 0 0) 0x7efecc003530 con 0x7efeec0a2190 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efeec1248e0 con 0x7efeec0a2190 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efedc0015e0 con 0x7efeec0a2190 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:00.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): dump: 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:00.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1229540772 0 0) 0x7efeec0a7fe0 con 0x7efeec0a2570 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efecc003530 con 0x7efeec0a2570 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2895685758 0 0) 0x7efeec1248e0 con 0x7efeec0a2190 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 >> v1:172.21.15.63:6789/0 conn(0x7efeec0a2980 legacy=0x7efeec1d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 >> v1:172.21.15.6:6789/0 conn(0x7efeec0a2570 legacy=0x7efeec1d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 monclient: found mon.c 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec0a0100 con 0x7efeec0a2190 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient: _finish_auth 0 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:00.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efedc003930 con 0x7efeec0a2190 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efedc003d00 con 0x7efeec0a2190 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efee8ff9640 10 monclient: dump: 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.015 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.995+0000 7efeea7fc640 4 set_mon_vals no callback set 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 5 monclient: authenticate success, global_id 4355 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 10 monclient: _renew_subs 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efeec124600 con 0x7efeec0a2190 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 10 monclient: _renew_subs 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efeec1405a0 con 0x7efeec0a2190 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeec0a3090 con 0x7efeec0a2190 2024-09-16T05:35:00.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7efedc003ea0 con 0x7efeec0a2190 2024-09-16T05:35:00.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:34:59.999+0000 7efee3fff640 10 monclient: get_auth_request con 0x7efecc036fa0 auth_method 0 2024-09-16T05:35:00.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.007+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7efedc042fd0 con 0x7efeec0a2190 2024-09-16T05:35:00.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.007+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7efedc015480 con 0x7efeec0a2190 2024-09-16T05:35:00.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.007+0000 7efee8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.007+0000 7efee8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:00.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.035+0000 7fc99966c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:35:00.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.035+0000 7fc99966c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:35:00.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.035+0000 7fc99966c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:00.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.035+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc974005380 con 0x7fc9940a2980 2024-09-16T05:35:00.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.035+0000 7fc97b7fe640 1 -- 172.21.15.6:0/83938017 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 532354816) 0x7fc98c013f30 con 0x7fc9940a2980 2024-09-16T05:35:00.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.035+0000 7fc97b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T05:35:00.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.035+0000 7fc97b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:00.043 INFO:teuthology.orchestra.run.smithi006.stdout:51539607557 2024-09-16T05:35:00.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.039+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc980036af0 legacy=0x7fc980038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.039+0000 7fc99966c640 10 monclient: shutdown 2024-09-16T05:35:00.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.039+0000 7fc99966c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.039+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 >> v1:172.21.15.6:6789/0 conn(0x7fc9940a2980 legacy=0x7fc9941217d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.039+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 shutdown_connections 2024-09-16T05:35:00.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.039+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 >> 172.21.15.6:0/83938017 conn(0x7fc9940a9850 msgr2=0x7fc99406ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.039+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 shutdown_connections 2024-09-16T05:35:00.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.039+0000 7fc99966c640 1 -- 172.21.15.6:0/83938017 wait complete. 2024-09-16T05:35:00.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-09-16T05:35:00.070 DEBUG:teuthology.parallel:result is None 2024-09-16T05:35:00.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.063+0000 7f4242e15640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:35:00.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.063+0000 7f4242e15640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:35:00.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.063+0000 7f4242e15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.063+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f423c123d00 con 0x7f423c0a2980 2024-09-16T05:35:00.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.063+0000 7f42397fa640 1 -- 172.21.15.6:0/3880834435 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 1169801742) 0x7f4234013c60 con 0x7f423c0a2980 2024-09-16T05:35:00.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.063+0000 7f42397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T05:35:00.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.063+0000 7f42397fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:00.071 INFO:teuthology.orchestra.run.smithi006.stdout:47244640261 2024-09-16T05:35:00.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.067+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4220036c80 legacy=0x7f4220039140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.067+0000 7f4242e15640 10 monclient: shutdown 2024-09-16T05:35:00.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.067+0000 7f4242e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.067+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 >> v1:172.21.15.6:6790/0 conn(0x7f423c0a2980 legacy=0x7f423c1d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.067+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 shutdown_connections 2024-09-16T05:35:00.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.067+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 >> 172.21.15.6:0/3880834435 conn(0x7f423c0a9850 msgr2=0x7f423c12f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.067+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 shutdown_connections 2024-09-16T05:35:00.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.067+0000 7f4242e15640 1 -- 172.21.15.6:0/3880834435 wait complete. 2024-09-16T05:35:00.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:35:00.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:35:00.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9da8075290 con 0x7f9da80a4290 2024-09-16T05:35:00.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9d9dffb640 1 -- 172.21.15.6:0/3733019013 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 1169801742) 0x7f9da00151f0 con 0x7f9da80a4290 2024-09-16T05:35:00.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9d9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T05:35:00.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9d9dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:00.076 INFO:teuthology.orchestra.run.smithi006.stdout:47244640261 2024-09-16T05:35:00.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9d8c036b10 legacy=0x7f9d8c038fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 10 monclient: shutdown 2024-09-16T05:35:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 >> v1:172.21.15.6:6790/0 conn(0x7f9da80a4290 legacy=0x7f9da81d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 shutdown_connections 2024-09-16T05:35:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 >> 172.21.15.6:0/3733019013 conn(0x7f9da80a58a0 msgr2=0x7f9da80a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 shutdown_connections 2024-09-16T05:35:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.071+0000 7f9daf6bb640 1 -- 172.21.15.6:0/3733019013 wait complete. 2024-09-16T05:35:00.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.3 2024-09-16T05:35:00.095 DEBUG:teuthology.parallel:result is None 2024-09-16T05:35:00.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-09-16T05:35:00.100 DEBUG:teuthology.parallel:result is None 2024-09-16T05:35:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.187+0000 7efef1ce3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:35:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.187+0000 7efef1ce3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:35:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.187+0000 7efef1ce3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.187+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7efeec075290 con 0x7efeec0a2190 2024-09-16T05:35:00.194 INFO:teuthology.orchestra.run.smithi006.stdout:51539607557 2024-09-16T05:35:00.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.187+0000 7efee8ff9640 1 -- 172.21.15.6:0/4012437608 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 532354816) 0x7efedc01d930 con 0x7efeec0a2190 2024-09-16T05:35:00.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.187+0000 7efee8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T05:35:00.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.187+0000 7efee8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 >> v1:172.21.15.63:6812/3643207211 conn(0x7efecc036fa0 legacy=0x7efecc039460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7efef1ce3640 10 monclient: shutdown 2024-09-16T05:35:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7efef1ce3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 >> v1:172.21.15.6:6790/0 conn(0x7efeec0a2190 legacy=0x7efeec1cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 shutdown_connections 2024-09-16T05:35:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 >> 172.21.15.6:0/4012437608 conn(0x7efeec0a9850 msgr2=0x7efeec06ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 shutdown_connections 2024-09-16T05:35:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7efef1ce3640 1 -- 172.21.15.6:0/4012437608 wait complete. 2024-09-16T05:35:00.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7fa4f08ba640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:35:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7fa4f08ba640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:35:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7fa4f08ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa4ec1292a0 con 0x7fa4ec0a2980 2024-09-16T05:35:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7fa4daffd640 1 -- 172.21.15.6:0/3951348711 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1444633209) 0x7fa4e40146b0 con 0x7fa4ec0a2980 2024-09-16T05:35:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7fa4daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T05:35:00.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.191+0000 7fa4daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:00.197 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2024-09-16T05:35:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.195+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa4cc036c70 legacy=0x7fa4cc039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.195+0000 7fa4f08ba640 10 monclient: shutdown 2024-09-16T05:35:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.195+0000 7fa4f08ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.195+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 >> v1:172.21.15.6:6789/0 conn(0x7fa4ec0a2980 legacy=0x7fa4ec1d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.195+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 shutdown_connections 2024-09-16T05:35:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.195+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 >> 172.21.15.6:0/3951348711 conn(0x7fa4ec0a9850 msgr2=0x7fa4ec09d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.195+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 shutdown_connections 2024-09-16T05:35:00.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.195+0000 7fa4f08ba640 1 -- 172.21.15.6:0/3951348711 wait complete. 2024-09-16T05:35:00.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-09-16T05:35:00.214 DEBUG:teuthology.parallel:result is None 2024-09-16T05:35:00.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.1 2024-09-16T05:35:00.217 DEBUG:teuthology.parallel:result is None 2024-09-16T05:35:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.231+0000 7f03254ed640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:35:00.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.231+0000 7f03254ed640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:35:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.231+0000 7f03254ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.231+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0300005380 con 0x7f03200a2980 2024-09-16T05:35:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.231+0000 7f03077fe640 1 -- 172.21.15.6:0/3323655771 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1169801742) 0x7f031801db80 con 0x7f03200a2980 2024-09-16T05:35:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.231+0000 7f03077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T05:35:00.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.231+0000 7f03077fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:00.238 INFO:teuthology.orchestra.run.smithi006.stdout:47244640261 2024-09-16T05:35:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.235+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 >> v1:172.21.15.63:6812/3643207211 conn(0x7f030c036aa0 legacy=0x7f030c038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.235+0000 7f03254ed640 10 monclient: shutdown 2024-09-16T05:35:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.235+0000 7f03254ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.235+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 >> v1:172.21.15.6:6789/0 conn(0x7f03200a2980 legacy=0x7f03201d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.235+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 shutdown_connections 2024-09-16T05:35:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.235+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 >> 172.21.15.6:0/3323655771 conn(0x7f03200a9850 msgr2=0x7f032012f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.235+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 shutdown_connections 2024-09-16T05:35:00.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.235+0000 7f03254ed640 1 -- 172.21.15.6:0/3323655771 wait complete. 2024-09-16T05:35:00.253 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.0 2024-09-16T05:35:00.254 DEBUG:teuthology.parallel:result is None 2024-09-16T05:35:00.254 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T05:35:00.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient: get_monmap_and_config 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient: build_initial_monmap 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient: monmap: 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:00.408581+0000 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:00.408581+0000 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:00.409 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:00.410 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:00.410 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 1 Processor -- start 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 1 -- start start 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient: init 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient(hunting): picked mon.noname-b con 0x7f702c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient(hunting): picked mon.noname-c con 0x7f702c09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient(hunting): picked mon.noname-a con 0x7f702c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702c0a1370 con 0x7f702c0a0da0 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702c09ead0 con 0x7f702c09e230 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702c09edb0 con 0x7f702c09aa30 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:00.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 10 monclient(hunting): authenticate will time out at 962.215129s 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f702affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f702c09aa30 0x7f702c09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42214/0 (socket says 172.21.15.6:42214) 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f702affd640 1 -- 172.21.15.6:0/1353170688 learned_addr learned my addr 172.21.15.6:0/1353170688 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f702b7fe640 1 --1- 172.21.15.6:0/1353170688 >> v1:172.21.15.6:6789/0 conn(0x7f702c0a0da0 0x7f702c13d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34298/0 (socket says 172.21.15.6:34298) 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942105665 0 0) 0x7f702c09edb0 con 0x7f702c09aa30 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient(hunting): my global_id is 4358 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7018003530 con 0x7f702c09aa30 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3132284381 0 0) 0x7f702c0a1370 con 0x7f702c0a0da0 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient(hunting): my global_id is 4359 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f702c09edb0 con 0x7f702c0a0da0 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3984569814 0 0) 0x7f7018003530 con 0x7f702c09aa30 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 >> v1:172.21.15.63:6789/0 conn(0x7f702c09e230 legacy=0x7f702c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 >> v1:172.21.15.6:6789/0 conn(0x7f702c0a0da0 legacy=0x7f702c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 monclient: found mon.noname-b 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702c09dd50 con 0x7f702c09aa30 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient: _finish_auth 0 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:35:00.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70240019a0 con 0x7f702c09aa30 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 5 monclient: authenticate success, global_id 4358 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f7029ffb640 10 monclient: dump: 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.419 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.403+0000 7f70312b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7024001b90 con 0x7f702c09aa30 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f7029ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f7029ffb640 1 -- 172.21.15.6:0/1353170688 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7024001fa0 con 0x7f702c09aa30 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 4 set_mon_vals no callback set 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f7029ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f7029ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f7029ffb640 10 monclient: dump: 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.420 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: shutdown 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- 172.21.15.6:0/1353170688 >> v1:172.21.15.6:6790/0 conn(0x7f702c09aa30 legacy=0x7f702c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- 172.21.15.6:0/1353170688 shutdown_connections 2024-09-16T05:35:00.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- 172.21.15.6:0/1353170688 >> 172.21.15.6:0/1353170688 conn(0x7f702c0a1650 msgr2=0x7f702c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- 172.21.15.6:0/1353170688 shutdown_connections 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- 172.21.15.6:0/1353170688 wait complete. 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: build_initial_monmap 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: monmap: 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:00.422 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 Processor -- start 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- start start 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: init 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient(hunting): picked mon.noname-c con 0x7f702c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient(hunting): picked mon.noname-a con 0x7f702c09e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient(hunting): picked mon.noname-b con 0x7f702c0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702c11c530 con 0x7f702c09e230 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702c11b880 con 0x7f702c0a0da0 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f702c13cd60 con 0x7f702c09aa30 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:00.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient(hunting): authenticate will time out at 962.217849s 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f702affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f702c09aa30 0x7f702c13b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42218/0 (socket says 172.21.15.6:42218) 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f702affd640 1 -- 172.21.15.6:0/1912692572 learned_addr learned my addr 172.21.15.6:0/1912692572 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2756661870 0 0) 0x7f702c11c530 con 0x7f702c09e230 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): my global_id is 4362 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f700c003530 con 0x7f702c09e230 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565380343 0 0) 0x7f702c13cd60 con 0x7f702c09aa30 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): my global_id is 4361 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f702c11c530 con 0x7f702c09aa30 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3530329489 0 0) 0x7f702c11c530 con 0x7f702c09aa30 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f702c13cd60 con 0x7f702c09aa30 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 524332584 0 0) 0x7f700c003530 con 0x7f702c09e230 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f702c11c530 con 0x7f702c09e230 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7024001ee0 con 0x7f702c09aa30 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): dump: 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.425 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7014001840 con 0x7f702c09e230 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient(hunting): dump: 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.426 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1898513979 0 0) 0x7f702c13cd60 con 0x7f702c09aa30 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 >> v1:172.21.15.63:6789/0 conn(0x7f702c0a0da0 legacy=0x7f702c13bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 >> v1:172.21.15.6:6789/0 conn(0x7f702c09e230 legacy=0x7f702c13b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient: _finish_hunting 0 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 monclient: found mon.c 2024-09-16T05:35:00.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702c13d8c0 con 0x7f702c09aa30 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient: _finish_auth 0 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient: _check_auth_tickets 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 5 monclient: authenticate success, global_id 4361 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: _renew_subs 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f702c1d2710 con 0x7f702c09aa30 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: _renew_subs 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f702c1d2ab0 con 0x7f702c09aa30 2024-09-16T05:35:00.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7024001b90 con 0x7f702c09aa30 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7024004f80 con 0x7f702c09aa30 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70297fa640 4 set_mon_vals no callback set 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 10 monclient: dump: 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.429 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f702400fd20 con 0x7f702c09aa30 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7010005090 con 0x7f702c09aa30 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.407+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f7024044500 con 0x7f702c09aa30 2024-09-16T05:35:00.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.411+0000 7f702a7fc640 10 monclient: get_auth_request con 0x7f700c036a40 auth_method 0 2024-09-16T05:35:00.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.411+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f70240157f0 con 0x7f702c09aa30 2024-09-16T05:35:00.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.411+0000 7f70237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.411+0000 7f70237fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:00.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.567+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7010000ba0 con 0x7f700c036a40 2024-09-16T05:35:00.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.571+0000 7f70237fe640 1 -- 172.21.15.6:0/1912692572 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34203 (unknown 2965378022 0 1965793931) 0x7f7010000ba0 con 0x7f700c036a40 2024-09-16T05:35:00.578 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:35:00.579 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:35:00.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.575+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 >> v1:172.21.15.63:6812/3643207211 conn(0x7f700c036a40 legacy=0x7f700c038f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.575+0000 7f70312b6640 10 monclient: shutdown 2024-09-16T05:35:00.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.575+0000 7f70312b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.575+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 >> v1:172.21.15.6:6790/0 conn(0x7f702c09aa30 legacy=0x7f702c13b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.575+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 shutdown_connections 2024-09-16T05:35:00.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.575+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 >> 172.21.15.6:0/1912692572 conn(0x7f702c0a1650 msgr2=0x7f702c071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.575+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 shutdown_connections 2024-09-16T05:35:00.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.575+0000 7f70312b6640 1 -- 172.21.15.6:0/1912692572 wait complete. 2024-09-16T05:35:00.595 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T05:34:59.261181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163272,"kb_used_data":1072,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067768,"statfs":{"total":579820584960,"available":579653394432,"internally_reserved":0,"allocated":1097728,"data_stored":1373734,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":165927586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.364593"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.104363+0000","last_change":"2024-09-16T05:34:57.104762+0000","last_active":"2024-09-16T05:34:57.104363+0000","last_peered":"2024-09-16T05:34:57.104363+0000","last_clean":"2024-09-16T05:34:57.104363+0000","last_became_active":"2024-09-16T05:34:55.334450+0000","last_became_peered":"2024-09-16T05:34:55.334450+0000","last_unstale":"2024-09-16T05:34:57.104363+0000","last_undegraded":"2024-09-16T05:34:57.104363+0000","last_fullsized":"2024-09-16T05:34:57.104363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:20.555116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.930290+0000","last_change":"2024-09-16T05:34:56.930773+0000","last_active":"2024-09-16T05:34:56.930290+0000","last_peered":"2024-09-16T05:34:56.930290+0000","last_clean":"2024-09-16T05:34:56.930290+0000","last_became_active":"2024-09-16T05:34:54.903232+0000","last_became_peered":"2024-09-16T05:34:54.903232+0000","last_unstale":"2024-09-16T05:34:56.930290+0000","last_undegraded":"2024-09-16T05:34:56.930290+0000","last_fullsized":"2024-09-16T05:34:56.930290+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:18:07.827355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103981+0000","last_change":"2024-09-16T05:34:57.104630+0000","last_active":"2024-09-16T05:34:57.103981+0000","last_peered":"2024-09-16T05:34:57.103981+0000","last_clean":"2024-09-16T05:34:57.103981+0000","last_became_active":"2024-09-16T05:34:55.167715+0000","last_became_peered":"2024-09-16T05:34:55.167715+0000","last_unstale":"2024-09-16T05:34:57.103981+0000","last_undegraded":"2024-09-16T05:34:57.103981+0000","last_fullsized":"2024-09-16T05:34:57.103981+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:17.176705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.930084+0000","last_change":"2024-09-16T05:34:56.930673+0000","last_active":"2024-09-16T05:34:56.930084+0000","last_peered":"2024-09-16T05:34:56.930084+0000","last_clean":"2024-09-16T05:34:56.930084+0000","last_became_active":"2024-09-16T05:34:55.165400+0000","last_became_peered":"2024-09-16T05:34:55.165400+0000","last_unstale":"2024-09-16T05:34:56.930084+0000","last_undegraded":"2024-09-16T05:34:56.930084+0000","last_fullsized":"2024-09-16T05:34:56.930084+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:42.301038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.940160+0000","last_change":"2024-09-16T05:34:56.938719+0000","last_active":"2024-09-16T05:34:56.940160+0000","last_peered":"2024-09-16T05:34:56.940160+0000","last_clean":"2024-09-16T05:34:56.940160+0000","last_became_active":"2024-09-16T05:34:54.914386+0000","last_became_peered":"2024-09-16T05:34:54.914386+0000","last_unstale":"2024-09-16T05:34:56.940160+0000","last_undegraded":"2024-09-16T05:34:56.940160+0000","last_fullsized":"2024-09-16T05:34:56.940160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:18.868561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.928651+0000","last_change":"2024-09-16T05:34:56.929169+0000","last_active":"2024-09-16T05:34:56.928651+0000","last_peered":"2024-09-16T05:34:56.928651+0000","last_clean":"2024-09-16T05:34:56.928651+0000","last_became_active":"2024-09-16T05:34:54.906976+0000","last_became_peered":"2024-09-16T05:34:54.906976+0000","last_unstale":"2024-09-16T05:34:56.928651+0000","last_undegraded":"2024-09-16T05:34:56.928651+0000","last_fullsized":"2024-09-16T05:34:56.928651+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:20.747928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103806+0000","last_change":"2024-09-16T05:34:57.104264+0000","last_active":"2024-09-16T05:34:57.103806+0000","last_peered":"2024-09-16T05:34:57.103806+0000","last_clean":"2024-09-16T05:34:57.103806+0000","last_became_active":"2024-09-16T05:34:54.909066+0000","last_became_peered":"2024-09-16T05:34:54.909066+0000","last_unstale":"2024-09-16T05:34:57.103806+0000","last_undegraded":"2024-09-16T05:34:57.103806+0000","last_fullsized":"2024-09-16T05:34:57.103806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:12.435781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.937943+0000","last_change":"2024-09-16T05:34:56.937943+0000","last_active":"2024-09-16T05:34:56.937943+0000","last_peered":"2024-09-16T05:34:56.937943+0000","last_clean":"2024-09-16T05:34:56.937943+0000","last_became_active":"2024-09-16T05:34:54.914265+0000","last_became_peered":"2024-09-16T05:34:54.914265+0000","last_unstale":"2024-09-16T05:34:56.937943+0000","last_undegraded":"2024-09-16T05:34:56.937943+0000","last_fullsized":"2024-09-16T05:34:56.937943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:32.558807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103169+0000","last_change":"2024-09-16T05:34:49.007631+0000","last_active":"2024-09-16T05:34:57.103169+0000","last_peered":"2024-09-16T05:34:57.103169+0000","last_clean":"2024-09-16T05:34:57.103169+0000","last_became_active":"2024-09-16T05:34:49.004914+0000","last_became_peered":"2024-09-16T05:34:49.004914+0000","last_unstale":"2024-09-16T05:34:57.103169+0000","last_undegraded":"2024-09-16T05:34:57.103169+0000","last_fullsized":"2024-09-16T05:34:57.103169+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_clean_scrub_stamp":"2024-09-16T05:34:47.359645+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:47:04.580831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":217088,"data_stored":581568,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":581568,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:35:00.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.695+0000 7f3d8f0a6640 10 monclient: get_monmap_and_config 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.695+0000 7f3d8f0a6640 10 monclient: build_initial_monmap 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.695+0000 7f3d8f0a6640 10 monclient: monmap: 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:00.701792+0000 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:00.701792+0000 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:00.703 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:00.703 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.695+0000 7f3d8f0a6640 1 Processor -- start 2024-09-16T05:35:00.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- start start 2024-09-16T05:35:00.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: init 2024-09-16T05:35:00.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:00.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:35:00.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): picked mon.noname-b con 0x7f3d880a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:00.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): picked mon.noname-a con 0x7f3d880a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): picked mon.noname-c con 0x7f3d880a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8806e520 con 0x7f3d880a4290 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d880a0780 con 0x7f3d880a1a00 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d880a4c70 con 0x7f3d880a37a0 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): authenticate will time out at 962.508868s 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8ce42640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3d880a37a0 0x7f3d880a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42224/0 (socket says 172.21.15.6:42224) 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8ce42640 1 -- 172.21.15.6:0/1541126465 learned_addr learned my addr 172.21.15.6:0/1541126465 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d87fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3d880a4290 0x7f3d8813bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34320/0 (socket says 172.21.15.6:34320) 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 232398004 0 0) 0x7f3d880a4c70 con 0x7f3d880a37a0 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient(hunting): my global_id is 4364 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d78003530 con 0x7f3d880a37a0 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174751036 0 0) 0x7f3d8806e520 con 0x7f3d880a4290 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient(hunting): my global_id is 4365 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d880a4c70 con 0x7f3d880a4290 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2659738864 0 0) 0x7f3d78003530 con 0x7f3d880a37a0 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 >> v1:172.21.15.63:6789/0 conn(0x7f3d880a1a00 legacy=0x7f3d8813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 >> v1:172.21.15.6:6789/0 conn(0x7f3d880a4290 legacy=0x7f3d8813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: _finish_hunting 0 2024-09-16T05:35:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 monclient: found mon.noname-b 2024-09-16T05:35:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d880a65f0 con 0x7f3d880a37a0 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: _finish_auth 0 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: _check_auth_tickets 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d7c001620 con 0x7f3d880a37a0 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: dump: 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 5 monclient: authenticate success, global_id 4364 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d7c001880 con 0x7f3d880a37a0 2024-09-16T05:35:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 1 -- 172.21.15.6:0/1541126465 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d7c001c50 con 0x7f3d880a37a0 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 4 set_mon_vals no callback set 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d877fe640 10 monclient: dump: 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: shutdown 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1541126465 >> v1:172.21.15.6:6790/0 conn(0x7f3d880a37a0 legacy=0x7f3d880a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1541126465 shutdown_connections 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1541126465 >> 172.21.15.6:0/1541126465 conn(0x7f3d880a58a0 msgr2=0x7f3d880a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1541126465 shutdown_connections 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1541126465 wait complete. 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: build_initial_monmap 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: monmap: 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:00.715 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 Processor -- start 2024-09-16T05:35:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- start start 2024-09-16T05:35:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: init 2024-09-16T05:35:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): picked mon.noname-a con 0x7f3d880a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): picked mon.noname-b con 0x7f3d880a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): picked mon.noname-c con 0x7f3d880a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8813f000 con 0x7f3d880a1a00 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d880a8080 con 0x7f3d880a37a0 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d880a0100 con 0x7f3d880a4290 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8f0a6640 10 monclient(hunting): authenticate will time out at 962.511817s 2024-09-16T05:35:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8ce42640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3d880a1a00 0x7f3d881c8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34334/0 (socket says 172.21.15.6:34334) 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8ce42640 1 -- 172.21.15.6:0/1183570773 learned_addr learned my addr 172.21.15.6:0/1183570773 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.699+0000 7f3d8d643640 1 --1- 172.21.15.6:0/1183570773 >> v1:172.21.15.6:6790/0 conn(0x7f3d880a4290 0x7f3d881d3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42226/0 (socket says 172.21.15.6:42226) 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 969524178 0 0) 0x7f3d880a0100 con 0x7f3d880a4290 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): my global_id is 4367 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d6c003530 con 0x7f3d880a4290 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008601307 0 0) 0x7f3d8813f000 con 0x7f3d880a1a00 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): _negotiate 2024-09-16T05:35:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): my global_id is 4368 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d880a0100 con 0x7f3d880a1a00 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3720622796 0 0) 0x7f3d6c003530 con 0x7f3d880a4290 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d8813f000 con 0x7f3d880a4290 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d80001b40 con 0x7f3d880a4290 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): dump: 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 789226108 0 0) 0x7f3d880a0100 con 0x7f3d880a1a00 2024-09-16T05:35:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d6c003530 con 0x7f3d880a1a00 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d7c001bd0 con 0x7f3d880a1a00 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient(hunting): dump: 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1607685678 0 0) 0x7f3d8813f000 con 0x7f3d880a4290 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 >> v1:172.21.15.63:6789/0 conn(0x7f3d880a37a0 legacy=0x7f3d881d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 >> v1:172.21.15.6:6789/0 conn(0x7f3d880a1a00 legacy=0x7f3d881c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient: _finish_hunting 0 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 monclient: found mon.c 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8813fba0 con 0x7f3d880a4290 2024-09-16T05:35:00.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient: _finish_auth 0 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient: _check_auth_tickets 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 5 monclient: authenticate success, global_id 4367 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 10 monclient: _renew_subs 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d8813ed20 con 0x7f3d880a4290 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 10 monclient: _renew_subs 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d8813f780 con 0x7f3d880a4290 2024-09-16T05:35:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d800020b0 con 0x7f3d880a4290 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d80003b40 con 0x7f3d880a4290 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d86ffd640 4 set_mon_vals no callback set 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 10 monclient: dump: 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:00.726 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d68005090 con 0x7f3d880a4290 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3d80003ce0 con 0x7f3d880a4290 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 306177491 0 0) 0x7f3d80044310 con 0x7f3d880a4290 2024-09-16T05:35:00.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.703+0000 7f3d8ce42640 10 monclient: get_auth_request con 0x7f3d6c036b60 auth_method 0 2024-09-16T05:35:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.707+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3d80015640 con 0x7f3d880a4290 2024-09-16T05:35:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.707+0000 7f3d857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.707+0000 7f3d857fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.863+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d68000ba0 con 0x7f3d6c036b60 2024-09-16T05:35:00.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.863+0000 7f3d857fa640 1 -- 172.21.15.6:0/1183570773 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34203 (unknown 2965378022 0 1965793931) 0x7f3d68000ba0 con 0x7f3d6c036b60 2024-09-16T05:35:00.871 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:35:00.871 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:35:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.867+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3d6c036b60 legacy=0x7f3d6c039020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.867+0000 7f3d8f0a6640 10 monclient: shutdown 2024-09-16T05:35:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.867+0000 7f3d8f0a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.867+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 >> v1:172.21.15.6:6790/0 conn(0x7f3d880a4290 legacy=0x7f3d881d3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.867+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 shutdown_connections 2024-09-16T05:35:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.867+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 >> 172.21.15.6:0/1183570773 conn(0x7f3d880a58a0 msgr2=0x7f3d8809f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.867+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 shutdown_connections 2024-09-16T05:35:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.867+0000 7f3d8f0a6640 1 -- 172.21.15.6:0/1183570773 wait complete. 2024-09-16T05:35:00.888 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T05:34:59.261181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163272,"kb_used_data":1072,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067768,"statfs":{"total":579820584960,"available":579653394432,"internally_reserved":0,"allocated":1097728,"data_stored":1373734,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":165927586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.364593"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.104363+0000","last_change":"2024-09-16T05:34:57.104762+0000","last_active":"2024-09-16T05:34:57.104363+0000","last_peered":"2024-09-16T05:34:57.104363+0000","last_clean":"2024-09-16T05:34:57.104363+0000","last_became_active":"2024-09-16T05:34:55.334450+0000","last_became_peered":"2024-09-16T05:34:55.334450+0000","last_unstale":"2024-09-16T05:34:57.104363+0000","last_undegraded":"2024-09-16T05:34:57.104363+0000","last_fullsized":"2024-09-16T05:34:57.104363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:20.555116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.930290+0000","last_change":"2024-09-16T05:34:56.930773+0000","last_active":"2024-09-16T05:34:56.930290+0000","last_peered":"2024-09-16T05:34:56.930290+0000","last_clean":"2024-09-16T05:34:56.930290+0000","last_became_active":"2024-09-16T05:34:54.903232+0000","last_became_peered":"2024-09-16T05:34:54.903232+0000","last_unstale":"2024-09-16T05:34:56.930290+0000","last_undegraded":"2024-09-16T05:34:56.930290+0000","last_fullsized":"2024-09-16T05:34:56.930290+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:18:07.827355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103981+0000","last_change":"2024-09-16T05:34:57.104630+0000","last_active":"2024-09-16T05:34:57.103981+0000","last_peered":"2024-09-16T05:34:57.103981+0000","last_clean":"2024-09-16T05:34:57.103981+0000","last_became_active":"2024-09-16T05:34:55.167715+0000","last_became_peered":"2024-09-16T05:34:55.167715+0000","last_unstale":"2024-09-16T05:34:57.103981+0000","last_undegraded":"2024-09-16T05:34:57.103981+0000","last_fullsized":"2024-09-16T05:34:57.103981+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:17.176705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.930084+0000","last_change":"2024-09-16T05:34:56.930673+0000","last_active":"2024-09-16T05:34:56.930084+0000","last_peered":"2024-09-16T05:34:56.930084+0000","last_clean":"2024-09-16T05:34:56.930084+0000","last_became_active":"2024-09-16T05:34:55.165400+0000","last_became_peered":"2024-09-16T05:34:55.165400+0000","last_unstale":"2024-09-16T05:34:56.930084+0000","last_undegraded":"2024-09-16T05:34:56.930084+0000","last_fullsized":"2024-09-16T05:34:56.930084+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:42.301038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.940160+0000","last_change":"2024-09-16T05:34:56.938719+0000","last_active":"2024-09-16T05:34:56.940160+0000","last_peered":"2024-09-16T05:34:56.940160+0000","last_clean":"2024-09-16T05:34:56.940160+0000","last_became_active":"2024-09-16T05:34:54.914386+0000","last_became_peered":"2024-09-16T05:34:54.914386+0000","last_unstale":"2024-09-16T05:34:56.940160+0000","last_undegraded":"2024-09-16T05:34:56.940160+0000","last_fullsized":"2024-09-16T05:34:56.940160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:18.868561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.928651+0000","last_change":"2024-09-16T05:34:56.929169+0000","last_active":"2024-09-16T05:34:56.928651+0000","last_peered":"2024-09-16T05:34:56.928651+0000","last_clean":"2024-09-16T05:34:56.928651+0000","last_became_active":"2024-09-16T05:34:54.906976+0000","last_became_peered":"2024-09-16T05:34:54.906976+0000","last_unstale":"2024-09-16T05:34:56.928651+0000","last_undegraded":"2024-09-16T05:34:56.928651+0000","last_fullsized":"2024-09-16T05:34:56.928651+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:20.747928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103806+0000","last_change":"2024-09-16T05:34:57.104264+0000","last_active":"2024-09-16T05:34:57.103806+0000","last_peered":"2024-09-16T05:34:57.103806+0000","last_clean":"2024-09-16T05:34:57.103806+0000","last_became_active":"2024-09-16T05:34:54.909066+0000","last_became_peered":"2024-09-16T05:34:54.909066+0000","last_unstale":"2024-09-16T05:34:57.103806+0000","last_undegraded":"2024-09-16T05:34:57.103806+0000","last_fullsized":"2024-09-16T05:34:57.103806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:12.435781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.937943+0000","last_change":"2024-09-16T05:34:56.937943+0000","last_active":"2024-09-16T05:34:56.937943+0000","last_peered":"2024-09-16T05:34:56.937943+0000","last_clean":"2024-09-16T05:34:56.937943+0000","last_became_active":"2024-09-16T05:34:54.914265+0000","last_became_peered":"2024-09-16T05:34:54.914265+0000","last_unstale":"2024-09-16T05:34:56.937943+0000","last_undegraded":"2024-09-16T05:34:56.937943+0000","last_fullsized":"2024-09-16T05:34:56.937943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:32.558807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103169+0000","last_change":"2024-09-16T05:34:49.007631+0000","last_active":"2024-09-16T05:34:57.103169+0000","last_peered":"2024-09-16T05:34:57.103169+0000","last_clean":"2024-09-16T05:34:57.103169+0000","last_became_active":"2024-09-16T05:34:49.004914+0000","last_became_peered":"2024-09-16T05:34:49.004914+0000","last_unstale":"2024-09-16T05:34:57.103169+0000","last_undegraded":"2024-09-16T05:34:57.103169+0000","last_fullsized":"2024-09-16T05:34:57.103169+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_clean_scrub_stamp":"2024-09-16T05:34:47.359645+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:47:04.580831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":217088,"data_stored":581568,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":581568,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:35:00.889 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T05:35:00.889 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T05:35:00.889 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T05:35:00.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T05:35:01.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient: get_monmap_and_config 2024-09-16T05:35:01.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient: build_initial_monmap 2024-09-16T05:35:01.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient: monmap: 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:01.000261+0000 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:01.000261+0000 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:01.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 1 Processor -- start 2024-09-16T05:35:01.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 1 -- start start 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient: init 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient(hunting): picked mon.noname-a con 0x7fd23c09bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient(hunting): picked mon.noname-c con 0x7fd23c09bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient(hunting): picked mon.noname-b con 0x7fd23c1397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c0a1350 con 0x7fd23c09bb40 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c0a10d0 con 0x7fd23c09bf20 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c09d940 con 0x7fd23c1397f0 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 10 monclient(hunting): authenticate will time out at 962.807108s 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd241c8f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd23c1397f0 0x7fd23c13bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42242/0 (socket says 172.21.15.6:42242) 2024-09-16T05:35:01.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd241c8f640 1 -- 172.21.15.6:0/2399829345 learned_addr learned my addr 172.21.15.6:0/2399829345 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd24148e640 1 --1- 172.21.15.6:0/2399829345 >> v1:172.21.15.6:6789/0 conn(0x7fd23c09bb40 0x7fd23c09e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34338/0 (socket says 172.21.15.6:34338) 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3487712923 0 0) 0x7fd23c0a1350 con 0x7fd23c09bb40 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient(hunting): my global_id is 4371 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd228003530 con 0x7fd23c09bb40 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4015297488 0 0) 0x7fd23c09d940 con 0x7fd23c1397f0 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient(hunting): my global_id is 4370 2024-09-16T05:35:01.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd23c0a1350 con 0x7fd23c1397f0 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2194568987 0 0) 0x7fd228003530 con 0x7fd23c09bb40 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 >> v1:172.21.15.6:6790/0 conn(0x7fd23c1397f0 legacy=0x7fd23c13bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 >> v1:172.21.15.63:6789/0 conn(0x7fd23c09bf20 legacy=0x7fd23c09c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient: _finish_hunting 0 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 monclient: found mon.noname-a 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23c09dc20 con 0x7fd23c09bb40 2024-09-16T05:35:01.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient: _finish_auth 0 2024-09-16T05:35:01.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient: _check_auth_tickets 2024-09-16T05:35:01.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:01.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd22c001620 con 0x7fd23c09bb40 2024-09-16T05:35:01.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 5 monclient: authenticate success, global_id 4371 2024-09-16T05:35:01.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:01.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:01.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd233fff640 10 monclient: dump: 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.995+0000 7fd2436f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd22c001890 con 0x7fd23c09bb40 2024-09-16T05:35:01.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd233fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd233fff640 1 -- 172.21.15.6:0/2399829345 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd22c001ca0 con 0x7fd23c09bb40 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 4 set_mon_vals no callback set 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd233fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd233fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd233fff640 10 monclient: dump: 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.013 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: shutdown 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- 172.21.15.6:0/2399829345 >> v1:172.21.15.6:6789/0 conn(0x7fd23c09bb40 legacy=0x7fd23c09e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- 172.21.15.6:0/2399829345 shutdown_connections 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- 172.21.15.6:0/2399829345 >> 172.21.15.6:0/2399829345 conn(0x7fd23c0a1650 msgr2=0x7fd23c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- 172.21.15.6:0/2399829345 shutdown_connections 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- 172.21.15.6:0/2399829345 wait complete. 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: build_initial_monmap 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: monmap: 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:01.014 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 Processor -- start 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- start start 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: init 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:01.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient(hunting): picked mon.noname-b con 0x7fd23c09bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient(hunting): picked mon.noname-a con 0x7fd23c09bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient(hunting): picked mon.noname-c con 0x7fd23c1397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c071120 con 0x7fd23c09bf20 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c0a39a0 con 0x7fd23c09bb40 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c0a3df0 con 0x7fd23c1397f0 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient(hunting): authenticate will time out at 962.809930s 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd241c8f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd23c1397f0 0x7fd23c1cf7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42258/0 (socket says 172.21.15.6:42258) 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd241c8f640 1 -- 172.21.15.6:0/1902170859 learned_addr learned my addr 172.21.15.6:0/1902170859 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd240c8d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd23c09bf20 0x7fd23c1cc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34354/0 (socket says 172.21.15.6:34354) 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1309404261 0 0) 0x7fd23c071120 con 0x7fd23c09bf20 2024-09-16T05:35:01.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): my global_id is 4374 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd220003530 con 0x7fd23c09bf20 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3043109285 0 0) 0x7fd23c0a3df0 con 0x7fd23c1397f0 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): my global_id is 4373 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd23c071120 con 0x7fd23c1397f0 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2208405833 0 0) 0x7fd23c071120 con 0x7fd23c1397f0 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd23c0a3df0 con 0x7fd23c1397f0 2024-09-16T05:35:01.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd234001660 con 0x7fd23c1397f0 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): dump: 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2057346315 0 0) 0x7fd220003530 con 0x7fd23c09bf20 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd23c071120 con 0x7fd23c09bf20 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd238001840 con 0x7fd23c09bf20 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): dump: 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.019 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2065091153 0 0) 0x7fd23c0a39a0 con 0x7fd23c09bb40 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient(hunting): my global_id is 4384 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd220003530 con 0x7fd23c09bb40 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3365365307 0 0) 0x7fd23c0a3df0 con 0x7fd23c1397f0 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 >> v1:172.21.15.63:6789/0 conn(0x7fd23c09bb40 legacy=0x7fd23c12e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 >> v1:172.21.15.6:6789/0 conn(0x7fd23c09bf20 legacy=0x7fd23c1cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:35:01.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 monclient: found mon.c 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23c070e40 con 0x7fd23c1397f0 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient: _finish_auth 0 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 5 monclient: authenticate success, global_id 4373 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: _renew_subs 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd23c06f710 con 0x7fd23c1397f0 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: _renew_subs 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd23c1380b0 con 0x7fd23c1397f0 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd234004b70 con 0x7fd23c1397f0 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd234004ec0 con 0x7fd23c1397f0 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 10 monclient: dump: 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2337fe640 4 set_mon_vals no callback set 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21c005090 con 0x7fd23c1397f0 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:00.999+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd23400fc60 con 0x7fd23c1397f0 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.003+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7fd234044400 con 0x7fd23c1397f0 2024-09-16T05:35:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.003+0000 7fd24148e640 10 monclient: get_auth_request con 0x7fd220036c20 auth_method 0 2024-09-16T05:35:01.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.007+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd2340156b0 con 0x7fd23c1397f0 2024-09-16T05:35:01.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.007+0000 7fd231ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.007+0000 7fd231ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.195+0000 7fd2436f2640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T05:35:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.195+0000 7fd2436f2640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T05:35:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.195+0000 7fd2436f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.195+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd21c005380 con 0x7fd23c1397f0 2024-09-16T05:35:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.195+0000 7fd231ffb640 1 -- 172.21.15.6:0/1902170859 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7fd234019360 con 0x7fd23c1397f0 2024-09-16T05:35:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.195+0000 7fd231ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T05:35:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.195+0000 7fd231ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:01.203 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:35:01.203 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T05:35:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.199+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd220036c20 legacy=0x7fd2200390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.199+0000 7fd2436f2640 10 monclient: shutdown 2024-09-16T05:35:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.199+0000 7fd2436f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.199+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 >> v1:172.21.15.6:6790/0 conn(0x7fd23c1397f0 legacy=0x7fd23c1cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.199+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 shutdown_connections 2024-09-16T05:35:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.199+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 >> 172.21.15.6:0/1902170859 conn(0x7fd23c0a1650 msgr2=0x7fd23c12fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.199+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 shutdown_connections 2024-09-16T05:35:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.199+0000 7fd2436f2640 1 -- 172.21.15.6:0/1902170859 wait complete. 2024-09-16T05:35:01.220 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T05:35:01.221 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T05:35:01.237 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T05:35:01.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 10 monclient: get_monmap_and_config 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 10 monclient: build_initial_monmap 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 10 monclient: monmap: 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:01.357565+0000 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:01.357565+0000 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:01.358 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:01.359 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 1 Processor -- start 2024-09-16T05:35:01.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 1 -- start start 2024-09-16T05:35:01.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 10 monclient: init 2024-09-16T05:35:01.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:01.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:01.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 10 monclient(hunting): picked mon.noname-c con 0x7f1eac09aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:01.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.351+0000 7f1eb206e640 10 monclient(hunting): picked mon.noname-a con 0x7f1eac09e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient(hunting): picked mon.noname-b con 0x7f1eac0a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac0a1370 con 0x7f1eac09e230 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac09ead0 con 0x7f1eac09aa30 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac09edb0 con 0x7f1eac0a0da0 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient(hunting): authenticate will time out at 963.164069s 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eabfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1eac0a0da0 0x7f1eac13d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42260/0 (socket says 172.21.15.6:42260) 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eabfff640 1 -- 172.21.15.6:0/1254254550 learned_addr learned my addr 172.21.15.6:0/1254254550 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:01.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903843769 0 0) 0x7f1eac09edb0 con 0x7f1eac0a0da0 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient(hunting): my global_id is 4376 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e94003530 con 0x7f1eac0a0da0 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3240332917 0 0) 0x7f1eac0a1370 con 0x7f1eac09e230 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient(hunting): my global_id is 4377 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eac09edb0 con 0x7f1eac09e230 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1095488380 0 0) 0x7f1e94003530 con 0x7f1eac0a0da0 2024-09-16T05:35:01.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 >> v1:172.21.15.63:6789/0 conn(0x7f1eac09aa30 legacy=0x7f1eac09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 >> v1:172.21.15.6:6789/0 conn(0x7f1eac09e230 legacy=0x7f1eac1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:35:01.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 monclient: found mon.noname-b 2024-09-16T05:35:01.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eac09dd50 con 0x7f1eac0a0da0 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: _finish_auth 0 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 5 monclient: authenticate success, global_id 4376 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1ea40019a0 con 0x7f1eac0a0da0 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:01.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: dump: 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ea4001b90 con 0x7f1eac0a0da0 2024-09-16T05:35:01.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 1 -- 172.21.15.6:0/1254254550 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1ea4001fa0 con 0x7f1eac0a0da0 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eaa7fc640 10 monclient: dump: 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.371 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.372 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.372 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:01.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 4 set_mon_vals no callback set 2024-09-16T05:35:01.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: shutdown 2024-09-16T05:35:01.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:01.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- 172.21.15.6:0/1254254550 >> v1:172.21.15.6:6790/0 conn(0x7f1eac0a0da0 legacy=0x7f1eac13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- 172.21.15.6:0/1254254550 shutdown_connections 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- 172.21.15.6:0/1254254550 >> 172.21.15.6:0/1254254550 conn(0x7f1eac0a1650 msgr2=0x7f1eac0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- 172.21.15.6:0/1254254550 shutdown_connections 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- 172.21.15.6:0/1254254550 wait complete. 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: build_initial_monmap 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: monmap: 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:01.373 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 Processor -- start 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- start start 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: init 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:01.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient(hunting): picked mon.noname-c con 0x7f1eac09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient(hunting): picked mon.noname-b con 0x7f1eac09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient(hunting): picked mon.noname-a con 0x7f1eac0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac0740c0 con 0x7f1eac0a0da0 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac11c8f0 con 0x7f1eac09e230 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac135270 con 0x7f1eac09aa30 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient(hunting): authenticate will time out at 963.167002s 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eabfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1eac0a0da0 0x7f1eac13c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34364/0 (socket says 172.21.15.6:34364) 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eab7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1eac09aa30 0x7f1eac13b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42272/0 (socket says 172.21.15.6:42272) 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eabfff640 1 -- 172.21.15.6:0/1742217394 learned_addr learned my addr 172.21.15.6:0/1742217394 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048408181 0 0) 0x7f1eac135270 con 0x7f1eac09aa30 2024-09-16T05:35:01.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): my global_id is 4379 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e98003530 con 0x7f1eac09aa30 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1977285810 0 0) 0x7f1eac09dd50 con 0x7f1eac0a0da0 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): my global_id is 4380 2024-09-16T05:35:01.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eac135270 con 0x7f1eac0a0da0 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1728553158 0 0) 0x7f1e98003530 con 0x7f1eac09aa30 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1eac09dd50 con 0x7f1eac09aa30 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e9c0018c0 con 0x7f1eac09aa30 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): dump: 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.377 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 426677525 0 0) 0x7f1eac135270 con 0x7f1eac0a0da0 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e98003530 con 0x7f1eac0a0da0 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1ea4001ee0 con 0x7f1eac0a0da0 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient(hunting): dump: 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2702058825 0 0) 0x7f1eac09dd50 con 0x7f1eac09aa30 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 >> v1:172.21.15.63:6789/0 conn(0x7f1eac09e230 legacy=0x7f1eac13b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 >> v1:172.21.15.6:6789/0 conn(0x7f1eac0a0da0 legacy=0x7f1eac13c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient: _finish_hunting 0 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 monclient: found mon.c 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eac06e6f0 con 0x7f1eac09aa30 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient: _finish_auth 0 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 10 monclient: _check_auth_tickets 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1e93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 5 monclient: authenticate success, global_id 4379 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: _renew_subs 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1eac13ce90 con 0x7f1eac09aa30 2024-09-16T05:35:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: _renew_subs 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.355+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1eac1d27a0 con 0x7f1eac09aa30 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1e9c003620 con 0x7f1eac09aa30 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1e93fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e9c0039f0 con 0x7f1eac09aa30 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1e93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1e93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1ea9ffb640 4 set_mon_vals no callback set 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1e93fff640 10 monclient: dump: 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.381 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1e9c003b90 con 0x7f1eac09aa30 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1eb206e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7f1e9c0440f0 con 0x7f1eac09aa30 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1eb206e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1eb206e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e8c005090 con 0x7f1eac09aa30 2024-09-16T05:35:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.359+0000 7f1eaaffd640 10 monclient: get_auth_request con 0x7f1e98036b40 auth_method 0 2024-09-16T05:35:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.363+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1e9c015400 con 0x7f1eac09aa30 2024-09-16T05:35:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.363+0000 7f1e93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.363+0000 7f1e93fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:01.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.527+0000 7f1eb206e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:35:01.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.527+0000 7f1eb206e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:35:01.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.527+0000 7f1eb206e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.527+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e8c005380 con 0x7f1eac09aa30 2024-09-16T05:35:01.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.527+0000 7f1e93fff640 1 -- 172.21.15.6:0/1742217394 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10846 (unknown 3413328019 0 2435609715) 0x7f1e9c014d00 con 0x7f1eac09aa30 2024-09-16T05:35:01.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.527+0000 7f1e93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:35:01.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.527+0000 7f1e93fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:01.535 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:35:01.535 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":20,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","created":"2024-09-16T05:34:36.231874+0000","modified":"2024-09-16T05:35:00.954361+0000","last_up_change":"2024-09-16T05:34:46.347254+0000","last_in_change":"2024-09-16T05:34:37.929266+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:34:47.340654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T05:34:53.191455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28573ac1-022a-424e-a024-a8ae14680604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":3069057739}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":3069057739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":3069057739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":3069057739}]},"public_addr":"172.21.15.6:6804/3069057739","cluster_addr":"172.21.15.6:6805/3069057739","heartbeat_back_addr":"172.21.15.6:6807/3069057739","heartbeat_front_addr":"172.21.15.6:6806/3069057739","state":["exists","up"]},{"osd":1,"uuid":"5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":4098222447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":4098222447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":4098222447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":4098222447}]},"public_addr":"172.21.15.6:6808/4098222447","cluster_addr":"172.21.15.6:6809/4098222447","heartbeat_back_addr":"172.21.15.6:6811/4098222447","heartbeat_front_addr":"172.21.15.6:6810/4098222447","state":["exists","up"]},{"osd":2,"uuid":"89377298-ce68-4fd4-bb63-2ffd60eeea3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":1572197727}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":1572197727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":1572197727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":1572197727}]},"public_addr":"172.21.15.6:6800/1572197727","cluster_addr":"172.21.15.6:6801/1572197727","heartbeat_back_addr":"172.21.15.6:6803/1572197727","heartbeat_front_addr":"172.21.15.6:6802/1572197727","state":["exists","up"]},{"osd":3,"uuid":"7c5624a2-d552-4295-b112-d87972274f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6800","nonce":709309215}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":709309215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":709309215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":709309215}]},"public_addr":"172.21.15.63:6800/709309215","cluster_addr":"172.21.15.63:6801/709309215","heartbeat_back_addr":"172.21.15.63:6803/709309215","heartbeat_front_addr":"172.21.15.63:6802/709309215","state":["exists","up"]},{"osd":4,"uuid":"52fa74b0-2bf6-4c82-acb0-4032eca1f08b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":118134224}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":118134224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":118134224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":118134224}]},"public_addr":"172.21.15.63:6804/118134224","cluster_addr":"172.21.15.63:6805/118134224","heartbeat_back_addr":"172.21.15.63:6807/118134224","heartbeat_front_addr":"172.21.15.63:6806/118134224","state":["exists","up"]},{"osd":5,"uuid":"a4dac512-ff9b-4381-9065-82bf9a711bdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":165284194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":165284194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":165284194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":165284194}]},"public_addr":"172.21.15.63:6808/165284194","cluster_addr":"172.21.15.63:6809/165284194","heartbeat_back_addr":"172.21.15.63:6811/165284194","heartbeat_front_addr":"172.21.15.63:6810/165284194","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.065166+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.182281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.088394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.071475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.333358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.397991+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:35:01.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.531+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1e98036b40 legacy=0x7f1e98039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.531+0000 7f1eb206e640 10 monclient: shutdown 2024-09-16T05:35:01.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.531+0000 7f1eb206e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:01.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.531+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 >> v1:172.21.15.6:6790/0 conn(0x7f1eac09aa30 legacy=0x7f1eac13b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.531+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 shutdown_connections 2024-09-16T05:35:01.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.531+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 >> 172.21.15.6:0/1742217394 conn(0x7f1eac0a1650 msgr2=0x7f1eac071c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:01.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.531+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 shutdown_connections 2024-09-16T05:35:01.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.531+0000 7f1eb206e640 1 -- 172.21.15.6:0/1742217394 wait complete. 2024-09-16T05:35:01.553 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T05:34:47.340654+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T05:34:53.191455+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T05:35:01.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T05:35:01.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.667+0000 7f86d1a3e640 10 monclient: get_monmap_and_config 2024-09-16T05:35:01.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.667+0000 7f86d1a3e640 10 monclient: build_initial_monmap 2024-09-16T05:35:01.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.667+0000 7f86d1a3e640 10 monclient: monmap: 2024-09-16T05:35:01.674 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:01.674 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.674 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:01.673528+0000 2024-09-16T05:35:01.674 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:01.673528+0000 2024-09-16T05:35:01.674 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:01.674 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:01.675 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:01.675 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.667+0000 7f86d1a3e640 1 Processor -- start 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.667+0000 7f86d1a3e640 1 -- start start 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.667+0000 7f86d1a3e640 10 monclient: init 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.667+0000 7f86d1a3e640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.667+0000 7f86d1a3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f86cc0a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f86cc0a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f86cc0a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86cc0a33a0 con 0x7f86cc0a2190 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86cc0a07a0 con 0x7f86cc0a2980 2024-09-16T05:35:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86cc0a2d90 con 0x7f86cc0a2570 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): authenticate will time out at 963.480104s 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86ca7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f86cc0a2570 0x7f86cc13b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42286/0 (socket says 172.21.15.6:42286) 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86caffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86cc0a2190 0x7f86cc0a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34366/0 (socket says 172.21.15.6:34366) 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86ca7fc640 1 -- 172.21.15.6:0/3548291401 learned_addr learned my addr 172.21.15.6:0/3548291401 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1248394814 0 0) 0x7f86cc0a33a0 con 0x7f86cc0a2190 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient(hunting): my global_id is 4383 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86b8003530 con 0x7f86cc0a2190 2024-09-16T05:35:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2466426942 0 0) 0x7f86cc0a2d90 con 0x7f86cc0a2570 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient(hunting): my global_id is 4382 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86cc0a33a0 con 0x7f86cc0a2570 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 858887200 0 0) 0x7f86b8003530 con 0x7f86cc0a2190 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 >> v1:172.21.15.6:6790/0 conn(0x7f86cc0a2570 legacy=0x7f86cc13b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 >> v1:172.21.15.63:6789/0 conn(0x7f86cc0a2980 legacy=0x7f86cc13f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:35:01.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 monclient: found mon.noname-a 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86cc0a66e0 con 0x7f86cc0a2190 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86b4001550 con 0x7f86cc0a2190 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: dump: 2024-09-16T05:35:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 5 monclient: authenticate success, global_id 4383 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86b4001830 con 0x7f86cc0a2190 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 1 -- 172.21.15.6:0/3548291401 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86b4001c00 con 0x7f86cc0a2190 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c9ffb640 10 monclient: dump: 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.687 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 4 set_mon_vals no callback set 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient: shutdown 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3548291401 >> v1:172.21.15.6:6789/0 conn(0x7f86cc0a2190 legacy=0x7f86cc0a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3548291401 shutdown_connections 2024-09-16T05:35:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3548291401 >> 172.21.15.6:0/3548291401 conn(0x7f86cc0a9850 msgr2=0x7f86cc0a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3548291401 shutdown_connections 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3548291401 wait complete. 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient: build_initial_monmap 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient: monmap: 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 Processor -- start 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- start start 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient: init 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:01.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f86cc0a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f86cc0a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f86cc0a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86cc124450 con 0x7f86cc0a2570 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86cc12d7d0 con 0x7f86cc0a2190 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86cc0a8460 con 0x7f86cc0a2980 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86d1a3e640 10 monclient(hunting): authenticate will time out at 963.483349s 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86ca7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86cc0a2570 0x7f86cc120eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34372/0 (socket says 172.21.15.6:34372) 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86ca7fc640 1 -- 172.21.15.6:0/3829522122 learned_addr learned my addr 172.21.15.6:0/3829522122 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3314289362 0 0) 0x7f86cc0a8460 con 0x7f86cc0a2980 2024-09-16T05:35:01.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4385 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86ac003530 con 0x7f86cc0a2980 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 523213151 0 0) 0x7f86cc124450 con 0x7f86cc0a2570 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4386 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86cc0a8460 con 0x7f86cc0a2570 2024-09-16T05:35:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.671+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1608113668 0 0) 0x7f86ac003530 con 0x7f86cc0a2980 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86cc124450 con 0x7f86cc0a2980 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86c4001b80 con 0x7f86cc0a2980 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): dump: 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1439724743 0 0) 0x7f86cc0a8460 con 0x7f86cc0a2570 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86ac003530 con 0x7f86cc0a2570 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86bc0018e0 con 0x7f86cc0a2570 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): dump: 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728710197 0 0) 0x7f86cc12d7d0 con 0x7f86cc0a2190 2024-09-16T05:35:01.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:01.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:01.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:01.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4396 2024-09-16T05:35:01.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86cc0a8460 con 0x7f86cc0a2190 2024-09-16T05:35:01.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3343212433 0 0) 0x7f86cc124450 con 0x7f86cc0a2980 2024-09-16T05:35:01.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 >> v1:172.21.15.63:6789/0 conn(0x7f86cc0a2190 legacy=0x7f86cc1207a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 >> v1:172.21.15.6:6789/0 conn(0x7f86cc0a2570 legacy=0x7f86cc120eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:35:01.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 monclient: found mon.c 2024-09-16T05:35:01.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:01.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86cc0a0100 con 0x7f86cc0a2980 2024-09-16T05:35:01.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient: _finish_auth 0 2024-09-16T05:35:01.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:35:01.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 5 monclient: authenticate success, global_id 4385 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 10 monclient: _renew_subs 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86cc123e90 con 0x7f86cc0a2980 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 10 monclient: _renew_subs 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f86cc1225c0 con 0x7f86cc0a2980 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86c4003980 con 0x7f86cc0a2980 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86c4003cd0 con 0x7f86cc0a2980 2024-09-16T05:35:01.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c97fa640 4 set_mon_vals no callback set 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 10 monclient: dump: 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:01.699 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86b0005090 con 0x7f86cc0a2980 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f86c4003e70 con 0x7f86cc0a2980 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7f86c4043240 con 0x7f86cc0a2980 2024-09-16T05:35:01.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.675+0000 7f86caffd640 10 monclient: get_auth_request con 0x7f86ac036c40 auth_method 0 2024-09-16T05:35:01.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.679+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f86c4014590 con 0x7f86cc0a2980 2024-09-16T05:35:01.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.679+0000 7f86c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:01.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.679+0000 7f86c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.847+0000 7f86d1a3e640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T05:35:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.847+0000 7f86d1a3e640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T05:35:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.847+0000 7f86d1a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.847+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f86b0005b90 con 0x7f86cc0a2980 2024-09-16T05:35:01.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86c3fff640 1 -- 172.21.15.6:0/3829522122 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (unknown 3471955896 0 2170607528) 0x7f86c4013e90 con 0x7f86cc0a2980 2024-09-16T05:35:01.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T05:35:01.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:01.856 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-09-16T05:35:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 >> v1:172.21.15.63:6812/3643207211 conn(0x7f86ac036c40 legacy=0x7f86ac039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86d1a3e640 10 monclient: shutdown 2024-09-16T05:35:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86d1a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 >> v1:172.21.15.6:6790/0 conn(0x7f86cc0a2980 legacy=0x7f86cc1d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 shutdown_connections 2024-09-16T05:35:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 >> 172.21.15.6:0/3829522122 conn(0x7f86cc0a9850 msgr2=0x7f86cc12fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 shutdown_connections 2024-09-16T05:35:01.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.851+0000 7f86d1a3e640 1 -- 172.21.15.6:0/3829522122 wait complete. 2024-09-16T05:35:01.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T05:35:02.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient: get_monmap_and_config 2024-09-16T05:35:02.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient: build_initial_monmap 2024-09-16T05:35:02.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient: monmap: 2024-09-16T05:35:02.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:02.005349+0000 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:02.005349+0000 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:02.006 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 1 Processor -- start 2024-09-16T05:35:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 1 -- start start 2024-09-16T05:35:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient: init 2024-09-16T05:35:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:35:02.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient(hunting): picked mon.noname-a con 0x7f6b20072300 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient(hunting): picked mon.noname-c con 0x7f6b20072df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient(hunting): picked mon.noname-b con 0x7f6b2013d320 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2009b330 con 0x7f6b20072300 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2009b050 con 0x7f6b20072df0 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2009b610 con 0x7f6b2013d320 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b25d0a640 10 monclient(hunting): authenticate will time out at 963.811872s 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b1ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6b2013d320 0x7f6b2013f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42314/0 (socket says 172.21.15.6:42314) 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:01.999+0000 7f6b1f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6b20072300 0x7f6b200726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34378/0 (socket says 172.21.15.6:34378) 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1ffff640 1 -- 172.21.15.6:0/450476042 learned_addr learned my addr 172.21.15.6:0/450476042 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:02.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331942814 0 0) 0x7f6b2009b610 con 0x7f6b2013d320 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient(hunting): my global_id is 4388 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b0c003530 con 0x7f6b2013d320 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3380526704 0 0) 0x7f6b10001120 con 0x7f6b20072300 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient(hunting): my global_id is 4389 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b2009b610 con 0x7f6b20072300 2024-09-16T05:35:02.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3196817697 0 0) 0x7f6b0c003530 con 0x7f6b2013d320 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 >> v1:172.21.15.63:6789/0 conn(0x7f6b20072df0 legacy=0x7f6b2013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 >> v1:172.21.15.6:6789/0 conn(0x7f6b20072300 legacy=0x7f6b200726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 monclient: found mon.noname-b 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2009c070 con 0x7f6b2013d320 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:35:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6b180019e0 con 0x7f6b2013d320 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 5 monclient: authenticate success, global_id 4388 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: dump: 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.018 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b18001c70 con 0x7f6b2013d320 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 1 -- 172.21.15.6:0/450476042 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6b18002040 con 0x7f6b2013d320 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1e7fc640 10 monclient: dump: 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.019 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 4 set_mon_vals no callback set 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: shutdown 2024-09-16T05:35:02.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- 172.21.15.6:0/450476042 >> v1:172.21.15.6:6790/0 conn(0x7f6b2013d320 legacy=0x7f6b2013f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- 172.21.15.6:0/450476042 shutdown_connections 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- 172.21.15.6:0/450476042 >> 172.21.15.6:0/450476042 conn(0x7f6b200a1650 msgr2=0x7f6b200a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- 172.21.15.6:0/450476042 shutdown_connections 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- 172.21.15.6:0/450476042 wait complete. 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: build_initial_monmap 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: monmap: 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 Processor -- start 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- start start 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: init 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient(hunting): picked mon.noname-a con 0x7f6b20072300 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:02.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient(hunting): picked mon.noname-c con 0x7f6b20072df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient(hunting): picked mon.noname-b con 0x7f6b2013d320 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2011e860 con 0x7f6b20072300 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b20140850 con 0x7f6b2013d320 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2013cec0 con 0x7f6b20072df0 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient(hunting): authenticate will time out at 963.814819s 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6b20072df0 0x7f6b2011b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42326/0 (socket says 172.21.15.6:42326) 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1effd640 1 -- 172.21.15.6:0/1476046377 learned_addr learned my addr 172.21.15.6:0/1476046377 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2463005700 0 0) 0x7f6b2011e860 con 0x7f6b20072300 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): my global_id is 4392 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b00003530 con 0x7f6b20072300 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349174844 0 0) 0x7f6b2013cec0 con 0x7f6b20072df0 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): my global_id is 4391 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b2011e860 con 0x7f6b20072df0 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2406632713 0 0) 0x7f6b00003530 con 0x7f6b20072300 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b2013cec0 con 0x7f6b20072300 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3155359610 0 0) 0x7f6b2011e860 con 0x7f6b20072df0 2024-09-16T05:35:02.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b00003530 con 0x7f6b20072df0 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6b10001ab0 con 0x7f6b20072300 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): dump: 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.025 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6b08001900 con 0x7f6b20072df0 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): dump: 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.026 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1670529297 0 0) 0x7f6b20140850 con 0x7f6b2013d320 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient(hunting): my global_id is 4402 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b2011e860 con 0x7f6b2013d320 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2247585516 0 0) 0x7f6b2013cec0 con 0x7f6b20072300 2024-09-16T05:35:02.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 >> v1:172.21.15.6:6790/0 conn(0x7f6b20072df0 legacy=0x7f6b2011b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 >> v1:172.21.15.63:6789/0 conn(0x7f6b2013d320 legacy=0x7f6b2011bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient: _finish_hunting 0 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 monclient: found mon.a 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2011e580 con 0x7f6b20072300 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient: _finish_auth 0 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient: _check_auth_tickets 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 5 monclient: authenticate success, global_id 4392 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: _renew_subs 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2011d160 con 0x7f6b20072300 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b100047f0 con 0x7f6b20072300 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6b10004b40 con 0x7f6b20072300 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b17fff640 10 monclient: dump: 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.029 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: _renew_subs 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b1dffb640 4 set_mon_vals no callback set 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.003+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b201d4f50 con 0x7f6b20072300 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.007+0000 7f6b25d0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.007+0000 7f6b25d0a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.007+0000 7f6b25d0a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.007+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b201392f0 con 0x7f6b20072300 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.007+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6b1000f8e0 con 0x7f6b20072300 2024-09-16T05:35:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.007+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7f6b10044060 con 0x7f6b20072300 2024-09-16T05:35:02.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.007+0000 7f6b1effd640 10 monclient: get_auth_request con 0x7f6b00036bb0 auth_method 0 2024-09-16T05:35:02.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.011+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6b2009b330 con 0x7f6b20072300 2024-09-16T05:35:02.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.011+0000 7f6b17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.011+0000 7f6b17fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:02.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.167+0000 7f6b25d0a640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T05:35:02.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.167+0000 7f6b25d0a640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T05:35:02.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.167+0000 7f6b25d0a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.167+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f6b2006bc10 con 0x7f6b20072300 2024-09-16T05:35:02.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.167+0000 7f6b17fff640 1 -- 172.21.15.6:0/1476046377 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (unknown 3412311620 0 265718887) 0x7f6b10015310 con 0x7f6b20072300 2024-09-16T05:35:02.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.167+0000 7f6b17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T05:35:02.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.167+0000 7f6b17fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:02.174 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 8 2024-09-16T05:35:02.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.171+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6b00036bb0 legacy=0x7f6b00039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.171+0000 7f6b25d0a640 10 monclient: shutdown 2024-09-16T05:35:02.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.171+0000 7f6b25d0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:02.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.171+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 >> v1:172.21.15.6:6789/0 conn(0x7f6b20072300 legacy=0x7f6b2011b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.171+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 shutdown_connections 2024-09-16T05:35:02.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.171+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 >> 172.21.15.6:0/1476046377 conn(0x7f6b200a1650 msgr2=0x7f6b200a2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:02.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.171+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 shutdown_connections 2024-09-16T05:35:02.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.171+0000 7f6b25d0a640 1 -- 172.21.15.6:0/1476046377 wait complete. 2024-09-16T05:35:02.191 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T05:35:02.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.307+0000 7fc0705c8640 10 monclient: get_monmap_and_config 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.307+0000 7fc0705c8640 10 monclient: build_initial_monmap 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.307+0000 7fc0705c8640 10 monclient: monmap: 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:02.313662+0000 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:02.313662+0000 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:02.314 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:02.315 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:02.315 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 Processor -- start 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- start start 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: init 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc0680a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0680a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc0680a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06806e520 con 0x7fc0680a1a00 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0680a0780 con 0x7fc0680a4290 2024-09-16T05:35:02.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0680a4c70 con 0x7fc0680a37a0 2024-09-16T05:35:02.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:02.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): authenticate will time out at 964.120840s 2024-09-16T05:35:02.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06e364640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc0680a37a0 0x7fc0680a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42334/0 (socket says 172.21.15.6:42334) 2024-09-16T05:35:02.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06e364640 1 -- 172.21.15.6:0/1660649274 learned_addr learned my addr 172.21.15.6:0/1660649274 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:02.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3925388710 0 0) 0x7fc06806e520 con 0x7fc0680a1a00 2024-09-16T05:35:02.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): my global_id is 4395 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc054003530 con 0x7fc0680a1a00 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3046221580 0 0) 0x7fc0680a4c70 con 0x7fc0680a37a0 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): my global_id is 4394 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc06806e520 con 0x7fc0680a37a0 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 758729469 0 0) 0x7fc0680a0780 con 0x7fc0680a4290 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient(hunting): my global_id is 4405 2024-09-16T05:35:02.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0680a4c70 con 0x7fc0680a4290 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3905376355 0 0) 0x7fc054003530 con 0x7fc0680a1a00 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 >> v1:172.21.15.6:6790/0 conn(0x7fc0680a37a0 legacy=0x7fc0680a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 >> v1:172.21.15.63:6789/0 conn(0x7fc0680a4290 legacy=0x7fc06813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: _finish_hunting 0 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 monclient: found mon.noname-a 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0680a65f0 con 0x7fc0680a1a00 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: _finish_auth 0 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: _check_auth_tickets 2024-09-16T05:35:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc060001550 con 0x7fc0680a1a00 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: dump: 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.325 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 5 monclient: authenticate success, global_id 4395 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc0600017b0 con 0x7fc0680a1a00 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 1 -- 172.21.15.6:0/1660649274 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc060001b80 con 0x7fc0680a1a00 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:02.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc06d362640 10 monclient: dump: 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 4 set_mon_vals no callback set 2024-09-16T05:35:02.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: shutdown 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- 172.21.15.6:0/1660649274 >> v1:172.21.15.6:6789/0 conn(0x7fc0680a1a00 legacy=0x7fc06813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- 172.21.15.6:0/1660649274 shutdown_connections 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- 172.21.15.6:0/1660649274 >> 172.21.15.6:0/1660649274 conn(0x7fc0680a58a0 msgr2=0x7fc0680a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- 172.21.15.6:0/1660649274 shutdown_connections 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- 172.21.15.6:0/1660649274 wait complete. 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: build_initial_monmap 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: monmap: 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 Processor -- start 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- start start 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: init 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc0680a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc0680a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0680a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0680a8010 con 0x7fc0680a37a0 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0680a0100 con 0x7fc0680a1a00 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06813eca0 con 0x7fc0680a4290 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.311+0000 7fc0705c8640 10 monclient(hunting): authenticate will time out at 964.123936s 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc06eb65640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc0680a4290 0x7fc0681d3d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42350/0 (socket says 172.21.15.6:42350) 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc06eb65640 1 -- 172.21.15.6:0/1002664515 learned_addr learned my addr 172.21.15.6:0/1002664515 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 169721722 0 0) 0x7fc06813eca0 con 0x7fc0680a4290 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): my global_id is 4397 2024-09-16T05:35:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc04c003530 con 0x7fc0680a4290 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485251232 0 0) 0x7fc0680a8010 con 0x7fc0680a37a0 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): my global_id is 4398 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc06813eca0 con 0x7fc0680a37a0 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3959638908 0 0) 0x7fc04c003530 con 0x7fc0680a4290 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0680a8010 con 0x7fc0680a4290 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc060001510 con 0x7fc0680a4290 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): dump: 2024-09-16T05:35:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1070159530 0 0) 0x7fc06813eca0 con 0x7fc0680a37a0 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc04c003530 con 0x7fc0680a37a0 2024-09-16T05:35:02.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc0580018c0 con 0x7fc0680a37a0 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): dump: 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 379567811 0 0) 0x7fc0680a0100 con 0x7fc0680a1a00 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient(hunting): my global_id is 4408 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc06813eca0 con 0x7fc0680a1a00 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2329911974 0 0) 0x7fc0680a8010 con 0x7fc0680a4290 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 >> v1:172.21.15.63:6789/0 conn(0x7fc0680a1a00 legacy=0x7fc068136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 >> v1:172.21.15.6:6789/0 conn(0x7fc0680a37a0 legacy=0x7fc0681d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient: _finish_hunting 0 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 monclient: found mon.c 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06813d570 con 0x7fc0680a4290 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient: _finish_auth 0 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient: _check_auth_tickets 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 5 monclient: authenticate success, global_id 4397 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 10 monclient: _renew_subs 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc06813da10 con 0x7fc0680a4290 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 10 monclient: _renew_subs 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc0681374c0 con 0x7fc0680a4290 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc0600017b0 con 0x7fc0680a4290 2024-09-16T05:35:02.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc060003af0 con 0x7fc0680a4290 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 10 monclient: dump: 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc06cb61640 4 set_mon_vals no callback set 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc050005090 con 0x7fc0680a4290 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc060003c90 con 0x7fc0680a4290 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7fc060044120 con 0x7fc0680a4290 2024-09-16T05:35:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.315+0000 7fc06e364640 10 monclient: get_auth_request con 0x7fc04c036c70 auth_method 0 2024-09-16T05:35:02.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.319+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc060015470 con 0x7fc0680a4290 2024-09-16T05:35:02.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.319+0000 7fc05effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.319+0000 7fc05effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.475+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc050000ba0 con 0x7fc04c036c70 2024-09-16T05:35:02.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc05effd640 1 -- 172.21.15.6:0/1002664515 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34203 (unknown 2965378022 0 113983366) 0x7fc050000ba0 con 0x7fc04c036c70 2024-09-16T05:35:02.486 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:35:02.486 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:35:02.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc04c036c70 legacy=0x7fc04c039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc0705c8640 10 monclient: shutdown 2024-09-16T05:35:02.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc0705c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:02.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 >> v1:172.21.15.6:6790/0 conn(0x7fc0680a4290 legacy=0x7fc0681d3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 shutdown_connections 2024-09-16T05:35:02.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 >> 172.21.15.6:0/1002664515 conn(0x7fc0680a58a0 msgr2=0x7fc0680a8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:02.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 shutdown_connections 2024-09-16T05:35:02.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.483+0000 7fc0705c8640 1 -- 172.21.15.6:0/1002664515 wait complete. 2024-09-16T05:35:02.503 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T05:35:01.261826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163272,"kb_used_data":1072,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067768,"statfs":{"total":579820584960,"available":579653394432,"internally_reserved":0,"allocated":1097728,"data_stored":1373734,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":165927586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001677"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.104363+0000","last_change":"2024-09-16T05:34:57.104762+0000","last_active":"2024-09-16T05:34:57.104363+0000","last_peered":"2024-09-16T05:34:57.104363+0000","last_clean":"2024-09-16T05:34:57.104363+0000","last_became_active":"2024-09-16T05:34:55.334450+0000","last_became_peered":"2024-09-16T05:34:55.334450+0000","last_unstale":"2024-09-16T05:34:57.104363+0000","last_undegraded":"2024-09-16T05:34:57.104363+0000","last_fullsized":"2024-09-16T05:34:57.104363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:20.555116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.930290+0000","last_change":"2024-09-16T05:34:56.930773+0000","last_active":"2024-09-16T05:34:56.930290+0000","last_peered":"2024-09-16T05:34:56.930290+0000","last_clean":"2024-09-16T05:34:56.930290+0000","last_became_active":"2024-09-16T05:34:54.903232+0000","last_became_peered":"2024-09-16T05:34:54.903232+0000","last_unstale":"2024-09-16T05:34:56.930290+0000","last_undegraded":"2024-09-16T05:34:56.930290+0000","last_fullsized":"2024-09-16T05:34:56.930290+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:18:07.827355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103981+0000","last_change":"2024-09-16T05:34:57.104630+0000","last_active":"2024-09-16T05:34:57.103981+0000","last_peered":"2024-09-16T05:34:57.103981+0000","last_clean":"2024-09-16T05:34:57.103981+0000","last_became_active":"2024-09-16T05:34:55.167715+0000","last_became_peered":"2024-09-16T05:34:55.167715+0000","last_unstale":"2024-09-16T05:34:57.103981+0000","last_undegraded":"2024-09-16T05:34:57.103981+0000","last_fullsized":"2024-09-16T05:34:57.103981+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:17.176705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.930084+0000","last_change":"2024-09-16T05:34:56.930673+0000","last_active":"2024-09-16T05:34:56.930084+0000","last_peered":"2024-09-16T05:34:56.930084+0000","last_clean":"2024-09-16T05:34:56.930084+0000","last_became_active":"2024-09-16T05:34:55.165400+0000","last_became_peered":"2024-09-16T05:34:55.165400+0000","last_unstale":"2024-09-16T05:34:56.930084+0000","last_undegraded":"2024-09-16T05:34:56.930084+0000","last_fullsized":"2024-09-16T05:34:56.930084+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:42.301038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.940160+0000","last_change":"2024-09-16T05:34:56.938719+0000","last_active":"2024-09-16T05:34:56.940160+0000","last_peered":"2024-09-16T05:34:56.940160+0000","last_clean":"2024-09-16T05:34:56.940160+0000","last_became_active":"2024-09-16T05:34:54.914386+0000","last_became_peered":"2024-09-16T05:34:54.914386+0000","last_unstale":"2024-09-16T05:34:56.940160+0000","last_undegraded":"2024-09-16T05:34:56.940160+0000","last_fullsized":"2024-09-16T05:34:56.940160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:18.868561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.928651+0000","last_change":"2024-09-16T05:34:56.929169+0000","last_active":"2024-09-16T05:34:56.928651+0000","last_peered":"2024-09-16T05:34:56.928651+0000","last_clean":"2024-09-16T05:34:56.928651+0000","last_became_active":"2024-09-16T05:34:54.906976+0000","last_became_peered":"2024-09-16T05:34:54.906976+0000","last_unstale":"2024-09-16T05:34:56.928651+0000","last_undegraded":"2024-09-16T05:34:56.928651+0000","last_fullsized":"2024-09-16T05:34:56.928651+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:20.747928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103806+0000","last_change":"2024-09-16T05:34:57.104264+0000","last_active":"2024-09-16T05:34:57.103806+0000","last_peered":"2024-09-16T05:34:57.103806+0000","last_clean":"2024-09-16T05:34:57.103806+0000","last_became_active":"2024-09-16T05:34:54.909066+0000","last_became_peered":"2024-09-16T05:34:54.909066+0000","last_unstale":"2024-09-16T05:34:57.103806+0000","last_undegraded":"2024-09-16T05:34:57.103806+0000","last_fullsized":"2024-09-16T05:34:57.103806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:12.435781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.937943+0000","last_change":"2024-09-16T05:34:56.937943+0000","last_active":"2024-09-16T05:34:56.937943+0000","last_peered":"2024-09-16T05:34:56.937943+0000","last_clean":"2024-09-16T05:34:56.937943+0000","last_became_active":"2024-09-16T05:34:54.914265+0000","last_became_peered":"2024-09-16T05:34:54.914265+0000","last_unstale":"2024-09-16T05:34:56.937943+0000","last_undegraded":"2024-09-16T05:34:56.937943+0000","last_fullsized":"2024-09-16T05:34:56.937943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:32.558807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103169+0000","last_change":"2024-09-16T05:34:49.007631+0000","last_active":"2024-09-16T05:34:57.103169+0000","last_peered":"2024-09-16T05:34:57.103169+0000","last_clean":"2024-09-16T05:34:57.103169+0000","last_became_active":"2024-09-16T05:34:49.004914+0000","last_became_peered":"2024-09-16T05:34:49.004914+0000","last_unstale":"2024-09-16T05:34:57.103169+0000","last_undegraded":"2024-09-16T05:34:57.103169+0000","last_fullsized":"2024-09-16T05:34:57.103169+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_clean_scrub_stamp":"2024-09-16T05:34:47.359645+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:47:04.580831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":217088,"data_stored":581568,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":581568,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:35:02.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:35:02.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.603+0000 7f68ff33e640 10 monclient: get_monmap_and_config 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.603+0000 7f68ff33e640 10 monclient: build_initial_monmap 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.603+0000 7f68ff33e640 10 monclient: monmap: 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:02.610605+0000 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:02.610605+0000 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:02.611 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:02.612 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 Processor -- start 2024-09-16T05:35:02.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- start start 2024-09-16T05:35:02.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient: init 2024-09-16T05:35:02.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:02.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:35:02.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient(hunting): picked mon.noname-a con 0x7f68f8073060 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:02.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient(hunting): picked mon.noname-b con 0x7f68f809edd0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient(hunting): picked mon.noname-c con 0x7f68f813cfe0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f80a35a0 con 0x7f68f8073060 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f806e680 con 0x7f68f813cfe0 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f8073b50 con 0x7f68f809edd0 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient(hunting): authenticate will time out at 964.417300s 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68fc8d9640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f68f809edd0 0x7f68f813b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42364/0 (socket says 172.21.15.6:42364) 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68fc8d9640 1 -- 172.21.15.6:0/721860869 learned_addr learned my addr 172.21.15.6:0/721860869 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 630955239 0 0) 0x7f68f8073b50 con 0x7f68f809edd0 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient(hunting): my global_id is 4400 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68e4003530 con 0x7f68f809edd0 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2816564473 0 0) 0x7f68f80a35a0 con 0x7f68f8073060 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient(hunting): my global_id is 4401 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68f8073b50 con 0x7f68f8073060 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 225034853 0 0) 0x7f68e4003530 con 0x7f68f809edd0 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 >> v1:172.21.15.63:6789/0 conn(0x7f68f813cfe0 legacy=0x7f68f813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 >> v1:172.21.15.6:6789/0 conn(0x7f68f8073060 legacy=0x7f68f8073440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: _finish_hunting 0 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 monclient: found mon.noname-b 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f80a1120 con 0x7f68f809edd0 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: _finish_auth 0 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: _check_auth_tickets 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f68ec001620 con 0x7f68f809edd0 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 5 monclient: authenticate success, global_id 4400 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: dump: 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f68ec001900 con 0x7f68f809edd0 2024-09-16T05:35:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 1 -- 172.21.15.6:0/721860869 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f68ec001d10 con 0x7f68f809edd0 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 4 set_mon_vals no callback set 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ebfff640 10 monclient: dump: 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.623 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient: shutdown 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- 172.21.15.6:0/721860869 >> v1:172.21.15.6:6790/0 conn(0x7f68f809edd0 legacy=0x7f68f813b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- 172.21.15.6:0/721860869 shutdown_connections 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- 172.21.15.6:0/721860869 >> 172.21.15.6:0/721860869 conn(0x7f68f80a37e0 msgr2=0x7f68f80a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- 172.21.15.6:0/721860869 shutdown_connections 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- 172.21.15.6:0/721860869 wait complete. 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient: build_initial_monmap 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient: monmap: 2024-09-16T05:35:02.624 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 Processor -- start 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 1 -- start start 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.607+0000 7f68ff33e640 10 monclient: init 2024-09-16T05:35:02.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient(hunting): picked mon.noname-b con 0x7f68f8073060 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient(hunting): picked mon.noname-c con 0x7f68f809edd0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient(hunting): picked mon.noname-a con 0x7f68f813cfe0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f8123b40 con 0x7f68f813cfe0 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f8122560 con 0x7f68f8073060 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f81407b0 con 0x7f68f809edd0 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient(hunting): authenticate will time out at 964.420629s 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68fd8db640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f68f813cfe0 0x7f68f81d3a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34440/0 (socket says 172.21.15.6:34440) 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68fd8db640 1 -- 172.21.15.6:0/2124980234 learned_addr learned my addr 172.21.15.6:0/2124980234 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2258662154 0 0) 0x7f68f8123b40 con 0x7f68f813cfe0 2024-09-16T05:35:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): my global_id is 4404 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68dc003560 con 0x7f68f813cfe0 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1550881439 0 0) 0x7f68f81407b0 con 0x7f68f809edd0 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): my global_id is 4403 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68f8123b40 con 0x7f68f809edd0 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2821604874 0 0) 0x7f68f8123b40 con 0x7f68f809edd0 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68f81407b0 con 0x7f68f809edd0 2024-09-16T05:35:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f68ec001ab0 con 0x7f68f809edd0 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): dump: 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.628 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3899061579 0 0) 0x7f68dc003560 con 0x7f68f813cfe0 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68f8123b40 con 0x7f68f813cfe0 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1985046151 0 0) 0x7f68f8122560 con 0x7f68f8073060 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): my global_id is 4414 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68dc003560 con 0x7f68f8073060 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f68f0001bd0 con 0x7f68f813cfe0 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient(hunting): dump: 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3460494178 0 0) 0x7f68f8123b40 con 0x7f68f813cfe0 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 >> v1:172.21.15.6:6790/0 conn(0x7f68f809edd0 legacy=0x7f68f8121e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 >> v1:172.21.15.63:6789/0 conn(0x7f68f8073060 legacy=0x7f68f8121740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 monclient: found mon.a 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f809d190 con 0x7f68f813cfe0 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 5 monclient: authenticate success, global_id 4404 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient: _renew_subs 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f68f8123e20 con 0x7f68f813cfe0 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient: _renew_subs 2024-09-16T05:35:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f68f813ca80 con 0x7f68f813cfe0 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f68f00039a0 con 0x7f68f813cfe0 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f68f0003cf0 con 0x7f68f813cfe0 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68eb7fe640 4 set_mon_vals no callback set 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 10 monclient: dump: 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.632 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f68f0003e90 con 0x7f68f813cfe0 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d8005090 con 0x7f68f813cfe0 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7f68f00443d0 con 0x7f68f813cfe0 2024-09-16T05:35:02.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.611+0000 7f68fd0da640 10 monclient: get_auth_request con 0x7f68dc036c00 auth_method 0 2024-09-16T05:35:02.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.615+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f68f0015700 con 0x7f68f813cfe0 2024-09-16T05:35:02.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.615+0000 7f68e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.615+0000 7f68e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:02.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.771+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f68d8000ba0 con 0x7f68dc036c00 2024-09-16T05:35:02.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68e9ffb640 1 -- 172.21.15.6:0/2124980234 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34203 (unknown 2965378022 0 113983366) 0x7f68d8000ba0 con 0x7f68dc036c00 2024-09-16T05:35:02.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:35:02.781 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:35:02.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 >> v1:172.21.15.63:6812/3643207211 conn(0x7f68dc036c00 legacy=0x7f68dc0390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68ff33e640 10 monclient: shutdown 2024-09-16T05:35:02.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68ff33e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:02.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 >> v1:172.21.15.6:6789/0 conn(0x7f68f813cfe0 legacy=0x7f68f81d3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 shutdown_connections 2024-09-16T05:35:02.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 >> 172.21.15.6:0/2124980234 conn(0x7f68f80a37e0 msgr2=0x7f68f809f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:02.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 shutdown_connections 2024-09-16T05:35:02.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.775+0000 7f68ff33e640 1 -- 172.21.15.6:0/2124980234 wait complete. 2024-09-16T05:35:02.798 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T05:35:01.261826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163272,"kb_used_data":1072,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067768,"statfs":{"total":579820584960,"available":579653394432,"internally_reserved":0,"allocated":1097728,"data_stored":1373734,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":165927586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001677"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.104363+0000","last_change":"2024-09-16T05:34:57.104762+0000","last_active":"2024-09-16T05:34:57.104363+0000","last_peered":"2024-09-16T05:34:57.104363+0000","last_clean":"2024-09-16T05:34:57.104363+0000","last_became_active":"2024-09-16T05:34:55.334450+0000","last_became_peered":"2024-09-16T05:34:55.334450+0000","last_unstale":"2024-09-16T05:34:57.104363+0000","last_undegraded":"2024-09-16T05:34:57.104363+0000","last_fullsized":"2024-09-16T05:34:57.104363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:20.555116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.930290+0000","last_change":"2024-09-16T05:34:56.930773+0000","last_active":"2024-09-16T05:34:56.930290+0000","last_peered":"2024-09-16T05:34:56.930290+0000","last_clean":"2024-09-16T05:34:56.930290+0000","last_became_active":"2024-09-16T05:34:54.903232+0000","last_became_peered":"2024-09-16T05:34:54.903232+0000","last_unstale":"2024-09-16T05:34:56.930290+0000","last_undegraded":"2024-09-16T05:34:56.930290+0000","last_fullsized":"2024-09-16T05:34:56.930290+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:18:07.827355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103981+0000","last_change":"2024-09-16T05:34:57.104630+0000","last_active":"2024-09-16T05:34:57.103981+0000","last_peered":"2024-09-16T05:34:57.103981+0000","last_clean":"2024-09-16T05:34:57.103981+0000","last_became_active":"2024-09-16T05:34:55.167715+0000","last_became_peered":"2024-09-16T05:34:55.167715+0000","last_unstale":"2024-09-16T05:34:57.103981+0000","last_undegraded":"2024-09-16T05:34:57.103981+0000","last_fullsized":"2024-09-16T05:34:57.103981+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:17.176705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.930084+0000","last_change":"2024-09-16T05:34:56.930673+0000","last_active":"2024-09-16T05:34:56.930084+0000","last_peered":"2024-09-16T05:34:56.930084+0000","last_clean":"2024-09-16T05:34:56.930084+0000","last_became_active":"2024-09-16T05:34:55.165400+0000","last_became_peered":"2024-09-16T05:34:55.165400+0000","last_unstale":"2024-09-16T05:34:56.930084+0000","last_undegraded":"2024-09-16T05:34:56.930084+0000","last_fullsized":"2024-09-16T05:34:56.930084+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:42.301038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.940160+0000","last_change":"2024-09-16T05:34:56.938719+0000","last_active":"2024-09-16T05:34:56.940160+0000","last_peered":"2024-09-16T05:34:56.940160+0000","last_clean":"2024-09-16T05:34:56.940160+0000","last_became_active":"2024-09-16T05:34:54.914386+0000","last_became_peered":"2024-09-16T05:34:54.914386+0000","last_unstale":"2024-09-16T05:34:56.940160+0000","last_undegraded":"2024-09-16T05:34:56.940160+0000","last_fullsized":"2024-09-16T05:34:56.940160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:18.868561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.928651+0000","last_change":"2024-09-16T05:34:56.929169+0000","last_active":"2024-09-16T05:34:56.928651+0000","last_peered":"2024-09-16T05:34:56.928651+0000","last_clean":"2024-09-16T05:34:56.928651+0000","last_became_active":"2024-09-16T05:34:54.906976+0000","last_became_peered":"2024-09-16T05:34:54.906976+0000","last_unstale":"2024-09-16T05:34:56.928651+0000","last_undegraded":"2024-09-16T05:34:56.928651+0000","last_fullsized":"2024-09-16T05:34:56.928651+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:20.747928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103806+0000","last_change":"2024-09-16T05:34:57.104264+0000","last_active":"2024-09-16T05:34:57.103806+0000","last_peered":"2024-09-16T05:34:57.103806+0000","last_clean":"2024-09-16T05:34:57.103806+0000","last_became_active":"2024-09-16T05:34:54.909066+0000","last_became_peered":"2024-09-16T05:34:54.909066+0000","last_unstale":"2024-09-16T05:34:57.103806+0000","last_undegraded":"2024-09-16T05:34:57.103806+0000","last_fullsized":"2024-09-16T05:34:57.103806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:12.435781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:56.937943+0000","last_change":"2024-09-16T05:34:56.937943+0000","last_active":"2024-09-16T05:34:56.937943+0000","last_peered":"2024-09-16T05:34:56.937943+0000","last_clean":"2024-09-16T05:34:56.937943+0000","last_became_active":"2024-09-16T05:34:54.914265+0000","last_became_peered":"2024-09-16T05:34:54.914265+0000","last_unstale":"2024-09-16T05:34:56.937943+0000","last_undegraded":"2024-09-16T05:34:56.937943+0000","last_fullsized":"2024-09-16T05:34:56.937943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:32.558807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T05:34:57.103169+0000","last_change":"2024-09-16T05:34:49.007631+0000","last_active":"2024-09-16T05:34:57.103169+0000","last_peered":"2024-09-16T05:34:57.103169+0000","last_clean":"2024-09-16T05:34:57.103169+0000","last_became_active":"2024-09-16T05:34:49.004914+0000","last_became_peered":"2024-09-16T05:34:49.004914+0000","last_unstale":"2024-09-16T05:34:57.103169+0000","last_undegraded":"2024-09-16T05:34:57.103169+0000","last_fullsized":"2024-09-16T05:34:57.103169+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_clean_scrub_stamp":"2024-09-16T05:34:47.359645+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:47:04.580831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":217088,"data_stored":581568,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":581568,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:35:02.798 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T05:35:02.799 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T05:35:02.799 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T05:35:02.813 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T05:35:02.813 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T05:35:02.814 DEBUG:teuthology.orchestra.run.smithi063:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T05:35:02.814 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T05:35:02.814 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726464902, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T05:35:02.814 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:35:02.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:35:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:35:02.821 INFO:teuthology.orchestra.run.smithi063.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T05:35:02.821 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T05:35:02.867 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T05:35:02.867 DEBUG:teuthology.orchestra.run.smithi063:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T05:35:02.916 INFO:tasks.workunit:timeout=3h 2024-09-16T05:35:02.916 INFO:tasks.workunit:cleanup=True 2024-09-16T05:35:02.917 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T05:35:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.927+0000 7fede5528640 10 monclient: get_monmap_and_config 2024-09-16T05:35:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.927+0000 7fede5528640 10 monclient: build_initial_monmap 2024-09-16T05:35:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.927+0000 7fede5528640 10 monclient: monmap: 2024-09-16T05:35:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:02.932942+0000 2024-09-16T05:35:02.933 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:02.932942+0000 2024-09-16T05:35:02.934 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:02.934 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.934 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:02.934 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:02.934 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:02.934 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.927+0000 7fede5528640 1 Processor -- start 2024-09-16T05:35:02.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.927+0000 7fede5528640 1 -- start start 2024-09-16T05:35:02.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.927+0000 7fede5528640 10 monclient: init 2024-09-16T05:35:02.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): picked mon.noname-a con 0x7fede00a0df0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): picked mon.noname-c con 0x7fede0135330 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): picked mon.noname-b con 0x7fede0138e40 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede009d2f0 con 0x7fede00a0df0 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede013d960 con 0x7fede0135330 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede013eb40 con 0x7fede0138e40 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): authenticate will time out at 964.740435s 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7feddeffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fede00a0df0 0x7fede009de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34450/0 (socket says 172.21.15.6:34450) 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7feddf7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fede0138e40 0x7fede013b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42390/0 (socket says 172.21.15.6:42390) 2024-09-16T05:35:02.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7feddeffd640 1 -- 172.21.15.6:0/1121529398 learned_addr learned my addr 172.21.15.6:0/1121529398 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678672426 0 0) 0x7fede013eb40 con 0x7fede0138e40 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient(hunting): my global_id is 4406 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedc8003530 con 0x7fede0138e40 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1145115198 0 0) 0x7fede009d2f0 con 0x7fede00a0df0 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient(hunting): my global_id is 4407 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fede013eb40 con 0x7fede00a0df0 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2283459694 0 0) 0x7fedc8003530 con 0x7fede0138e40 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 >> v1:172.21.15.63:6789/0 conn(0x7fede0135330 legacy=0x7fede0137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 >> v1:172.21.15.6:6789/0 conn(0x7fede00a0df0 legacy=0x7fede009de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: _finish_hunting 0 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 monclient: found mon.noname-b 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede013fd20 con 0x7fede0138e40 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: _finish_auth 0 2024-09-16T05:35:02.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: _check_auth_tickets 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fedd80019a0 con 0x7fede0138e40 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: dump: 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.941 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 5 monclient: authenticate success, global_id 4406 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedd8001c00 con 0x7fede0138e40 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 1 -- 172.21.15.6:0/1121529398 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fedd8001f50 con 0x7fede0138e40 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:02.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedddffb640 10 monclient: dump: 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:02.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 4 set_mon_vals no callback set 2024-09-16T05:35:02.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: shutdown 2024-09-16T05:35:02.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:02.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- 172.21.15.6:0/1121529398 >> v1:172.21.15.6:6790/0 conn(0x7fede0138e40 legacy=0x7fede013b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- 172.21.15.6:0/1121529398 shutdown_connections 2024-09-16T05:35:02.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- 172.21.15.6:0/1121529398 >> 172.21.15.6:0/1121529398 conn(0x7fede00a1650 msgr2=0x7fede00a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:02.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- 172.21.15.6:0/1121529398 shutdown_connections 2024-09-16T05:35:02.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- 172.21.15.6:0/1121529398 wait complete. 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: build_initial_monmap 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: monmap: 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:02.945 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:02.946 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:02.946 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 Processor -- start 2024-09-16T05:35:02.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- start start 2024-09-16T05:35:02.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: init 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): picked mon.noname-c con 0x7fede00a0df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): picked mon.noname-b con 0x7fede0135330 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): picked mon.noname-a con 0x7fede0138e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede01389f0 con 0x7fede0138e40 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede006ed10 con 0x7fede0135330 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede0070d70 con 0x7fede00a0df0 2024-09-16T05:35:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7feddeffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fede00a0df0 0x7fede006def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42404/0 (socket says 172.21.15.6:42404) 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fede5528640 10 monclient(hunting): authenticate will time out at 964.743535s 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7feddeffd640 1 -- 172.21.15.6:0/3421713960 learned_addr learned my addr 172.21.15.6:0/3421713960 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372055728 0 0) 0x7fede0070d70 con 0x7fede00a0df0 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 10 monclient(hunting): my global_id is 4409 2024-09-16T05:35:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedcc003560 con 0x7fede00a0df0 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 915754678 0 0) 0x7fede013fd20 con 0x7fede0138e40 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 10 monclient(hunting): my global_id is 4410 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.931+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fede0070d70 con 0x7fede0138e40 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2554498733 0 0) 0x7fedcc003560 con 0x7fede00a0df0 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fede013fd20 con 0x7fede00a0df0 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fedd00018a0 con 0x7fede00a0df0 2024-09-16T05:35:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): dump: 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3929192232 0 0) 0x7fede0070d70 con 0x7fede0138e40 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedcc003560 con 0x7fede0138e40 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fedd80018a0 con 0x7fede0138e40 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): dump: 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.954 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4291496067 0 0) 0x7fede006ed10 con 0x7fede0135330 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient(hunting): my global_id is 4420 2024-09-16T05:35:02.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fede0070d70 con 0x7fede0135330 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3164636713 0 0) 0x7fede013fd20 con 0x7fede00a0df0 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 >> v1:172.21.15.63:6789/0 conn(0x7fede0135330 legacy=0x7fede006e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 >> v1:172.21.15.6:6789/0 conn(0x7fede0138e40 legacy=0x7fede01cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 monclient: found mon.c 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede00707b0 con 0x7fede00a0df0 2024-09-16T05:35:02.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient: _finish_auth 0 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 5 monclient: authenticate success, global_id 4409 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 10 monclient: _renew_subs 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fede006f360 con 0x7fede00a0df0 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 10 monclient: _renew_subs 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fede00727f0 con 0x7fede00a0df0 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedd00047f0 con 0x7fede00a0df0 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fedd0004b40 con 0x7fede00a0df0 2024-09-16T05:35:02.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 10 monclient: dump: 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:02.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7feddd7fa640 4 set_mon_vals no callback set 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc0005090 con 0x7fede00a0df0 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fedd000f960 con 0x7fede00a0df0 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7fedd0044210 con 0x7fede00a0df0 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.935+0000 7fedde7fc640 10 monclient: get_auth_request con 0x7fedcc036c80 auth_method 0 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.939+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fedd0015500 con 0x7fede00a0df0 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.939+0000 7fedc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:02.939+0000 7fedc77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:02.967 INFO:tasks.workunit.client.0.smithi063.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T05:35:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.163+0000 7fede5528640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:35:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.163+0000 7fede5528640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:35:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.163+0000 7fede5528640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.163+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fedc0005380 con 0x7fede00a0df0 2024-09-16T05:35:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.163+0000 7fedc77fe640 1 -- 172.21.15.6:0/3421713960 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2335401564) 0x7fedd0019150 con 0x7fede00a0df0 2024-09-16T05:35:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.163+0000 7fedc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:35:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.163+0000 7fedc77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:03.170 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:35:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.167+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 >> v1:172.21.15.63:6812/3643207211 conn(0x7fedcc036c80 legacy=0x7fedcc039140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.167+0000 7fede5528640 10 monclient: shutdown 2024-09-16T05:35:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.167+0000 7fede5528640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.167+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 >> v1:172.21.15.6:6790/0 conn(0x7fede00a0df0 legacy=0x7fede006def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.167+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 shutdown_connections 2024-09-16T05:35:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.167+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 >> 172.21.15.6:0/3421713960 conn(0x7fede00a1650 msgr2=0x7fede009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.167+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 shutdown_connections 2024-09-16T05:35:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.167+0000 7fede5528640 1 -- 172.21.15.6:0/3421713960 wait complete. 2024-09-16T05:35:03.188 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:35:03.188 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:35:03.188 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:35:03.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:35:03.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient: get_monmap_and_config 2024-09-16T05:35:03.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient: build_initial_monmap 2024-09-16T05:35:03.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient: monmap: 2024-09-16T05:35:03.313 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:03.313 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.313 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:03.312384+0000 2024-09-16T05:35:03.314 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:03.312384+0000 2024-09-16T05:35:03.314 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:03.314 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.314 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:03.314 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:03.314 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:03.315 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 1 Processor -- start 2024-09-16T05:35:03.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 1 -- start start 2024-09-16T05:35:03.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient: init 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient(hunting): picked mon.noname-a con 0x7faa3009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient(hunting): picked mon.noname-b con 0x7faa3009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient(hunting): picked mon.noname-c con 0x7faa300a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa300a1370 con 0x7faa3009aa30 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa3009ead0 con 0x7faa300a0da0 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa3009edb0 con 0x7faa3009e230 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 10 monclient(hunting): authenticate will time out at 965.119024s 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa360b6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa3009aa30 0x7faa3009ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34466/0 (socket says 172.21.15.6:34466) 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa360b6640 1 -- 172.21.15.6:0/4070860262 learned_addr learned my addr 172.21.15.6:0/4070860262 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477393224 0 0) 0x7faa300a1370 con 0x7faa3009aa30 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient(hunting): my global_id is 4413 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa1c003530 con 0x7faa3009aa30 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4256739999 0 0) 0x7faa3009edb0 con 0x7faa3009e230 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient(hunting): my global_id is 4412 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa300a1370 con 0x7faa3009e230 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 770474165 0 0) 0x7faa1c003530 con 0x7faa3009aa30 2024-09-16T05:35:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 >> v1:172.21.15.6:6790/0 conn(0x7faa3009e230 legacy=0x7faa301399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 >> v1:172.21.15.63:6789/0 conn(0x7faa300a0da0 legacy=0x7faa3013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient: _finish_hunting 0 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 monclient: found mon.noname-a 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3009dd50 con 0x7faa3009aa30 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient: _finish_auth 0 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient: _check_auth_tickets 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 5 monclient: authenticate success, global_id 4413 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa2c001620 con 0x7faa3009aa30 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa350b4640 10 monclient: dump: 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.307+0000 7faa3831a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa2c001880 con 0x7faa3009aa30 2024-09-16T05:35:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa350b4640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa350b4640 1 -- 172.21.15.6:0/4070860262 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa2c001bd0 con 0x7faa3009aa30 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 4 set_mon_vals no callback set 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa350b4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa350b4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa350b4640 10 monclient: dump: 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: shutdown 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- 172.21.15.6:0/4070860262 >> v1:172.21.15.6:6789/0 conn(0x7faa3009aa30 legacy=0x7faa3009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- 172.21.15.6:0/4070860262 shutdown_connections 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- 172.21.15.6:0/4070860262 >> 172.21.15.6:0/4070860262 conn(0x7faa300a1650 msgr2=0x7faa300a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- 172.21.15.6:0/4070860262 shutdown_connections 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- 172.21.15.6:0/4070860262 wait complete. 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: build_initial_monmap 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: monmap: 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 Processor -- start 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- start start 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: init 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient(hunting): picked mon.noname-a con 0x7faa3009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient(hunting): picked mon.noname-c con 0x7faa3009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient(hunting): picked mon.noname-b con 0x7faa300a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa3013c950 con 0x7faa3009aa30 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa3009c990 con 0x7faa300a0da0 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa3013b0c0 con 0x7faa3009e230 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient(hunting): authenticate will time out at 965.122105s 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa360b6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa3009aa30 0x7faa3009bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34480/0 (socket says 172.21.15.6:34480) 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa360b6640 1 -- 172.21.15.6:0/3939826770 learned_addr learned my addr 172.21.15.6:0/3939826770 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2570212334 0 0) 0x7faa3013b0c0 con 0x7faa3009e230 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): my global_id is 4415 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa14003530 con 0x7faa3009e230 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961671327 0 0) 0x7faa3013c950 con 0x7faa3009aa30 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): my global_id is 4416 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa3013b0c0 con 0x7faa3009aa30 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3922514321 0 0) 0x7faa14003530 con 0x7faa3009e230 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa3013c950 con 0x7faa3009e230 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3858767744 0 0) 0x7faa3013b0c0 con 0x7faa3009aa30 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa14003530 con 0x7faa3009aa30 2024-09-16T05:35:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa200018c0 con 0x7faa3009e230 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): dump: 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.333 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa2c001520 con 0x7faa3009aa30 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): dump: 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914236945 0 0) 0x7faa3009c990 con 0x7faa300a0da0 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient(hunting): my global_id is 4426 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa3013b0c0 con 0x7faa300a0da0 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 431337238 0 0) 0x7faa3013c950 con 0x7faa3009e230 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 >> v1:172.21.15.63:6789/0 conn(0x7faa300a0da0 legacy=0x7faa301d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 >> v1:172.21.15.6:6789/0 conn(0x7faa3009aa30 legacy=0x7faa3009bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 monclient: found mon.c 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3013cc30 con 0x7faa3009e230 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient: _finish_auth 0 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 5 monclient: authenticate success, global_id 4415 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: _renew_subs 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa3013c670 con 0x7faa3009e230 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: _renew_subs 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7faa301d5050 con 0x7faa3009e230 2024-09-16T05:35:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa20003620 con 0x7faa3009e230 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa20003970 con 0x7faa3009e230 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa348b3640 4 set_mon_vals no callback set 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 10 monclient: dump: 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faa180065d0 con 0x7faa18002870 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.311+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7faa2000e710 con 0x7faa3009e230 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7faa20042ed0 con 0x7faa3009e230 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 429775796 0 0) 0x7faa180065d0 con 0x7faa18002870 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient(hunting): my global_id is 4419 2024-09-16T05:35:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa3013c950 con 0x7faa18002870 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa368b7640 10 monclient: get_auth_request con 0x7faa14036cc0 auth_method 0 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2339914179 0 0) 0x7faa3013c950 con 0x7faa18002870 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa180065d0 con 0x7faa18002870 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa2c004aa0 con 0x7faa18002870 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient: dump: 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3511668838 0 0) 0x7faa180065d0 con 0x7faa18002870 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa1800a160 con 0x7faa18002870 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7faa2c004d20 con 0x7faa18002870 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 >> v1:172.21.15.6:6789/0 conn(0x7faa18002870 legacy=0x7faa18004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.315+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 >> v1:172.21.15.6:6789/0 conn(0x7faa18002870 legacy=0x7faa18004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa3831a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faa18009e50 con 0x7faa18009110 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3782859982 0 0) 0x7faa18009e50 con 0x7faa18009110 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 10 monclient(hunting): my global_id is 4422 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa180065d0 con 0x7faa18009110 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1800353324 0 0) 0x7faa180065d0 con 0x7faa18009110 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa18009e50 con 0x7faa18009110 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.319+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa2c005370 con 0x7faa18009110 2024-09-16T05:35:03.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 10 monclient: dump: 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 352479439 0 0) 0x7faa18009e50 con 0x7faa18009110 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa1800c3e0 con 0x7faa18009110 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4130 (unknown 0 0 347899015) 0x7faa2c0066c0 con 0x7faa18009110 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 >> v1:172.21.15.6:6789/0 conn(0x7faa18009110 legacy=0x7faa180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa26ffd640 1 -- 172.21.15.6:0/3939826770 >> v1:172.21.15.6:6789/0 conn(0x7faa18009110 legacy=0x7faa180095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:35:03.344 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 8, 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 32379, 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 27, 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:35:03.345 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:35:03.346 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.347 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:35:03.348 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.349 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:35:03.350 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:35:03.351 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.352 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.354 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:35:03.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.356 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.356 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.356 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-09-16T05:35:03.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.356 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.356 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:35:03.356 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:35:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 >> v1:172.21.15.63:6812/3643207211 conn(0x7faa14036cc0 legacy=0x7faa14039180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa3831a640 10 monclient: shutdown 2024-09-16T05:35:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa3831a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 >> v1:172.21.15.6:6790/0 conn(0x7faa3009e230 legacy=0x7faa3009c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa368b7640 1 -- 172.21.15.6:0/3939826770 reap_dead start 2024-09-16T05:35:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa368b7640 1 -- 172.21.15.6:0/3939826770 reap_dead start 2024-09-16T05:35:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 shutdown_connections 2024-09-16T05:35:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 >> 172.21.15.6:0/3939826770 conn(0x7faa300a1650 msgr2=0x7faa301359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 shutdown_connections 2024-09-16T05:35:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.323+0000 7faa3831a640 1 -- 172.21.15.6:0/3939826770 wait complete. 2024-09-16T05:35:03.359 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:35:03.359 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:35:03.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient: get_monmap_and_config 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient: build_initial_monmap 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient: monmap: 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:03.483781+0000 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:03.483781+0000 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:03.484 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:03.485 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 1 Processor -- start 2024-09-16T05:35:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 1 -- start start 2024-09-16T05:35:03.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient: init 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient(hunting): picked mon.noname-c con 0x7ff5780a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient(hunting): picked mon.noname-a con 0x7ff5780a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient(hunting): picked mon.noname-b con 0x7ff5780a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5780a33a0 con 0x7ff5780a2570 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5780a07a0 con 0x7ff5780a2190 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5780a2d90 con 0x7ff5780a2980 2024-09-16T05:35:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57e960640 10 monclient(hunting): authenticate will time out at 965.291219s 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57cefd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff5780a2980 0x7ff57813f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42440/0 (socket says 172.21.15.6:42440) 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff57cefd640 1 -- 172.21.15.6:0/3133336531 learned_addr learned my addr 172.21.15.6:0/3133336531 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff5777fe640 1 --1- 172.21.15.6:0/3133336531 >> v1:172.21.15.6:6789/0 conn(0x7ff5780a2570 0x7ff57813b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34514/0 (socket says 172.21.15.6:34514) 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199003129 0 0) 0x7ff5780a2d90 con 0x7ff5780a2980 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient(hunting): my global_id is 4418 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff564003530 con 0x7ff5780a2980 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831828658 0 0) 0x7ff5780a33a0 con 0x7ff5780a2570 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient(hunting): my global_id is 4425 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5780a2d90 con 0x7ff5780a2570 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1679852139 0 0) 0x7ff564003530 con 0x7ff5780a2980 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 >> v1:172.21.15.63:6789/0 conn(0x7ff5780a2190 legacy=0x7ff5780a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 >> v1:172.21.15.6:6789/0 conn(0x7ff5780a2570 legacy=0x7ff57813b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 monclient: found mon.noname-b 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5780a66e0 con 0x7ff5780a2980 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient: _finish_auth 0 2024-09-16T05:35:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.479+0000 7ff576ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff570001620 con 0x7ff5780a2980 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 10 monclient: dump: 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 5 monclient: authenticate success, global_id 4418 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5700018c0 con 0x7ff5780a2980 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 1 -- 172.21.15.6:0/3133336531 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff570001cd0 con 0x7ff5780a2980 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 4 set_mon_vals no callback set 2024-09-16T05:35:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff576ffd640 10 monclient: dump: 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: shutdown 2024-09-16T05:35:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- 172.21.15.6:0/3133336531 >> v1:172.21.15.6:6790/0 conn(0x7ff5780a2980 legacy=0x7ff57813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- 172.21.15.6:0/3133336531 shutdown_connections 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- 172.21.15.6:0/3133336531 >> 172.21.15.6:0/3133336531 conn(0x7ff5780a9850 msgr2=0x7ff5780a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- 172.21.15.6:0/3133336531 shutdown_connections 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- 172.21.15.6:0/3133336531 wait complete. 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: build_initial_monmap 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: monmap: 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 Processor -- start 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- start start 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: init 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient(hunting): picked mon.noname-c con 0x7ff5780a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient(hunting): picked mon.noname-a con 0x7ff5780a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient(hunting): picked mon.noname-b con 0x7ff5780a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff578134a20 con 0x7ff5780a2570 2024-09-16T05:35:03.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5780a8320 con 0x7ff5780a2980 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5780a0100 con 0x7ff5780a2190 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient(hunting): authenticate will time out at 965.294372s 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff577fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff5780a2190 0x7ff578133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42450/0 (socket says 172.21.15.6:42450) 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff577fff640 1 -- 172.21.15.6:0/1016292791 learned_addr learned my addr 172.21.15.6:0/1016292791 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff5777fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff5780a2570 0x7ff578134310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34526/0 (socket says 172.21.15.6:34526) 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 763408927 0 0) 0x7ff5780a0100 con 0x7ff5780a2190 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): my global_id is 4421 2024-09-16T05:35:03.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff558003530 con 0x7ff5780a2190 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776126707 0 0) 0x7ff578134a20 con 0x7ff5780a2570 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): my global_id is 4428 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5780a0100 con 0x7ff5780a2570 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 735696322 0 0) 0x7ff558003530 con 0x7ff5780a2190 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff578134a20 con 0x7ff5780a2190 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff568001ca0 con 0x7ff5780a2190 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): dump: 2024-09-16T05:35:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2626028478 0 0) 0x7ff5780a0100 con 0x7ff5780a2570 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff558003530 con 0x7ff5780a2570 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2725656978 0 0) 0x7ff5780a8320 con 0x7ff5780a2980 2024-09-16T05:35:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): my global_id is 4432 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5780a0100 con 0x7ff5780a2980 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff560001830 con 0x7ff5780a2570 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient(hunting): dump: 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 926412598 0 0) 0x7ff578134a20 con 0x7ff5780a2190 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 >> v1:172.21.15.63:6789/0 conn(0x7ff5780a2980 legacy=0x7ff5781d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 >> v1:172.21.15.6:6789/0 conn(0x7ff5780a2570 legacy=0x7ff578134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 monclient: found mon.c 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5781358a0 con 0x7ff5780a2190 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient: _finish_auth 0 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 5 monclient: authenticate success, global_id 4421 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: _renew_subs 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff57813f7d0 con 0x7ff5780a2190 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: _renew_subs 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff578136180 con 0x7ff5780a2190 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5680039a0 con 0x7ff5780a2190 2024-09-16T05:35:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff568003cf0 con 0x7ff5780a2190 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff5767fc640 4 set_mon_vals no callback set 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.483+0000 7ff574ff9640 10 monclient: dump: 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff57e960640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff55c0065d0 con 0x7ff55c002870 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff568003f10 con 0x7ff5780a2190 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7ff568044530 con 0x7ff5780a2190 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201463702 0 0) 0x7ff55c0065d0 con 0x7ff55c002870 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient(hunting): my global_id is 4424 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff578134a20 con 0x7ff55c002870 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2613388909 0 0) 0x7ff578134a20 con 0x7ff55c002870 2024-09-16T05:35:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff55c0065d0 con 0x7ff55c002870 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff5600047f0 con 0x7ff55c002870 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient: dump: 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.513 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff57cefd640 10 monclient: get_auth_request con 0x7ff558036bf0 auth_method 0 2024-09-16T05:35:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2106527533 0 0) 0x7ff55c0065d0 con 0x7ff55c002870 2024-09-16T05:35:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff55c00a160 con 0x7ff55c002870 2024-09-16T05:35:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff560006af0 con 0x7ff55c002870 2024-09-16T05:35:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 >> v1:172.21.15.6:6790/0 conn(0x7ff55c002870 legacy=0x7ff55c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.487+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 >> v1:172.21.15.6:6790/0 conn(0x7ff55c002870 legacy=0x7ff55c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff57e960640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:35:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff55c009e30 con 0x7ff55c0090f0 2024-09-16T05:35:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1584329843 0 0) 0x7ff55c009e30 con 0x7ff55c0090f0 2024-09-16T05:35:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 10 monclient(hunting): my global_id is 4427 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff55c0065d0 con 0x7ff55c0090f0 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 703158054 0 0) 0x7ff55c0065d0 con 0x7ff55c0090f0 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff55c009e30 con 0x7ff55c0090f0 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff5600060a0 con 0x7ff55c0090f0 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 10 monclient: dump: 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3286822260 0 0) 0x7ff55c009e30 con 0x7ff55c0090f0 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.491+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff55c00c3a0 con 0x7ff55c0090f0 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4128 (unknown 0 0 652702559) 0x7ff560005fd0 con 0x7ff55c0090f0 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff574ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:35:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff574ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 >> v1:172.21.15.6:6790/0 conn(0x7ff55c0090f0 legacy=0x7ff55c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff574ff9640 1 -- 172.21.15.6:0/1016292791 >> v1:172.21.15.6:6790/0 conn(0x7ff55c0090f0 legacy=0x7ff55c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 8, 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 32551, 2024-09-16T05:35:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 27, 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:35:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:35:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:35:03.530 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:35:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff558036bf0 legacy=0x7ff5580390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57e960640 10 monclient: shutdown 2024-09-16T05:35:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57e960640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 >> v1:172.21.15.6:6790/0 conn(0x7ff5780a2190 legacy=0x7ff578133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57cefd640 1 -- 172.21.15.6:0/1016292791 reap_dead start 2024-09-16T05:35:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57cefd640 1 -- 172.21.15.6:0/1016292791 reap_dead start 2024-09-16T05:35:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 shutdown_connections 2024-09-16T05:35:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 >> 172.21.15.6:0/1016292791 conn(0x7ff5780a9850 msgr2=0x7ff57812faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 shutdown_connections 2024-09-16T05:35:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.495+0000 7ff57e960640 1 -- 172.21.15.6:0/1016292791 wait complete. 2024-09-16T05:35:03.532 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:35:03.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:35:03.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:35:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.651+0000 7fa3f80b5640 10 monclient: get_monmap_and_config 2024-09-16T05:35:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.651+0000 7fa3f80b5640 10 monclient: build_initial_monmap 2024-09-16T05:35:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.651+0000 7fa3f80b5640 10 monclient: monmap: 2024-09-16T05:35:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:03.658891+0000 2024-09-16T05:35:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:03.658891+0000 2024-09-16T05:35:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:03.660 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 Processor -- start 2024-09-16T05:35:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- start start 2024-09-16T05:35:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient: init 2024-09-16T05:35:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f00a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f00a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f00a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f006e520 con 0x7fa3f00a4290 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f00a0780 con 0x7fa3f00a37a0 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f00a4c70 con 0x7fa3f00a1a00 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient(hunting): authenticate will time out at 965.465811s 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f5650640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3f00a4290 0x7fa3f013bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34532/0 (socket says 172.21.15.6:34532) 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f5650640 1 -- 172.21.15.6:0/1567026910 learned_addr learned my addr 172.21.15.6:0/1567026910 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699290861 0 0) 0x7fa3f00a4c70 con 0x7fa3f00a1a00 2024-09-16T05:35:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient(hunting): my global_id is 4430 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3dc003530 con 0x7fa3f00a1a00 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886470187 0 0) 0x7fa3f006e520 con 0x7fa3f00a4290 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient(hunting): my global_id is 4431 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3f00a4c70 con 0x7fa3f00a4290 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1610687247 0 0) 0x7fa3f00a4c70 con 0x7fa3f00a4290 2024-09-16T05:35:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 >> v1:172.21.15.6:6790/0 conn(0x7fa3f00a1a00 legacy=0x7fa3f013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 >> v1:172.21.15.63:6789/0 conn(0x7fa3f00a37a0 legacy=0x7fa3f00a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: _finish_hunting 0 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 monclient: found mon.noname-a 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f00a65f0 con 0x7fa3f00a4290 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: _finish_auth 0 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: _check_auth_tickets 2024-09-16T05:35:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 5 monclient: authenticate success, global_id 4431 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3e0001600 con 0x7fa3f00a4290 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: dump: 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3e00018e0 con 0x7fa3f00a4290 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 1 -- 172.21.15.6:0/1567026910 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3e0001cf0 con 0x7fa3f00a4290 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 4 set_mon_vals no callback set 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f4e4f640 10 monclient: dump: 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.669 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 10 monclient: shutdown 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- 172.21.15.6:0/1567026910 >> v1:172.21.15.6:6789/0 conn(0x7fa3f00a4290 legacy=0x7fa3f013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- 172.21.15.6:0/1567026910 shutdown_connections 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- 172.21.15.6:0/1567026910 >> 172.21.15.6:0/1567026910 conn(0x7fa3f00a58a0 msgr2=0x7fa3f00a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- 172.21.15.6:0/1567026910 shutdown_connections 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.655+0000 7fa3f80b5640 1 -- 172.21.15.6:0/1567026910 wait complete. 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: build_initial_monmap 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: monmap: 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 1 Processor -- start 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 1 -- start start 2024-09-16T05:35:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: init 2024-09-16T05:35:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:35:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f00a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f00a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:03.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f00a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f013ed10 con 0x7fa3f00a1a00 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f013eff0 con 0x7fa3f00a37a0 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f013be80 con 0x7fa3f00a4290 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient(hunting): authenticate will time out at 965.469045s 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f6652640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa3f00a4290 0x7fa3f01d3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42496/0 (socket says 172.21.15.6:42496) 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f6652640 1 -- 172.21.15.6:0/4143784629 learned_addr learned my addr 172.21.15.6:0/4143784629 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f5e51640 1 --1- 172.21.15.6:0/4143784629 >> v1:172.21.15.6:6789/0 conn(0x7fa3f00a1a00 0x7fa3f013b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34536/0 (socket says 172.21.15.6:34536) 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3607296645 0 0) 0x7fa3f013ed10 con 0x7fa3f00a1a00 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): my global_id is 4434 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3d4003530 con 0x7fa3f00a1a00 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951163598 0 0) 0x7fa3f013be80 con 0x7fa3f00a4290 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): my global_id is 4433 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3f013ed10 con 0x7fa3f00a4290 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1257028747 0 0) 0x7fa3d4003530 con 0x7fa3f00a1a00 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3f013be80 con 0x7fa3f00a1a00 2024-09-16T05:35:03.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3ec001830 con 0x7fa3f00a1a00 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): dump: 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.679 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2072260263 0 0) 0x7fa3f013ed10 con 0x7fa3f00a4290 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3d4003530 con 0x7fa3f00a4290 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3e80015c0 con 0x7fa3f00a4290 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): dump: 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.680 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3819937060 0 0) 0x7fa3f013eff0 con 0x7fa3f00a37a0 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): my global_id is 4438 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3f013ed10 con 0x7fa3f00a37a0 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2328568400 0 0) 0x7fa3f013be80 con 0x7fa3f00a1a00 2024-09-16T05:35:03.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 >> v1:172.21.15.6:6790/0 conn(0x7fa3f00a4290 legacy=0x7fa3f01d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 >> v1:172.21.15.63:6789/0 conn(0x7fa3f00a37a0 legacy=0x7fa3f013b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 monclient: found mon.a 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f00a7f20 con 0x7fa3f00a1a00 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient: _finish_auth 0 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 5 monclient: authenticate success, global_id 4434 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: _renew_subs 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:03.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f013da80 con 0x7fa3f00a1a00 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: _renew_subs 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3f013d5e0 con 0x7fa3f00a1a00 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3ec001c60 con 0x7fa3f00a1a00 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3ec003af0 con 0x7fa3f00a1a00 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e77fe640 4 set_mon_vals no callback set 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient: dump: 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.683 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa3d80065d0 con 0x7fa3d8002870 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa3ec003c90 con 0x7fa3f00a1a00 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7fa3ec044180 con 0x7fa3f00a1a00 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 810466926 0 0) 0x7fa3d80065d0 con 0x7fa3d8002870 2024-09-16T05:35:03.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 10 monclient(hunting): my global_id is 4441 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3f013be80 con 0x7fa3d8002870 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.659+0000 7fa3f6652640 10 monclient: get_auth_request con 0x7fa3d4036bf0 auth_method 0 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1300173613 0 0) 0x7fa3f013be80 con 0x7fa3d8002870 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3d80065d0 con 0x7fa3d8002870 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3e0001a10 con 0x7fa3d8002870 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 10 monclient: dump: 2024-09-16T05:35:03.685 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1160788761 0 0) 0x7fa3d80065d0 con 0x7fa3d8002870 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3d800a160 con 0x7fa3d8002870 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa3e0001d00 con 0x7fa3d8002870 2024-09-16T05:35:03.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 >> v1:172.21.15.63:6789/0 conn(0x7fa3d8002870 legacy=0x7fa3d8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.663+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 >> v1:172.21.15.63:6789/0 conn(0x7fa3d8002870 legacy=0x7fa3d8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.667+0000 7fa3f80b5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.667+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa3d8009e50 con 0x7fa3d8009110 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.667+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 460725309 0 0) 0x7fa3d8009e50 con 0x7fa3d8009110 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.667+0000 7fa3e67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.667+0000 7fa3e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.667+0000 7fa3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.667+0000 7fa3e67fc640 10 monclient(hunting): my global_id is 4444 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.667+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3d80065d0 con 0x7fa3d8009110 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 648486308 0 0) 0x7fa3d80065d0 con 0x7fa3d8009110 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3d8009e50 con 0x7fa3d8009110 2024-09-16T05:35:03.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3e0004bc0 con 0x7fa3d8009110 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 10 monclient: dump: 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.688 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2967635608 0 0) 0x7fa3d8009e50 con 0x7fa3d8009110 2024-09-16T05:35:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa3d800c3e0 con 0x7fa3d8009110 2024-09-16T05:35:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4295 (unknown 0 0 630744052) 0x7fa3e0004bc0 con 0x7fa3d8009110 2024-09-16T05:35:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:35:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:03.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 >> v1:172.21.15.63:6789/0 conn(0x7fa3d8009110 legacy=0x7fa3d80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3e67fc640 1 -- 172.21.15.6:0/4143784629 >> v1:172.21.15.63:6789/0 conn(0x7fa3d8009110 legacy=0x7fa3d80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:03.693 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 8, 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 32687, 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 27, 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:03.694 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.695 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.696 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:35:03.697 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:03.698 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:35:03.699 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:35:03.700 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.701 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.701 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.703 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-09-16T05:35:03.705 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:03.706 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:03.707 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:03.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:03.707 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:35:03.707 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:35:03.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa3d4036bf0 legacy=0x7fa3d40390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f80b5640 10 monclient: shutdown 2024-09-16T05:35:03.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f80b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:03.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 >> v1:172.21.15.6:6789/0 conn(0x7fa3f00a1a00 legacy=0x7fa3f013b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f6652640 1 -- 172.21.15.6:0/4143784629 reap_dead start 2024-09-16T05:35:03.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f6652640 1 -- 172.21.15.6:0/4143784629 reap_dead start 2024-09-16T05:35:03.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 shutdown_connections 2024-09-16T05:35:03.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 >> 172.21.15.6:0/4143784629 conn(0x7fa3f00a58a0 msgr2=0x7fa3f01401e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:03.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 shutdown_connections 2024-09-16T05:35:03.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.671+0000 7fa3f80b5640 1 -- 172.21.15.6:0/4143784629 wait complete. 2024-09-16T05:35:03.709 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:35:03.709 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:35:03.710 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:35:03.710 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:35:03.710 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:35:03.710 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:35:03.710 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:35:03.733 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:35:03.733 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:35:03.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:35:03.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient: get_monmap_and_config 2024-09-16T05:35:03.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient: build_initial_monmap 2024-09-16T05:35:03.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient: monmap: 2024-09-16T05:35:03.848 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:03.848178+0000 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:03.848178+0000 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:03.849 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 1 Processor -- start 2024-09-16T05:35:03.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 1 -- start start 2024-09-16T05:35:03.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient: init 2024-09-16T05:35:03.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:03.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient(hunting): picked mon.noname-b con 0x7fd18006cd20 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient(hunting): picked mon.noname-c con 0x7fd1800a3220 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient(hunting): picked mon.noname-a con 0x7fd18013cb80 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1800a2fe0 con 0x7fd18013cb80 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18006ecb0 con 0x7fd1800a3220 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd180073a30 con 0x7fd18006cd20 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd1863ad640 10 monclient(hunting): authenticate will time out at 965.655282s 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd1800a3220 legacy=0x7fd18006d810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd18494a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd18013cb80 0x7fd18013ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34572/0 (socket says 172.21.15.6:34572) 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd18006cd20 0x7fd18006d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42532/0 (socket says 172.21.15.6:42532) 2024-09-16T05:35:03.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd18494a640 1 -- 172.21.15.6:0/3493993579 learned_addr learned my addr 172.21.15.6:0/3493993579 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293477847 0 0) 0x7fd180073a30 con 0x7fd18006cd20 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 10 monclient(hunting): my global_id is 4436 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd168003530 con 0x7fd18006cd20 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3174455970 0 0) 0x7fd1800a2fe0 con 0x7fd18013cb80 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 10 monclient(hunting): my global_id is 4437 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.843+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd180073a30 con 0x7fd18013cb80 2024-09-16T05:35:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 680382949 0 0) 0x7fd180073a30 con 0x7fd18013cb80 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 >> v1:172.21.15.6:6790/0 conn(0x7fd18006cd20 legacy=0x7fd18006d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 >> v1:172.21.15.63:6789/0 conn(0x7fd1800a3220 legacy=0x7fd18006d810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: _finish_hunting 0 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 1 monclient: found mon.noname-a 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1800a29f0 con 0x7fd18013cb80 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: _finish_auth 0 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: _check_auth_tickets 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd178001600 con 0x7fd18013cb80 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 5 monclient: authenticate success, global_id 4437 2024-09-16T05:35:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: dump: 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1780018e0 con 0x7fd18013cb80 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 1 -- 172.21.15.6:0/3493993579 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd178001cf0 con 0x7fd18013cb80 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 4 set_mon_vals no callback set 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17effd640 10 monclient: dump: 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: shutdown 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- 172.21.15.6:0/3493993579 >> v1:172.21.15.6:6789/0 conn(0x7fd18013cb80 legacy=0x7fd18013ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- 172.21.15.6:0/3493993579 shutdown_connections 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- 172.21.15.6:0/3493993579 >> 172.21.15.6:0/3493993579 conn(0x7fd1800a3880 msgr2=0x7fd1800a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- 172.21.15.6:0/3493993579 shutdown_connections 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- 172.21.15.6:0/3493993579 wait complete. 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: build_initial_monmap 2024-09-16T05:35:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: monmap: 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:03.859 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 Processor -- start 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- start start 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: init 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient(hunting): picked mon.noname-a con 0x7fd18006cd20 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient(hunting): picked mon.noname-c con 0x7fd1800a3220 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient(hunting): picked mon.noname-b con 0x7fd18013cb80 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18009bdb0 con 0x7fd18006cd20 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1800a06e0 con 0x7fd18013cb80 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1800a0200 con 0x7fd1800a3220 2024-09-16T05:35:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient(hunting): authenticate will time out at 965.658480s 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd18006cd20 0x7fd18009af90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34574/0 (socket says 172.21.15.6:34574) 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17ffff640 1 -- 172.21.15.6:0/3594012425 learned_addr learned my addr 172.21.15.6:0/3594012425 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17f7fe640 1 --1- 172.21.15.6:0/3594012425 >> v1:172.21.15.6:6790/0 conn(0x7fd1800a3220 0x7fd18009b6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42534/0 (socket says 172.21.15.6:42534) 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd18494a640 1 -- 172.21.15.6:0/3594012425 >> v1:172.21.15.63:6789/0 conn(0x7fd18013cb80 legacy=0x7fd1801d3ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2489410259 0 0) 0x7fd18009bdb0 con 0x7fd18006cd20 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): my global_id is 4440 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd16c003530 con 0x7fd18006cd20 2024-09-16T05:35:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538652572 0 0) 0x7fd1800a0200 con 0x7fd1800a3220 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): my global_id is 4439 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd18009bdb0 con 0x7fd1800a3220 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 647452269 0 0) 0x7fd18009bdb0 con 0x7fd1800a3220 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1800a0200 con 0x7fd1800a3220 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd174004100 con 0x7fd1800a3220 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): dump: 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3323949944 0 0) 0x7fd16c003530 con 0x7fd18006cd20 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd18009bdb0 con 0x7fd18006cd20 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd170001b20 con 0x7fd18006cd20 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient(hunting): dump: 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2569821982 0 0) 0x7fd1800a0200 con 0x7fd1800a3220 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 >> v1:172.21.15.63:6789/0 conn(0x7fd18013cb80 legacy=0x7fd1801d3ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:35:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 >> v1:172.21.15.6:6789/0 conn(0x7fd18006cd20 legacy=0x7fd18009af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 monclient: found mon.c 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18009e0e0 con 0x7fd1800a3220 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient: _finish_auth 0 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 5 monclient: authenticate success, global_id 4439 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: _renew_subs 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd180138840 con 0x7fd1800a3220 2024-09-16T05:35:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: _renew_subs 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd18009d4c0 con 0x7fd1800a3220 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1740046a0 con 0x7fd1800a3220 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd174004a70 con 0x7fd1800a3220 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17cff9640 10 monclient: dump: 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:03.869 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.847+0000 7fd17dffb640 4 set_mon_vals no callback set 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.851+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd174004c90 con 0x7fd1800a3220 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.851+0000 7fd1863ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.851+0000 7fd1863ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.851+0000 7fd1863ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.851+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd160005090 con 0x7fd1800a3220 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.851+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7fd1740441d0 con 0x7fd1800a3220 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.851+0000 7fd17ffff640 10 monclient: get_auth_request con 0x7fd16c036a00 auth_method 0 2024-09-16T05:35:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.855+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd174015480 con 0x7fd1800a3220 2024-09-16T05:35:03.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.855+0000 7fd17cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:03.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:03.855+0000 7fd17cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.079+0000 7fd1863ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:35:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.079+0000 7fd1863ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:35:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.079+0000 7fd1863ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.079+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd160005380 con 0x7fd1800a3220 2024-09-16T05:35:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.079+0000 7fd17cff9640 1 -- 172.21.15.6:0/3594012425 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2290618487) 0x7fd174014d80 con 0x7fd1800a3220 2024-09-16T05:35:04.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.079+0000 7fd17cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:35:04.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.079+0000 7fd17cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:04.086 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:35:04.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.083+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd16c036a00 legacy=0x7fd16c038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:04.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.083+0000 7fd1863ad640 10 monclient: shutdown 2024-09-16T05:35:04.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.083+0000 7fd1863ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:04.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.083+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 >> v1:172.21.15.6:6790/0 conn(0x7fd1800a3220 legacy=0x7fd18009b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:04.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.083+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 shutdown_connections 2024-09-16T05:35:04.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.083+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 >> 172.21.15.6:0/3594012425 conn(0x7fd1800a3880 msgr2=0x7fd18013c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:04.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.083+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 shutdown_connections 2024-09-16T05:35:04.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:04.083+0000 7fd1863ad640 1 -- 172.21.15.6:0/3594012425 wait complete. 2024-09-16T05:35:07.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:35:07.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient: get_monmap_and_config 2024-09-16T05:35:07.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient: build_initial_monmap 2024-09-16T05:35:07.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient: monmap: 2024-09-16T05:35:07.240 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:07.240175+0000 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:07.240175+0000 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:07.241 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:07.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 1 Processor -- start 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 1 -- start start 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient: init 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d2809b2f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d28136b70 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d2809b6d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d280a19c0 con 0x7f7d2809b6d0 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d28099ae0 con 0x7f7d2809b2f0 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d28099dc0 con 0x7f7d28136b70 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient(hunting): authenticate will time out at 969.046735s 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d277fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7d2809b6d0 0x7f7d2813f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34586/0 (socket says 172.21.15.6:34586) 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d267fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7d28136b70 0x7f7d280a12b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42542/0 (socket says 172.21.15.6:42542) 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d26ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7d2809b2f0 legacy=0x7f7d280a0ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d277fe640 1 -- 172.21.15.6:0/1484113910 learned_addr learned my addr 172.21.15.6:0/1484113910 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225843825 0 0) 0x7f7d280a19c0 con 0x7f7d2809b6d0 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient(hunting): my global_id is 4443 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d10003530 con 0x7f7d2809b6d0 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 593219414 0 0) 0x7f7d10003530 con 0x7f7d2809b6d0 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 >> v1:172.21.15.6:6790/0 conn(0x7f7d28136b70 legacy=0x7f7d280a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:07.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 >> v1:172.21.15.63:6789/0 conn(0x7f7d2809b2f0 legacy=0x7f7d280a0ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 monclient: found mon.noname-a 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2809bae0 con 0x7f7d2809b6d0 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: _finish_auth 0 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 5 monclient: authenticate success, global_id 4443 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d200019a0 con 0x7f7d2809b6d0 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:07.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: dump: 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:07.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d20001c00 con 0x7f7d2809b6d0 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 1 -- 172.21.15.6:0/1484113910 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d20002050 con 0x7f7d2809b6d0 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d2d77d640 4 set_mon_vals no callback set 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.235+0000 7f7d25ffb640 10 monclient: dump: 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:07.252 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: shutdown 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- 172.21.15.6:0/1484113910 >> v1:172.21.15.6:6789/0 conn(0x7f7d2809b6d0 legacy=0x7f7d2813f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- 172.21.15.6:0/1484113910 shutdown_connections 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- 172.21.15.6:0/1484113910 >> 172.21.15.6:0/1484113910 conn(0x7f7d280a1c90 msgr2=0x7f7d280a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- 172.21.15.6:0/1484113910 shutdown_connections 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- 172.21.15.6:0/1484113910 wait complete. 2024-09-16T05:35:07.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: build_initial_monmap 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: monmap: 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 Processor -- start 2024-09-16T05:35:07.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- start start 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: init 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d2809b2f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d2809b6d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d28136b70 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d281301b0 con 0x7f7d2809b2f0 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d28132000 con 0x7f7d2809b6d0 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d280a0660 con 0x7f7d28136b70 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient(hunting): authenticate will time out at 969.049957s 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d267fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7d2809b6d0 legacy=0x7f7d28133530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:07.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d277fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7d28136b70 0x7f7d281d38e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42552/0 (socket says 172.21.15.6:42552) 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d277fe640 1 -- 172.21.15.6:0/2797968270 learned_addr learned my addr 172.21.15.6:0/2797968270 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 683679122 0 0) 0x7f7d281301b0 con 0x7f7d2809b2f0 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient(hunting): my global_id is 4446 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d14003560 con 0x7f7d2809b2f0 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1042761119 0 0) 0x7f7d14003560 con 0x7f7d2809b2f0 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d281301b0 con 0x7f7d2809b2f0 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d18004100 con 0x7f7d2809b2f0 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:07.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient(hunting): dump: 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:07.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3037971948 0 0) 0x7f7d281301b0 con 0x7f7d2809b2f0 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 >> v1:172.21.15.6:6790/0 conn(0x7f7d28136b70 legacy=0x7f7d281d38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 >> v1:172.21.15.63:6789/0 conn(0x7f7d2809b6d0 legacy=0x7f7d28133530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 monclient: found mon.a 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d280a0a00 con 0x7f7d2809b2f0 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:07.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 5 monclient: authenticate success, global_id 4446 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: _renew_subs 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d28131a40 con 0x7f7d2809b2f0 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: _renew_subs 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d28133c40 con 0x7f7d2809b2f0 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d180043f0 con 0x7f7d2809b2f0 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d180047c0 con 0x7f7d2809b2f0 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:07.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d257fa640 4 set_mon_vals no callback set 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 10 monclient: dump: 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7d180049e0 con 0x7f7d2809b2f0 2024-09-16T05:35:07.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:07.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:07.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:07.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.239+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d08005090 con 0x7f7d2809b2f0 2024-09-16T05:35:07.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.243+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7f7d18044200 con 0x7f7d2809b2f0 2024-09-16T05:35:07.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.243+0000 7f7d267fc640 10 monclient: get_auth_request con 0x7f7d140369d0 auth_method 0 2024-09-16T05:35:07.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.247+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7d18015450 con 0x7f7d2809b2f0 2024-09-16T05:35:07.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.247+0000 7f7d0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:07.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.247+0000 7f7d0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:35:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:35:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d08005380 con 0x7f7d2809b2f0 2024-09-16T05:35:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d0f7fe640 1 -- 172.21.15.6:0/2797968270 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 941931038) 0x7f7d18019100 con 0x7f7d2809b2f0 2024-09-16T05:35:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:35:07.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:07.472 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:35:07.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7d140369d0 legacy=0x7f7d14038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:07.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 10 monclient: shutdown 2024-09-16T05:35:07.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:07.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 >> v1:172.21.15.6:6789/0 conn(0x7f7d2809b2f0 legacy=0x7f7d281cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:07.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 shutdown_connections 2024-09-16T05:35:07.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 >> 172.21.15.6:0/2797968270 conn(0x7f7d280a1c90 msgr2=0x7f7d28134ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:07.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 shutdown_connections 2024-09-16T05:35:07.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:07.467+0000 7f7d2d77d640 1 -- 172.21.15.6:0/2797968270 wait complete. 2024-09-16T05:35:10.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient: get_monmap_and_config 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient: build_initial_monmap 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient: monmap: 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:10.627660+0000 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:10.627660+0000 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:10.629 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:10.629 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 1 Processor -- start 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 1 -- start start 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient: init 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient(hunting): picked mon.noname-b con 0x7fc424134880 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient(hunting): picked mon.noname-a con 0x7fc4240a0f20 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient(hunting): picked mon.noname-c con 0x7fc42413ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4240a3180 con 0x7fc4240a0f20 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc42409f2a0 con 0x7fc42413ae70 2024-09-16T05:35:10.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4240a0260 con 0x7fc424134880 2024-09-16T05:35:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc42a204640 10 monclient(hunting): authenticate will time out at 972.434421s 2024-09-16T05:35:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc4237fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc424134880 0x7fc424134c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49846/0 (socket says 172.21.15.6:49846) 2024-09-16T05:35:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc4237fe640 1 -- 172.21.15.6:0/4007624875 learned_addr learned my addr 172.21.15.6:0/4007624875 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:10.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.623+0000 7fc423fff640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc42413d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:10.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:10.827+0000 7fc423fff640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc42413d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:11.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:11.231+0000 7fc423fff640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc42413d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:12.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:12.035+0000 7fc423fff640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc42413d260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:13.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 10 monclient(hunting): tick 2024-09-16T05:35:13.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:35:13.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 1 monclient(hunting): continuing hunt 2024-09-16T05:35:13.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:35:13.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.6:6790/0 conn(0x7fc424134880 legacy=0x7fc424134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:13.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc42413d260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.6:6789/0 conn(0x7fc4240a0f20 legacy=0x7fc424139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc42413ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc410006e60 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc424134880 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4240a0540 con 0x7fc410006e60 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc42409f2a0 con 0x7fc42413ae70 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4237fe640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4100103b0 con 0x7fc424134880 2024-09-16T05:35:13.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.623+0000 7fc4217fa640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:13.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:13.827+0000 7fc4237fe640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:14.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:14.231+0000 7fc4237fe640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:15.035+0000 7fc4237fe640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 10 monclient(hunting): tick 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 1 monclient(hunting): continuing hunt 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.6:6790/0 conn(0x7fc424134880 legacy=0x7fc41000cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.6:6789/0 conn(0x7fc410006e60 legacy=0x7fc4100092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc410010680 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc42413ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc410006e60 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:16.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4100116c0 con 0x7fc410006e60 2024-09-16T05:35:16.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc42409f2a0 con 0x7fc42413ae70 2024-09-16T05:35:16.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 1 -- 172.21.15.6:0/4007624875 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc410011370 con 0x7fc410010680 2024-09-16T05:35:16.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc4217fa640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:16.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.623+0000 7fc422ffd640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:16.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:16.827+0000 7fc422ffd640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:17.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:17.231+0000 7fc422ffd640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:18.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865781255 0 0) 0x7fc410011370 con 0x7fc410010680 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient(hunting): my global_id is 4442 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc40c003530 con 0x7fc410010680 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1362201899 0 0) 0x7fc40c003530 con 0x7fc410010680 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.63:6789/0 conn(0x7fc42413ae70 legacy=0x7fc410005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.6:6789/0 conn(0x7fc410006e60 legacy=0x7fc4100092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: _finish_hunting 0 2024-09-16T05:35:18.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 monclient: found mon.noname-b 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41000cc10 con 0x7fc410010680 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: _finish_auth 0 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: _check_auth_tickets 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4140015d0 con 0x7fc410010680 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 5 monclient: authenticate success, global_id 4442 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: dump: 2024-09-16T05:35:18.014 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc414001830 con 0x7fc410010680 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:18.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 4 set_mon_vals no callback set 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 1 -- 172.21.15.6:0/4007624875 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc414001c80 con 0x7fc410010680 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc4227fc640 10 monclient: dump: 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.016 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.017 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 10 monclient: shutdown 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 1 -- 172.21.15.6:0/4007624875 >> v1:172.21.15.6:6790/0 conn(0x7fc410010680 legacy=0x7fc410010ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 1 -- 172.21.15.6:0/4007624875 shutdown_connections 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 1 -- 172.21.15.6:0/4007624875 >> 172.21.15.6:0/4007624875 conn(0x7fc4240a37e0 msgr2=0x7fc4240a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 1 -- 172.21.15.6:0/4007624875 shutdown_connections 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.007+0000 7fc42a204640 1 -- 172.21.15.6:0/4007624875 wait complete. 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: build_initial_monmap 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: monmap: 2024-09-16T05:35:18.020 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 1 Processor -- start 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 1 -- start start 2024-09-16T05:35:18.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: init 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient(hunting): picked mon.noname-b con 0x7fc410006e60 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient(hunting): picked mon.noname-a con 0x7fc410010680 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient(hunting): picked mon.noname-c con 0x7fc42413ae70 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4240742c0 con 0x7fc410010680 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4240a6590 con 0x7fc410006e60 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc42413a730 con 0x7fc42413ae70 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient(hunting): authenticate will time out at 979.822109s 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc4237fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc410006e60 legacy=0x7fc424133540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc422ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc410010680 0x7fc4241ce310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52334/0 (socket says 172.21.15.6:52334) 2024-09-16T05:35:18.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc422ffd640 1 -- 172.21.15.6:0/1239366362 learned_addr learned my addr 172.21.15.6:0/1239366362 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668737603 0 0) 0x7fc42413a730 con 0x7fc42413ae70 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient(hunting): my global_id is 4445 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc41000ce20 con 0x7fc42413ae70 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1798167522 0 0) 0x7fc41000ce20 con 0x7fc42413ae70 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc42413a730 con 0x7fc42413ae70 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc41c004570 con 0x7fc42413ae70 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient(hunting): dump: 2024-09-16T05:35:18.023 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 929584132 0 0) 0x7fc42413a730 con 0x7fc42413ae70 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 >> v1:172.21.15.63:6789/0 conn(0x7fc410006e60 legacy=0x7fc424133540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:35:18.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 >> v1:172.21.15.6:6789/0 conn(0x7fc410010680 legacy=0x7fc4241ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 monclient: found mon.c 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42411e920 con 0x7fc42413ae70 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient: _finish_auth 0 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 5 monclient: authenticate success, global_id 4445 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: _renew_subs 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc42411ec00 con 0x7fc42413ae70 2024-09-16T05:35:18.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: _renew_subs 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc424133c50 con 0x7fc42413ae70 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc41c0049c0 con 0x7fc42413ae70 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc41c004d10 con 0x7fc42413ae70 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc40bfff640 10 monclient: dump: 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.011+0000 7fc421ffb640 4 set_mon_vals no callback set 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.015+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc41c004eb0 con 0x7fc42413ae70 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.015+0000 7fc42a204640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.015+0000 7fc42a204640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.015+0000 7fc42a204640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.015+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc404005090 con 0x7fc42413ae70 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.015+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7fc41c0442d0 con 0x7fc42413ae70 2024-09-16T05:35:18.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.015+0000 7fc4237fe640 10 monclient: get_auth_request con 0x7fc410037b50 auth_method 0 2024-09-16T05:35:18.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.019+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc41c015580 con 0x7fc42413ae70 2024-09-16T05:35:18.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.019+0000 7fc40bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:18.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.019+0000 7fc40bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:18.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.255+0000 7fc42a204640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:35:18.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.255+0000 7fc42a204640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:35:18.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.255+0000 7fc42a204640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:18.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.255+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc404005380 con 0x7fc42413ae70 2024-09-16T05:35:18.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.255+0000 7fc40bfff640 1 -- 172.21.15.6:0/1239366362 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 2210019845) 0x7fc41c017580 con 0x7fc42413ae70 2024-09-16T05:35:18.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.255+0000 7fc40bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:35:18.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.255+0000 7fc40bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:18.262 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:35:18.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.259+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc410037b50 legacy=0x7fc41003a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.259+0000 7fc42a204640 10 monclient: shutdown 2024-09-16T05:35:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.259+0000 7fc42a204640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.259+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 >> v1:172.21.15.6:6790/0 conn(0x7fc42413ae70 legacy=0x7fc4241d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.259+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 shutdown_connections 2024-09-16T05:35:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.259+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 >> 172.21.15.6:0/1239366362 conn(0x7fc4240a37e0 msgr2=0x7fc424070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.259+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 shutdown_connections 2024-09-16T05:35:18.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.259+0000 7fc42a204640 1 -- 172.21.15.6:0/1239366362 wait complete. 2024-09-16T05:35:18.281 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:35:18.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:35:18.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:35:18.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient: get_monmap_and_config 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient: build_initial_monmap 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient: monmap: 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:18.407695+0000 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:18.407695+0000 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:18.408 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:18.409 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:18.409 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 1 Processor -- start 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 1 -- start start 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient: init 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient(hunting): picked mon.noname-a con 0x7f31040a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f31040a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f31040a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310406e520 con 0x7f31040a37a0 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31040a0780 con 0x7f31040a4290 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31040a4c70 con 0x7f31040a1a00 2024-09-16T05:35:18.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient(hunting): authenticate will time out at 980.214325s 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3102ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f31040a4290 legacy=0x7f310413bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3103fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f31040a1a00 0x7f310413f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49878/0 (socket says 172.21.15.6:49878) 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31037fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f31040a37a0 0x7f31040a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52342/0 (socket says 172.21.15.6:52342) 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3103fff640 1 -- 172.21.15.6:0/2447354433 learned_addr learned my addr 172.21.15.6:0/2447354433 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278230483 0 0) 0x7f31040a4c70 con 0x7f31040a1a00 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient(hunting): my global_id is 4448 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30ec003530 con 0x7f31040a1a00 2024-09-16T05:35:18.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287582921 0 0) 0x7f310406e520 con 0x7f31040a37a0 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient(hunting): my global_id is 4449 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31040a4c70 con 0x7f31040a37a0 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4109256044 0 0) 0x7f31040a4c70 con 0x7f31040a37a0 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 >> v1:172.21.15.6:6790/0 conn(0x7f31040a1a00 legacy=0x7f310413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 >> v1:172.21.15.63:6789/0 conn(0x7f31040a4290 legacy=0x7f310413bab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: _finish_hunting 0 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 monclient: found mon.noname-a 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:18.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31040a65f0 con 0x7f31040a37a0 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: _finish_auth 0 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: _check_auth_tickets 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f30f4001600 con 0x7f31040a37a0 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 5 monclient: authenticate success, global_id 4449 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: dump: 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.415 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30f4001960 con 0x7f31040a37a0 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 1 -- 172.21.15.6:0/2447354433 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f30f4001d30 con 0x7f31040a37a0 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f3109ac5640 4 set_mon_vals no callback set 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.403+0000 7f31027fc640 10 monclient: dump: 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: shutdown 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:18.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- 172.21.15.6:0/2447354433 >> v1:172.21.15.6:6789/0 conn(0x7f31040a37a0 legacy=0x7f31040a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- 172.21.15.6:0/2447354433 shutdown_connections 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- 172.21.15.6:0/2447354433 >> 172.21.15.6:0/2447354433 conn(0x7f31040a58a0 msgr2=0x7f31040a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- 172.21.15.6:0/2447354433 shutdown_connections 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- 172.21.15.6:0/2447354433 wait complete. 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: build_initial_monmap 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: monmap: 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.418 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:18.419 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:18.419 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:18.419 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 Processor -- start 2024-09-16T05:35:18.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- start start 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: init 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient(hunting): picked mon.noname-a con 0x7f31040a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f31040a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f31040a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3104135900 con 0x7f31040a1a00 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31040a7bb0 con 0x7f31040a37a0 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31040a80e0 con 0x7f31040a4290 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient(hunting): authenticate will time out at 980.218268s 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f31037fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f31040a1a00 0x7f3104134ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52356/0 (socket says 172.21.15.6:52356) 2024-09-16T05:35:18.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3102ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f31040a37a0 legacy=0x7f31041351f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f31037fe640 1 -- 172.21.15.6:0/2923148775 learned_addr learned my addr 172.21.15.6:0/2923148775 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3415854575 0 0) 0x7f3104135900 con 0x7f31040a1a00 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): my global_id is 4452 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30f0003560 con 0x7f31040a1a00 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3844541093 0 0) 0x7f31040a80e0 con 0x7f31040a4290 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): my global_id is 4451 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3104135900 con 0x7f31040a4290 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3977968227 0 0) 0x7f30f0003560 con 0x7f31040a1a00 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31040a80e0 con 0x7f31040a1a00 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f30f4001600 con 0x7f31040a1a00 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): dump: 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.427 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4141547620 0 0) 0x7f3104135900 con 0x7f31040a4290 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30f0003560 con 0x7f31040a4290 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f30fc001400 con 0x7f31040a4290 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient(hunting): dump: 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.428 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2996095150 0 0) 0x7f31040a80e0 con 0x7f31040a1a00 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 >> v1:172.21.15.6:6790/0 conn(0x7f31040a4290 legacy=0x7f31041d3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 >> v1:172.21.15.63:6789/0 conn(0x7f31040a37a0 legacy=0x7f31041351f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient: _finish_hunting 0 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 monclient: found mon.a 2024-09-16T05:35:18.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31040a0100 con 0x7f31040a1a00 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient: _finish_auth 0 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient: _check_auth_tickets 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 5 monclient: authenticate success, global_id 4452 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: _renew_subs 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31041386d0 con 0x7f31040a1a00 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: _renew_subs 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3104137060 con 0x7f31040a1a00 2024-09-16T05:35:18.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30f4001960 con 0x7f31040a1a00 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f30f4004db0 con 0x7f31040a1a00 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f3101ffb640 4 set_mon_vals no callback set 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.407+0000 7f30ebfff640 10 monclient: dump: 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.431 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f30f4004f50 con 0x7f31040a1a00 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f3109ac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f30e40065d0 con 0x7f30e4002870 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7f30f4044220 con 0x7f31040a1a00 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341782649 0 0) 0x7f30e40065d0 con 0x7f30e4002870 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient(hunting): my global_id is 4455 2024-09-16T05:35:18.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31040a80e0 con 0x7f30e4002870 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f3103fff640 10 monclient: get_auth_request con 0x7f30f0036aa0 auth_method 0 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1697613937 0 0) 0x7f31040a80e0 con 0x7f30e4002870 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30e40065d0 con 0x7f30e4002870 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f30f8003fc0 con 0x7f30e4002870 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient: dump: 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.433 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1138641067 0 0) 0x7f30e40065d0 con 0x7f30e4002870 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30e400a160 con 0x7f30e4002870 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f30f8006340 con 0x7f30e4002870 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 >> v1:172.21.15.6:6789/0 conn(0x7f30e4002870 legacy=0x7f30e4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.411+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 >> v1:172.21.15.6:6789/0 conn(0x7f30e4002870 legacy=0x7f30e4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.415+0000 7f3109ac5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:35:18.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.415+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f30e4009e50 con 0x7f30e4009110 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.415+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838359882 0 0) 0x7f30e4009e50 con 0x7f30e4009110 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.415+0000 7f30ebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.415+0000 7f30ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.415+0000 7f30ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.415+0000 7f30ebfff640 10 monclient(hunting): my global_id is 4458 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.415+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30e40065d0 con 0x7f30e4009110 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1254201573 0 0) 0x7f30e40065d0 con 0x7f30e4009110 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30e4009e50 con 0x7f30e4009110 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f30f8004570 con 0x7f30e4009110 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 10 monclient: dump: 2024-09-16T05:35:18.435 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1969788209 0 0) 0x7f30e4009e50 con 0x7f30e4009110 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f30e400c3e0 con 0x7f30e4009110 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4286 (unknown 0 0 2214251887) 0x7f30f8005900 con 0x7f30e4009110 2024-09-16T05:35:18.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 >> v1:172.21.15.6:6789/0 conn(0x7f30e4009110 legacy=0x7f30e40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f30ebfff640 1 -- 172.21.15.6:0/2923148775 >> v1:172.21.15.6:6789/0 conn(0x7f30e4009110 legacy=0x7f30e40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 12, 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 47475, 2024-09-16T05:35:18.437 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:18.438 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:18.439 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:35:18.440 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:18.441 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:35:18.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:18.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.445 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:35:18.446 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.447 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:35:18.448 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:35:18.449 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:35:18.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 >> v1:172.21.15.63:6812/3643207211 conn(0x7f30f0036aa0 legacy=0x7f30f0038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3109ac5640 10 monclient: shutdown 2024-09-16T05:35:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3109ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 >> v1:172.21.15.6:6789/0 conn(0x7f31040a1a00 legacy=0x7f3104134ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3103fff640 1 -- 172.21.15.6:0/2923148775 reap_dead start 2024-09-16T05:35:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3103fff640 1 -- 172.21.15.6:0/2923148775 reap_dead start 2024-09-16T05:35:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 shutdown_connections 2024-09-16T05:35:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 >> 172.21.15.6:0/2923148775 conn(0x7f31040a58a0 msgr2=0x7f31040a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 shutdown_connections 2024-09-16T05:35:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.419+0000 7f3109ac5640 1 -- 172.21.15.6:0/2923148775 wait complete. 2024-09-16T05:35:18.451 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:35:18.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:35:18.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:35:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient: get_monmap_and_config 2024-09-16T05:35:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient: build_initial_monmap 2024-09-16T05:35:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient: monmap: 2024-09-16T05:35:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:35:18.564714+0000 2024-09-16T05:35:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:35:18.564714+0000 2024-09-16T05:35:18.565 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:18.566 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 1 Processor -- start 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 1 -- start start 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient: init 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient(hunting): picked mon.noname-c con 0x7f07140a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient(hunting): picked mon.noname-b con 0x7f07140a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient(hunting): picked mon.noname-a con 0x7f07140a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07140a33a0 con 0x7f07140a2980 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07140a07a0 con 0x7f07140a2190 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07140a2d90 con 0x7f07140a2570 2024-09-16T05:35:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f071afa3640 10 monclient(hunting): authenticate will time out at 980.371282s 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f0718d3f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f07140a2190 legacy=0x7f07140a1a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f0719540640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f07140a2980 0x7f071413f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52388/0 (socket says 172.21.15.6:52388) 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f0719540640 1 -- 172.21.15.6:0/3041137444 learned_addr learned my addr 172.21.15.6:0/3041137444 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 135550504 0 0) 0x7f07140a33a0 con 0x7f07140a2980 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 10 monclient(hunting): my global_id is 4461 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06fc003530 con 0x7f07140a2980 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375167930 0 0) 0x7f07140a2d90 con 0x7f07140a2570 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 10 monclient(hunting): my global_id is 4454 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.559+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07140a33a0 con 0x7f07140a2570 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2283532508 0 0) 0x7f06fc003530 con 0x7f07140a2980 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 >> v1:172.21.15.6:6790/0 conn(0x7f07140a2570 legacy=0x7f071413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 >> v1:172.21.15.63:6789/0 conn(0x7f07140a2190 legacy=0x7f07140a1a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: _finish_hunting 0 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 1 monclient: found mon.noname-a 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07140a66e0 con 0x7f07140a2980 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: _finish_auth 0 2024-09-16T05:35:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: _check_auth_tickets 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f070c0019a0 con 0x7f07140a2980 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: dump: 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 5 monclient: authenticate success, global_id 4461 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f070c001c40 con 0x7f07140a2980 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 1 -- 172.21.15.6:0/3041137444 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f070c002050 con 0x7f07140a2980 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07137fe640 10 monclient: dump: 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 4 set_mon_vals no callback set 2024-09-16T05:35:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: shutdown 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- 172.21.15.6:0/3041137444 >> v1:172.21.15.6:6789/0 conn(0x7f07140a2980 legacy=0x7f071413f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- 172.21.15.6:0/3041137444 shutdown_connections 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- 172.21.15.6:0/3041137444 >> 172.21.15.6:0/3041137444 conn(0x7f07140a9850 msgr2=0x7f07140a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- 172.21.15.6:0/3041137444 shutdown_connections 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- 172.21.15.6:0/3041137444 wait complete. 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: build_initial_monmap 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: monmap: 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:18.576 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 Processor -- start 2024-09-16T05:35:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- start start 2024-09-16T05:35:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: init 2024-09-16T05:35:18.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:18.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:35:18.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient(hunting): picked mon.noname-c con 0x7f07140a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:18.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient(hunting): picked mon.noname-a con 0x7f07140a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:18.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient(hunting): picked mon.noname-b con 0x7f07140a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07141382a0 con 0x7f07140a2570 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0714138580 con 0x7f07140a2980 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0714135a70 con 0x7f07140a2190 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient(hunting): authenticate will time out at 980.374380s 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f0718d3f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f07140a2190 0x7f0714134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49904/0 (socket says 172.21.15.6:49904) 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f0713fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f07140a2570 0x7f0714135360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52400/0 (socket says 172.21.15.6:52400) 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f0718d3f640 1 -- 172.21.15.6:0/4083099301 learned_addr learned my addr 172.21.15.6:0/4083099301 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f0719540640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.63:6789/0 conn(0x7f07140a2980 legacy=0x7f07141d3ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2817613349 0 0) 0x7f07141382a0 con 0x7f07140a2570 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): my global_id is 4464 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0700003530 con 0x7f07140a2570 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3638339416 0 0) 0x7f0714135a70 con 0x7f07140a2190 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): my global_id is 4457 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07141382a0 con 0x7f07140a2190 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3992761247 0 0) 0x7f0700003530 con 0x7f07140a2570 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0714135a70 con 0x7f07140a2570 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0708001880 con 0x7f07140a2570 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): dump: 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2113129929 0 0) 0x7f07141382a0 con 0x7f07140a2190 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0700003530 con 0x7f07140a2190 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0704004220 con 0x7f07140a2190 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient(hunting): dump: 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4275991823 0 0) 0x7f0714135a70 con 0x7f07140a2570 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.6:6790/0 conn(0x7f07140a2190 legacy=0x7f0714134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.63:6789/0 conn(0x7f07140a2980 legacy=0x7f07141d3ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient: _finish_hunting 0 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 monclient: found mon.a 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07140a8380 con 0x7f07140a2570 2024-09-16T05:35:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient: _finish_auth 0 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient: _check_auth_tickets 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 5 monclient: authenticate success, global_id 4464 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: _renew_subs 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07140a7e50 con 0x7f07140a2570 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: _renew_subs 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0714137130 con 0x7f07140a2570 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0708001c50 con 0x7f07140a2570 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0708003a50 con 0x7f07140a2570 2024-09-16T05:35:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f07117fa640 10 monclient: dump: 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.588 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.563+0000 7f0712ffd640 4 set_mon_vals no callback set 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f071afa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f06f40065d0 con 0x7f06f4002870 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0708003c70 con 0x7f07140a2570 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7f0708044200 con 0x7f07140a2570 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723191699 0 0) 0x7f06f40065d0 con 0x7f06f4002870 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient(hunting): my global_id is 4460 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0714135a70 con 0x7f06f4002870 2024-09-16T05:35:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1725521128 0 0) 0x7f0714135a70 con 0x7f06f4002870 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06f40065d0 con 0x7f06f4002870 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07040048a0 con 0x7f06f4002870 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient: dump: 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f0719540640 10 monclient: get_auth_request con 0x7f0700036a20 auth_method 0 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3653049944 0 0) 0x7f06f40065d0 con 0x7f06f4002870 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06f400a160 con 0x7f06f4002870 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0704004ac0 con 0x7f06f4002870 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.6:6790/0 conn(0x7f06f4002870 legacy=0x7f06f4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.567+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.6:6790/0 conn(0x7f06f4002870 legacy=0x7f06f4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:18.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.571+0000 7f071afa3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.571+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f06f4009e30 con 0x7f06f40090f0 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.571+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3508995055 0 0) 0x7f06f4009e30 con 0x7f06f40090f0 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.571+0000 7f07117fa640 10 monclient(hunting): _negotiate 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.571+0000 7f07117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.571+0000 7f07117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.571+0000 7f07117fa640 10 monclient(hunting): my global_id is 4463 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.571+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06f40065d0 con 0x7f06f40090f0 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 561537132 0 0) 0x7f06f40065d0 con 0x7f06f40090f0 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06f4009e30 con 0x7f06f40090f0 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0704005080 con 0x7f06f40090f0 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:18.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 10 monclient: dump: 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 871012616 0 0) 0x7f06f4009e30 con 0x7f06f40090f0 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f06f400c3a0 con 0x7f06f40090f0 2024-09-16T05:35:18.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4117 (unknown 0 0 2312472140) 0x7f0704006470 con 0x7f06f40090f0 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.6:6790/0 conn(0x7f06f40090f0 legacy=0x7f06f4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f07117fa640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.6:6790/0 conn(0x7f06f40090f0 legacy=0x7f06f4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 12, 2024-09-16T05:35:18.594 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 47631, 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:18.595 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:18.596 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:35:18.597 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:35:18.598 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:35:18.599 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:35:18.600 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:35:18.601 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.603 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.604 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:35:18.605 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:35:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:35:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:35:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:35:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:35:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:35:18.606 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:35:18.606 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:35:18.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0700036a20 legacy=0x7f0700038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f071afa3640 10 monclient: shutdown 2024-09-16T05:35:18.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f071afa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:18.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 >> v1:172.21.15.6:6789/0 conn(0x7f07140a2570 legacy=0x7f0714135360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:18.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f0719540640 1 -- 172.21.15.6:0/4083099301 reap_dead start 2024-09-16T05:35:18.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f0719540640 1 -- 172.21.15.6:0/4083099301 reap_dead start 2024-09-16T05:35:18.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 shutdown_connections 2024-09-16T05:35:18.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 >> 172.21.15.6:0/4083099301 conn(0x7f07140a9850 msgr2=0x7f071409d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:18.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 shutdown_connections 2024-09-16T05:35:18.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:35:18.575+0000 7f071afa3640 1 -- 172.21.15.6:0/4083099301 wait complete. 2024-09-16T05:35:18.608 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:35:18.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:35:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:Updating files: 94% (12118/12889) Updating files: 95% (12245/12889) Updating files: 96% (12374/12889) Updating files: 97% (12503/12889) Updating files: 98% (12632/12889) Updating files: 99% (12761/12889) Updating files: 100% (12889/12889) Updating files: 100% (12889/12889), done. 2024-09-16T05:35:57.339 INFO:tasks.workunit.client.0.smithi063.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr: git switch -c 2024-09-16T05:35:57.340 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:57.341 INFO:tasks.workunit.client.0.smithi063.stderr:Or undo this operation with: 2024-09-16T05:35:57.341 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:57.341 INFO:tasks.workunit.client.0.smithi063.stderr: git switch - 2024-09-16T05:35:57.341 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:57.341 INFO:tasks.workunit.client.0.smithi063.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T05:35:57.341 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:57.341 INFO:tasks.workunit.client.0.smithi063.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-16T05:35:57.346 DEBUG:teuthology.orchestra.run.smithi063:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T05:35:57.352 INFO:tasks.workunit.client.0.smithi063.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T05:35:57.355 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T05:35:57.355 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T05:35:58.162 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T05:35:58.243 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T05:35:58.297 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T05:35:58.300 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T05:35:58.300 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T05:35:58.346 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T05:35:58.350 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-16T05:35:58.350 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T05:35:58.397 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2024-09-16T05:35:58.398 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2024-09-16T05:35:58.398 DEBUG:teuthology.orchestra.run.smithi063:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-09-16T05:35:58.449 INFO:tasks.workunit.client.0.smithi063.stderr:+ TEST_POOL=testpool1234 2024-09-16T05:35:58.449 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2024-09-16T05:35:58.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.569+0000 7f20bc923640 10 monclient: get_monmap_and_config 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.569+0000 7f20bc923640 10 monclient: build_initial_monmap 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.569+0000 7f20bc923640 10 monclient: monmap: 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:35:58.575551+0000 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:35:58.575551+0000 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:58.575 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:58.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.569+0000 7f20bc923640 1 Processor -- start 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.569+0000 7f20bc923640 1 -- start start 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.569+0000 7f20bc923640 10 monclient: init 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient(hunting): picked mon.noname-b con 0x7f20b80a28c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient(hunting): picked mon.noname-c con 0x7f20b80a2ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient(hunting): picked mon.noname-a con 0x7f20b813ccb0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b8074200 con 0x7f20b813ccb0 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b809f920 con 0x7f20b80a2ca0 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b80a05e0 con 0x7f20b80a28c0 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:58.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient(hunting): authenticate will time out at 1015.352599s 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b5d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f20b80a2ca0 legacy=0x7f20b80a30b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b6d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f20b813ccb0 0x7f20b813f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42366/0 (socket says 172.21.15.63:42366) 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f20b80a28c0 0x7f20b80a0ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53368/0 (socket says 172.21.15.63:53368) 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b6d9d640 1 -- 172.21.15.63:0/4066179499 learned_addr learned my addr 172.21.15.63:0/4066179499 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2826585949 0 0) 0x7f20b8074200 con 0x7f20b813ccb0 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient(hunting): _negotiate 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient(hunting): my global_id is 4467 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20a4003530 con 0x7f20b813ccb0 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365941140 0 0) 0x7f20b80a05e0 con 0x7f20b80a28c0 2024-09-16T05:35:58.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient(hunting): _negotiate 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient(hunting): my global_id is 4466 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20b8074200 con 0x7f20b80a28c0 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3498554558 0 0) 0x7f20b8074200 con 0x7f20b80a28c0 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 >> v1:172.21.15.63:6789/0 conn(0x7f20b80a2ca0 legacy=0x7f20b80a30b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 >> v1:172.21.15.6:6789/0 conn(0x7f20b813ccb0 legacy=0x7f20b813f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: _finish_hunting 0 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 monclient: found mon.noname-b 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:58.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b809ed70 con 0x7f20b80a28c0 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: _finish_auth 0 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: _check_auth_tickets 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20a8001980 con 0x7f20b80a28c0 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: dump: 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.582 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 5 monclient: authenticate success, global_id 4466 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20a8001c70 con 0x7f20b80a28c0 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 1 -- 172.21.15.63:0/4066179499 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20a8002080 con 0x7f20b80a28c0 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20b559a640 10 monclient: dump: 2024-09-16T05:35:58.583 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 4 set_mon_vals no callback set 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient: shutdown 2024-09-16T05:35:58.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- 172.21.15.63:0/4066179499 >> v1:172.21.15.6:6790/0 conn(0x7f20b80a28c0 legacy=0x7f20b80a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- 172.21.15.63:0/4066179499 shutdown_connections 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- 172.21.15.63:0/4066179499 >> 172.21.15.63:0/4066179499 conn(0x7f20b80a3fb0 msgr2=0x7f20b8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- 172.21.15.63:0/4066179499 shutdown_connections 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- 172.21.15.63:0/4066179499 wait complete. 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient: build_initial_monmap 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient: monmap: 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:35:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:58.586 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:58.586 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:58.586 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:58.586 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:58.586 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 Processor -- start 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 1 -- start start 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.573+0000 7f20bc923640 10 monclient: init 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient(hunting): picked mon.noname-b con 0x7f20b80a28c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient(hunting): picked mon.noname-a con 0x7f20b80a2ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient(hunting): picked mon.noname-c con 0x7f20b813ccb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b806f640 con 0x7f20b80a2ca0 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b806e340 con 0x7f20b80a28c0 2024-09-16T05:35:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20b659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f20b80a28c0 legacy=0x7f20b8071e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b809e110 con 0x7f20b813ccb0 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient(hunting): authenticate will time out at 1015.356443s 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20b5d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f20b80a2ca0 0x7f20b81d0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42370/0 (socket says 172.21.15.63:42370) 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20b5d9b640 1 -- 172.21.15.63:0/4251915625 learned_addr learned my addr 172.21.15.63:0/4251915625 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3562126921 0 0) 0x7f20b809e110 con 0x7f20b813ccb0 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): my global_id is 4469 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2098003530 con 0x7f20b813ccb0 2024-09-16T05:35:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531218627 0 0) 0x7f20b806f640 con 0x7f20b80a2ca0 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): my global_id is 4470 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20b809e110 con 0x7f20b80a2ca0 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3944861961 0 0) 0x7f2098003530 con 0x7f20b813ccb0 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20b806f640 con 0x7f20b813ccb0 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20b0001140 con 0x7f20b813ccb0 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): dump: 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:58.593 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1395578878 0 0) 0x7f20b809e110 con 0x7f20b80a2ca0 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2098003530 con 0x7f20b80a2ca0 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20a00040e0 con 0x7f20b80a2ca0 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient(hunting): dump: 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 973139381 0 0) 0x7f20b806f640 con 0x7f20b813ccb0 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 >> v1:172.21.15.63:6789/0 conn(0x7f20b80a28c0 legacy=0x7f20b8071e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 >> v1:172.21.15.6:6789/0 conn(0x7f20b80a2ca0 legacy=0x7f20b81d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 monclient: found mon.c 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b813c9f0 con 0x7f20b813ccb0 2024-09-16T05:35:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient: _finish_auth 0 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 5 monclient: authenticate success, global_id 4469 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient: _renew_subs 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20b806f080 con 0x7f20b813ccb0 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient: _renew_subs 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f20b8072570 con 0x7f20b813ccb0 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20b00047f0 con 0x7f20b813ccb0 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20b0004b40 con 0x7f20b813ccb0 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20b4d99640 4 set_mon_vals no callback set 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 10 monclient: dump: 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f209c005090 con 0x7f20b813ccb0 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f20b000f8e0 con 0x7f20b813ccb0 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.577+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 1170774142 0 0) 0x7f20b0043f60 con 0x7f20b813ccb0 2024-09-16T05:35:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.581+0000 7f20b659c640 10 monclient: get_auth_request con 0x7f2098036c60 auth_method 0 2024-09-16T05:35:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.593+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f20b0015210 con 0x7f20b813ccb0 2024-09-16T05:35:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.593+0000 7f20aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.593+0000 7f20aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:58.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.757+0000 7f20bc923640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T05:35:58.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.757+0000 7f20bc923640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T05:35:58.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.757+0000 7f20bc923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:58.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.757+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) -- 0x7f209c005380 con 0x7f20b813ccb0 2024-09-16T05:35:58.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.857+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' created v21) ==== 152+0+0 (unknown 1375261595 0 0) 0x7f20b0014b10 con 0x7f20b813ccb0 2024-09-16T05:35:58.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.857+0000 7f20aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T05:35:58.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.857+0000 7f20aeffd640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' created 2024-09-16T05:35:58.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.957+0000 7f20bc923640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T05:35:58.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.957+0000 7f20bc923640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T05:35:58.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.957+0000 7f20bc923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:58.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.957+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) -- 0x7f209c001180 con 0x7f20b813ccb0 2024-09-16T05:35:58.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20aeffd640 1 -- 172.21.15.63:0/4251915625 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' already exists v21) ==== 159+0+0 (unknown 1305974967 0 0) 0x7f209c001180 con 0x7f20b813ccb0 2024-09-16T05:35:58.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20aeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T05:35:58.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20aeffd640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' already exists 2024-09-16T05:35:58.965 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'testpool1234' already exists 2024-09-16T05:35:58.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2098036c60 legacy=0x7f2098039120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:58.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20bc923640 10 monclient: shutdown 2024-09-16T05:35:58.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20bc923640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:58.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 >> v1:172.21.15.6:6790/0 conn(0x7f20b813ccb0 legacy=0x7f20b81d3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:58.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 shutdown_connections 2024-09-16T05:35:58.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 >> 172.21.15.63:0/4251915625 conn(0x7f20b80a3fb0 msgr2=0x7f20b80a4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:58.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 shutdown_connections 2024-09-16T05:35:58.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:58.961+0000 7f20bc923640 1 -- 172.21.15.63:0/4251915625 wait complete. 2024-09-16T05:35:58.982 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient: get_monmap_and_config 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient: build_initial_monmap 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient: monmap: 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:35:59.109851+0000 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:35:59.109851+0000 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:59.109 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:59.110 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:59.110 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:35:59.110 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:35:59.110 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:59.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 1 Processor -- start 2024-09-16T05:35:59.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 1 -- start start 2024-09-16T05:35:59.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient: init 2024-09-16T05:35:59.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient(hunting): picked mon.noname-b con 0x7f12dc0a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient(hunting): picked mon.noname-a con 0x7f12dc0a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient(hunting): picked mon.noname-c con 0x7f12dc0a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12dc0a53b0 con 0x7f12dc0a07a0 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12dc0a0290 con 0x7f12dc0a4600 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12dc0a0dc0 con 0x7f12dc0a3b10 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12e0e43640 10 monclient(hunting): authenticate will time out at 1015.887642s 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.105+0000 7f12dad9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f12dc0a4600 legacy=0x7f12dc13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12da59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f12dc0a3b10 0x7f12dc0a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53392/0 (socket says 172.21.15.63:53392) 2024-09-16T05:35:59.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12da59c640 1 -- 172.21.15.63:0/240500964 learned_addr learned my addr 172.21.15.63:0/240500964 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:35:59.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2704247568 0 0) 0x7f12dc0a53b0 con 0x7f12dc0a07a0 2024-09-16T05:35:59.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient(hunting): _negotiate 2024-09-16T05:35:59.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:59.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:59.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient(hunting): my global_id is 4473 2024-09-16T05:35:59.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12c4003530 con 0x7f12dc0a07a0 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1486406668 0 0) 0x7f12dc0a0dc0 con 0x7f12dc0a3b10 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient(hunting): _negotiate 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient(hunting): my global_id is 4472 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12dc0a53b0 con 0x7f12dc0a3b10 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2085532348 0 0) 0x7f12c4003530 con 0x7f12dc0a07a0 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 >> v1:172.21.15.6:6790/0 conn(0x7f12dc0a3b10 legacy=0x7f12dc0a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 >> v1:172.21.15.63:6789/0 conn(0x7f12dc0a4600 legacy=0x7f12dc13f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: _finish_hunting 0 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 monclient: found mon.noname-a 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:59.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12dc0a10a0 con 0x7f12dc0a07a0 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: _finish_auth 0 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: _check_auth_tickets 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12d0001550 con 0x7f12dc0a07a0 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: dump: 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.116 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 5 monclient: authenticate success, global_id 4473 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12d00017b0 con 0x7f12dc0a07a0 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 1 -- 172.21.15.63:0/240500964 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12d0001b40 con 0x7f12dc0a07a0 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:59.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12d959a640 10 monclient: dump: 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 10 monclient: get_monmap_and_config success 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 4 set_mon_vals no callback set 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 10 monclient: shutdown 2024-09-16T05:35:59.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:35:59.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 1 -- 172.21.15.63:0/240500964 >> v1:172.21.15.6:6789/0 conn(0x7f12dc0a07a0 legacy=0x7f12dc13be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:59.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 1 -- 172.21.15.63:0/240500964 shutdown_connections 2024-09-16T05:35:59.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 1 -- 172.21.15.63:0/240500964 >> 172.21.15.63:0/240500964 conn(0x7f12dc075470 msgr2=0x7f12dc075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:35:59.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 1 -- 172.21.15.63:0/240500964 shutdown_connections 2024-09-16T05:35:59.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 1 -- 172.21.15.63:0/240500964 wait complete. 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 10 monclient: build_initial_monmap 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 10 monclient: monmap: 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:59.123 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 1 Processor -- start 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 1 -- start start 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.109+0000 7f12e0e43640 10 monclient: init 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient: _reopen_session rank -1 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient(hunting): picked mon.noname-b con 0x7f12dc0a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient(hunting): picked mon.noname-a con 0x7f12dc0a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient(hunting): picked mon.noname-c con 0x7f12dc0a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12dc126a80 con 0x7f12dc0a3b10 2024-09-16T05:35:59.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12da59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f12dc0a07a0 legacy=0x7f12dc136090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12dc0a8510 con 0x7f12dc0a07a0 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12dc125480 con 0x7f12dc0a4600 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient(hunting): _renew_subs 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient(hunting): authenticate will time out at 1015.892657s 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12dad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f12dc0a4600 0x7f12dc1d3ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53394/0 (socket says 172.21.15.63:53394) 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12dad9d640 1 -- 172.21.15.63:0/1220988123 learned_addr learned my addr 172.21.15.63:0/1220988123 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107866287 0 0) 0x7f12dc125480 con 0x7f12dc0a4600 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): my global_id is 4475 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12bc003530 con 0x7f12dc0a4600 2024-09-16T05:35:59.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556792040 0 0) 0x7f12dc126a80 con 0x7f12dc0a3b10 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): _negotiate 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): my global_id is 4476 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12dc125480 con 0x7f12dc0a3b10 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3748686496 0 0) 0x7f12bc003530 con 0x7f12dc0a4600 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12dc126a80 con 0x7f12dc0a4600 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12cc004040 con 0x7f12dc0a4600 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): dump: 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:59.126 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2461414777 0 0) 0x7f12dc125480 con 0x7f12dc0a3b10 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12bc003530 con 0x7f12dc0a3b10 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12d0001510 con 0x7f12dc0a3b10 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:35:59.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient(hunting): dump: 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1300001482 0 0) 0x7f12dc126a80 con 0x7f12dc0a4600 2024-09-16T05:35:59.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 >> v1:172.21.15.63:6789/0 conn(0x7f12dc0a07a0 legacy=0x7f12dc136090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 >> v1:172.21.15.6:6789/0 conn(0x7f12dc0a3b10 legacy=0x7f12dc1d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient: _finish_hunting 0 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 monclient: found mon.c 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12dc0a0500 con 0x7f12dc0a4600 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient: _finish_auth 0 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient: _check_auth_tickets 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 5 monclient: authenticate success, global_id 4475 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient: _renew_subs 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:59.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12dc1267a0 con 0x7f12dc0a4600 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient: _renew_subs 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f12dc1cb130 con 0x7f12dc0a4600 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12cc0042a0 con 0x7f12dc0a4600 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12cc0045f0 con 0x7f12dc0a4600 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 10 monclient: dump: 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.130 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12d8d99640 4 set_mon_vals no callback set 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:59.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c0005090 con 0x7f12dc0a4600 2024-09-16T05:35:59.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.113+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f12cc004810 con 0x7f12dc0a4600 2024-09-16T05:35:59.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.117+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3857+0+0 (unknown 3398105686 0 0) 0x7f12cc044130 con 0x7f12dc0a4600 2024-09-16T05:35:59.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.117+0000 7f12da59c640 10 monclient: get_auth_request con 0x7f12bc0369b0 auth_method 0 2024-09-16T05:35:59.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.129+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f12cc015290 con 0x7f12dc0a4600 2024-09-16T05:35:59.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.129+0000 7f12caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:35:59.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.129+0000 7f12caffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:35:59.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.309+0000 7f12e0e43640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T05:35:59.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.309+0000 7f12e0e43640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T05:35:59.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.309+0000 7f12e0e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:59.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.309+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) -- 0x7f12c0005380 con 0x7f12dc0a4600 2024-09-16T05:35:59.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.861+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v22) ==== 143+0+0 (unknown 3538591414 0 0) 0x7f12cc014b90 con 0x7f12dc0a4600 2024-09-16T05:35:59.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.861+0000 7f12caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T05:35:59.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.861+0000 7f12caffd640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 2 2024-09-16T05:35:59.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.965+0000 7f12e0e43640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T05:35:59.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.965+0000 7f12e0e43640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T05:35:59.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.965+0000 7f12e0e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:35:59.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:35:59.965+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) -- 0x7f12c00010a0 con 0x7f12dc0a4600 2024-09-16T05:36:00.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.889+0000 7f12caffd640 1 -- 172.21.15.63:0/1220988123 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v23) ==== 143+0+0 (unknown 3973590169 0 0) 0x7f12cc018ee0 con 0x7f12dc0a4600 2024-09-16T05:36:00.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.889+0000 7f12caffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T05:36:00.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.889+0000 7f12caffd640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 2 2024-09-16T05:36:00.896 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 3 pg_num_min to 2 2024-09-16T05:36:00.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.893+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 >> v1:172.21.15.63:6812/3643207211 conn(0x7f12bc0369b0 legacy=0x7f12bc038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:00.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.893+0000 7f12e0e43640 10 monclient: shutdown 2024-09-16T05:36:00.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.893+0000 7f12e0e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:00.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.893+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 >> v1:172.21.15.6:6790/0 conn(0x7f12dc0a4600 legacy=0x7f12dc1d3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:00.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.893+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 shutdown_connections 2024-09-16T05:36:00.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.893+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 >> 172.21.15.63:0/1220988123 conn(0x7f12dc075470 msgr2=0x7f12dc0a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:00.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.893+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 shutdown_connections 2024-09-16T05:36:00.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:00.893+0000 7f12e0e43640 1 -- 172.21.15.63:0/1220988123 wait complete. 2024-09-16T05:36:00.922 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-16T05:36:00.923 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 2 2024-09-16T05:36:01.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.045+0000 7fda19084640 10 monclient: get_monmap_and_config 2024-09-16T05:36:01.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.045+0000 7fda19084640 10 monclient: build_initial_monmap 2024-09-16T05:36:01.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.045+0000 7fda19084640 10 monclient: monmap: 2024-09-16T05:36:01.050 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:01.050 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.051 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:01.051074+0000 2024-09-16T05:36:01.051 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:01.051074+0000 2024-09-16T05:36:01.051 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:01.051 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.051 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:01.051 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:01.051 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:01.051 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.045+0000 7fda19084640 1 Processor -- start 2024-09-16T05:36:01.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- start start 2024-09-16T05:36:01.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient: init 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient(hunting): picked mon.noname-a con 0x7fda1409ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient(hunting): picked mon.noname-b con 0x7fda1409b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient(hunting): picked mon.noname-c con 0x7fda14139fc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda140a14f0 con 0x7fda1409ac90 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1409c680 con 0x7fda14139fc0 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1409e930 con 0x7fda1409b780 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient(hunting): authenticate will time out at 1017.828875s 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1359e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fda14139fc0 legacy=0x7fda1413c3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:01.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda0bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fda1409b780 0x7fda141388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53406/0 (socket says 172.21.15.63:53406) 2024-09-16T05:36:01.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda0bfff640 1 -- 172.21.15.63:0/3009285597 learned_addr learned my addr 172.21.15.63:0/3009285597 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:01.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 854611804 0 0) 0x7fda1409e930 con 0x7fda1409b780 2024-09-16T05:36:01.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient(hunting): _negotiate 2024-09-16T05:36:01.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:01.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:01.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient(hunting): my global_id is 4478 2024-09-16T05:36:01.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda00003530 con 0x7fda1409b780 2024-09-16T05:36:01.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127506296 0 0) 0x7fda140a14f0 con 0x7fda1409ac90 2024-09-16T05:36:01.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient(hunting): _negotiate 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient(hunting): my global_id is 4479 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda1409e930 con 0x7fda1409ac90 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2061621270 0 0) 0x7fda00003530 con 0x7fda1409b780 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 >> v1:172.21.15.63:6789/0 conn(0x7fda14139fc0 legacy=0x7fda1413c3b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 >> v1:172.21.15.6:6789/0 conn(0x7fda1409ac90 legacy=0x7fda1409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: _finish_hunting 0 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 monclient: found mon.noname-b 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:01.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1409c400 con 0x7fda1409b780 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: _finish_auth 0 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: _check_auth_tickets 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 5 monclient: authenticate success, global_id 4478 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd9fc001620 con 0x7fda1409b780 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: dump: 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.058 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9fc0018d0 con 0x7fda1409b780 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 1 -- 172.21.15.63:0/3009285597 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd9fc001ce0 con 0x7fda1409b780 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:01.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda1259c640 10 monclient: dump: 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.061 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 4 set_mon_vals no callback set 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 10 monclient: shutdown 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- 172.21.15.63:0/3009285597 >> v1:172.21.15.6:6790/0 conn(0x7fda1409b780 legacy=0x7fda141388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- 172.21.15.63:0/3009285597 shutdown_connections 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- 172.21.15.63:0/3009285597 >> 172.21.15.63:0/3009285597 conn(0x7fda140a5290 msgr2=0x7fda140a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- 172.21.15.63:0/3009285597 shutdown_connections 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.049+0000 7fda19084640 1 -- 172.21.15.63:0/3009285597 wait complete. 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: build_initial_monmap 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: monmap: 2024-09-16T05:36:01.062 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 1 Processor -- start 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 1 -- start start 2024-09-16T05:36:01.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: init 2024-09-16T05:36:01.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:01.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:01.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient(hunting): picked mon.noname-a con 0x7fda1409ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:01.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient(hunting): picked mon.noname-b con 0x7fda1409b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:01.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient(hunting): picked mon.noname-c con 0x7fda14139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:01.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1411f8b0 con 0x7fda1409ac90 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1409c0f0 con 0x7fda1409b780 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fda1409b780 legacy=0x7fda141ce390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1411f5d0 con 0x7fda14139fc0 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient(hunting): authenticate will time out at 1017.833246s 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda12d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fda1409ac90 0x7fda141c93b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42416/0 (socket says 172.21.15.63:42416) 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda12d9d640 1 -- 172.21.15.63:0/3263280519 learned_addr learned my addr 172.21.15.63:0/3263280519 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790236296 0 0) 0x7fda1411f5d0 con 0x7fda14139fc0 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:01.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): my global_id is 4481 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9f4003530 con 0x7fda14139fc0 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3650822928 0 0) 0x7fda1411f8b0 con 0x7fda1409ac90 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): my global_id is 4482 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda1411f5d0 con 0x7fda1409ac90 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1121039151 0 0) 0x7fda1411f5d0 con 0x7fda1409ac90 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda1411f8b0 con 0x7fda1409ac90 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 440782849 0 0) 0x7fd9f4003530 con 0x7fda14139fc0 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda1411f5d0 con 0x7fda14139fc0 2024-09-16T05:36:01.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fda0c0042c0 con 0x7fda14139fc0 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fda04001870 con 0x7fda1409ac90 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.071 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 999844412 0 0) 0x7fda1411f8b0 con 0x7fda1409ac90 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 >> v1:172.21.15.6:6790/0 conn(0x7fda14139fc0 legacy=0x7fda141d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 >> v1:172.21.15.63:6789/0 conn(0x7fda1409b780 legacy=0x7fda141ce390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 monclient: found mon.a 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1411f2f0 con 0x7fda1409ac90 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:01.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 5 monclient: authenticate success, global_id 4482 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: _renew_subs 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda1411e0e0 con 0x7fda1409ac90 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: _renew_subs 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fda1411ceb0 con 0x7fda1409ac90 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda04003600 con 0x7fda1409ac90 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda11d9b640 4 set_mon_vals no callback set 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fda04003950 con 0x7fda1409ac90 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:01.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.053+0000 7fda0b7fe640 10 monclient: dump: 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.057+0000 7fda19084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:01.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.057+0000 7fda19084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:01.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.057+0000 7fda19084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:01.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.057+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f8005090 con 0x7fda1409ac90 2024-09-16T05:36:01.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.057+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fda04003b70 con 0x7fda1409ac90 2024-09-16T05:36:01.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.057+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 3873+0+0 (unknown 964039368 0 0) 0x7fda04044120 con 0x7fda1409ac90 2024-09-16T05:36:01.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.057+0000 7fda0bfff640 10 monclient: get_auth_request con 0x7fd9f4036a00 auth_method 0 2024-09-16T05:36:01.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.069+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fda04015270 con 0x7fda1409ac90 2024-09-16T05:36:01.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.069+0000 7fda0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:01.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.069+0000 7fda0b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:01.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.233+0000 7fda19084640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:01.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.233+0000 7fda19084640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:01.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.233+0000 7fda19084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:01.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.233+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) -- 0x7fd9f8005b90 con 0x7fda1409ac90 2024-09-16T05:36:01.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.237+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v23) ==== 105+0+14 (unknown 2937417259 0 3052767636) 0x7fda04018f20 con 0x7fda1409ac90 2024-09-16T05:36:01.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.237+0000 7fda0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:01.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.237+0000 7fda0b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:01.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.329+0000 7fda19084640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:01.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.329+0000 7fda19084640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:01.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.329+0000 7fda19084640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:01.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.329+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) -- 0x7fd9f8005610 con 0x7fda1409ac90 2024-09-16T05:36:01.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.329+0000 7fda0b7fe640 1 -- 172.21.15.63:0/3263280519 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v23) ==== 105+0+14 (unknown 2937417259 0 3052767636) 0x7fda0401d7e0 con 0x7fda1409ac90 2024-09-16T05:36:01.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.329+0000 7fda0b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:01.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.329+0000 7fda0b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:01.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.333+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd9f4036a00 legacy=0x7fd9f4038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:01.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.333+0000 7fda19084640 10 monclient: shutdown 2024-09-16T05:36:01.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.333+0000 7fda19084640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:01.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.333+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 >> v1:172.21.15.6:6789/0 conn(0x7fda1409ac90 legacy=0x7fda141c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:01.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.333+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 shutdown_connections 2024-09-16T05:36:01.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.333+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 >> 172.21.15.63:0/3263280519 conn(0x7fda140a5290 msgr2=0x7fda14071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:01.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.333+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 shutdown_connections 2024-09-16T05:36:01.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.333+0000 7fda19084640 1 -- 172.21.15.63:0/3263280519 wait complete. 2024-09-16T05:36:01.351 INFO:tasks.workunit.client.0.smithi063.stdout:pg_num_min: 2 2024-09-16T05:36:01.352 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2024-09-16T05:36:01.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.457+0000 7f63a56f0640 10 monclient: get_monmap_and_config 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.457+0000 7f63a56f0640 10 monclient: build_initial_monmap 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.457+0000 7f63a56f0640 10 monclient: monmap: 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:01.462569+0000 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:01.462569+0000 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:01.462 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:01.463 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:01.463 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.457+0000 7f63a56f0640 1 Processor -- start 2024-09-16T05:36:01.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.457+0000 7f63a56f0640 1 -- start start 2024-09-16T05:36:01.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.457+0000 7f63a56f0640 10 monclient: init 2024-09-16T05:36:01.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient(hunting): picked mon.noname-b con 0x7f63a00a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient(hunting): picked mon.noname-c con 0x7f63a00a4d90 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient(hunting): picked mon.noname-a con 0x7f63a00a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a51e0 con 0x7f63a00a4790 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a0d60 con 0x7f63a00a4d90 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a1040 con 0x7f63a00a3ca0 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient(hunting): authenticate will time out at 1018.240354s 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f6397fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f63a00a4d90 legacy=0x7f63a013bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f63a00a3ca0 0x7f63a00a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53420/0 (socket says 172.21.15.63:53420) 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f63a00a4790 0x7f63a013f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42420/0 (socket says 172.21.15.63:42420) 2024-09-16T05:36:01.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639effd640 1 -- 172.21.15.63:0/1974474347 learned_addr learned my addr 172.21.15.63:0/1974474347 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:01.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 130327974 0 0) 0x7f63a00a1040 con 0x7f63a00a3ca0 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient(hunting): my global_id is 4484 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f638c003530 con 0x7f63a00a3ca0 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276425804 0 0) 0x7f63a00a51e0 con 0x7f63a00a4790 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient(hunting): my global_id is 4485 2024-09-16T05:36:01.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63a00a1040 con 0x7f63a00a4790 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3776936815 0 0) 0x7f63a00a1040 con 0x7f63a00a4790 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 >> v1:172.21.15.6:6790/0 conn(0x7f63a00a3ca0 legacy=0x7f63a00a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 >> v1:172.21.15.63:6789/0 conn(0x7f63a00a4d90 legacy=0x7f63a013bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 monclient: found mon.noname-a 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a00a0810 con 0x7f63a00a4790 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:01.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6398001980 con 0x7f63a00a4790 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: dump: 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.470 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 5 monclient: authenticate success, global_id 4485 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6398001be0 con 0x7f63a00a4790 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 4 set_mon_vals no callback set 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 1 -- 172.21.15.63:0/1974474347 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6398001fb0 con 0x7f63a00a4790 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:01.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f639e7fc640 10 monclient: dump: 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient: shutdown 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:01.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 1 -- 172.21.15.63:0/1974474347 >> v1:172.21.15.6:6789/0 conn(0x7f63a00a4790 legacy=0x7f63a013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 1 -- 172.21.15.63:0/1974474347 shutdown_connections 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 1 -- 172.21.15.63:0/1974474347 >> 172.21.15.63:0/1974474347 conn(0x7f63a0075470 msgr2=0x7f63a0075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 1 -- 172.21.15.63:0/1974474347 shutdown_connections 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 1 -- 172.21.15.63:0/1974474347 wait complete. 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient: build_initial_monmap 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.461+0000 7f63a56f0640 10 monclient: monmap: 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:01.473 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:01.474 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:01.474 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 1 Processor -- start 2024-09-16T05:36:01.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 1 -- start start 2024-09-16T05:36:01.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: init 2024-09-16T05:36:01.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:01.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:01.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient(hunting): picked mon.noname-a con 0x7f63a00a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:01.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient(hunting): picked mon.noname-c con 0x7f63a00a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:01.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient(hunting): picked mon.noname-b con 0x7f63a00a4d90 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:01.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a83b0 con 0x7f63a00a3ca0 2024-09-16T05:36:01.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a0072a70 con 0x7f63a00a4d90 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a0138cc0 con 0x7f63a00a4790 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient(hunting): authenticate will time out at 1018.244874s 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f639f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f63a00a4d90 legacy=0x7f63a01d38f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f639effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f63a00a3ca0 0x7f63a013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42432/0 (socket says 172.21.15.63:42432) 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f639effd640 1 -- 172.21.15.63:0/4002795427 learned_addr learned my addr 172.21.15.63:0/4002795427 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932777594 0 0) 0x7f63a0138cc0 con 0x7f63a00a4790 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): my global_id is 4487 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6380003530 con 0x7f63a00a4790 2024-09-16T05:36:01.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3557738936 0 0) 0x7f6390001240 con 0x7f63a00a3ca0 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): my global_id is 4488 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63a0138cc0 con 0x7f63a00a3ca0 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2144724443 0 0) 0x7f6380003530 con 0x7f63a00a4790 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6390001240 con 0x7f63a00a4790 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f63880040d0 con 0x7f63a00a4790 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): dump: 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.480 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4075393311 0 0) 0x7f63a0138cc0 con 0x7f63a00a3ca0 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6380003530 con 0x7f63a00a3ca0 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6390001a40 con 0x7f63a00a3ca0 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:01.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient(hunting): dump: 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2804484283 0 0) 0x7f6390001240 con 0x7f63a00a4790 2024-09-16T05:36:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 >> v1:172.21.15.63:6789/0 conn(0x7f63a00a4d90 legacy=0x7f63a01d38f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 >> v1:172.21.15.6:6789/0 conn(0x7f63a00a3ca0 legacy=0x7f63a013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 monclient: found mon.c 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a0138fa0 con 0x7f63a00a4790 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 5 monclient: authenticate success, global_id 4487 2024-09-16T05:36:01.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: _renew_subs 2024-09-16T05:36:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63a0137450 con 0x7f63a00a4790 2024-09-16T05:36:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: _renew_subs 2024-09-16T05:36:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f63a013ff10 con 0x7f63a00a4790 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6388004270 con 0x7f63a00a4790 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f639dffb640 4 set_mon_vals no callback set 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6388004640 con 0x7f63a00a4790 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63977fe640 10 monclient: dump: 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.485 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.465+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6384005090 con 0x7f63a00a4790 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.469+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6388004860 con 0x7f63a00a4790 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.469+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 3873+0+0 (unknown 964039368 0 0) 0x7f6388044130 con 0x7f63a00a4790 2024-09-16T05:36:01.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.469+0000 7f639effd640 10 monclient: get_auth_request con 0x7f63800369e0 auth_method 0 2024-09-16T05:36:01.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.481+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6388018ed0 con 0x7f63a00a4790 2024-09-16T05:36:01.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.481+0000 7f63977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:01.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.481+0000 7f63977fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:01.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.641+0000 7f63a56f0640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T05:36:01.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.641+0000 7f63a56f0640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T05:36:01.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.641+0000 7f63a56f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:01.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.641+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) -- 0x7f6384005380 con 0x7f63a00a4790 2024-09-16T05:36:01.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.973+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v24) ==== 145+0+0 (unknown 2945105049 0 0) 0x7f638801d790 con 0x7f63a00a4790 2024-09-16T05:36:01.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.973+0000 7f63977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T05:36:01.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:01.973+0000 7f63977fe640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 33 2024-09-16T05:36:02.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.073+0000 7f63a56f0640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T05:36:02.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.073+0000 7f63a56f0640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T05:36:02.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.073+0000 7f63a56f0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:02.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.073+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) -- 0x7f63840010a0 con 0x7f63a00a4790 2024-09-16T05:36:02.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.977+0000 7f63977fe640 1 -- 172.21.15.63:0/4002795427 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v25) ==== 145+0+0 (unknown 1683384682 0 0) 0x7f63880400f0 con 0x7f63a00a4790 2024-09-16T05:36:02.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.977+0000 7f63977fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T05:36:02.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.977+0000 7f63977fe640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 33 2024-09-16T05:36:02.983 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 3 pg_num_max to 33 2024-09-16T05:36:02.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.981+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 >> v1:172.21.15.63:6812/3643207211 conn(0x7f63800369e0 legacy=0x7f6380038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:02.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.981+0000 7f63a56f0640 10 monclient: shutdown 2024-09-16T05:36:02.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.981+0000 7f63a56f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:02.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.981+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 >> v1:172.21.15.6:6790/0 conn(0x7f63a00a4790 legacy=0x7f63a01d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:02.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.981+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 shutdown_connections 2024-09-16T05:36:02.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.981+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 >> 172.21.15.63:0/4002795427 conn(0x7f63a0075470 msgr2=0x7f63a00a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:02.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.981+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 shutdown_connections 2024-09-16T05:36:02.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:02.981+0000 7f63a56f0640 1 -- 172.21.15.63:0/4002795427 wait complete. 2024-09-16T05:36:03.004 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-16T05:36:03.005 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 33 2024-09-16T05:36:03.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.125+0000 7f1464da8640 10 monclient: get_monmap_and_config 2024-09-16T05:36:03.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.125+0000 7f1464da8640 10 monclient: build_initial_monmap 2024-09-16T05:36:03.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.125+0000 7f1464da8640 10 monclient: monmap: 2024-09-16T05:36:03.131 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:03.131 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.132 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:03.132030+0000 2024-09-16T05:36:03.132 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:03.132030+0000 2024-09-16T05:36:03.132 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:03.132 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.132 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:03.132 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:03.132 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:03.132 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 Processor -- start 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- start start 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient: init 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient(hunting): picked mon.noname-a con 0x7f146009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient(hunting): picked mon.noname-b con 0x7f14600a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient(hunting): picked mon.noname-c con 0x7f146009b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1460074200 con 0x7f146009ac90 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f146009f900 con 0x7f146009b780 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f146009e870 con 0x7f14600a0280 2024-09-16T05:36:03.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient(hunting): authenticate will time out at 1019.909137s 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145ed9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f146009b780 legacy=0x7f146013cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145e59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f146009ac90 0x7f146009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42436/0 (socket says 172.21.15.63:42436) 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1457fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f14600a0280 0x7f14601398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53442/0 (socket says 172.21.15.63:53442) 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145e59c640 1 -- 172.21.15.63:0/3753889386 learned_addr learned my addr 172.21.15.63:0/3753889386 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418103276 0 0) 0x7f1460074200 con 0x7f146009ac90 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient(hunting): my global_id is 4491 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f144c003530 con 0x7f146009ac90 2024-09-16T05:36:03.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942475279 0 0) 0x7f146009e870 con 0x7f14600a0280 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient(hunting): my global_id is 4490 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1460074200 con 0x7f14600a0280 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 676675901 0 0) 0x7f144c003530 con 0x7f146009ac90 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 >> v1:172.21.15.6:6790/0 conn(0x7f14600a0280 legacy=0x7f14601398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 >> v1:172.21.15.63:6789/0 conn(0x7f146009b780 legacy=0x7f146013cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 monclient: found mon.noname-a 2024-09-16T05:36:03.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1460141730 con 0x7f146009ac90 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: _finish_auth 0 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 5 monclient: authenticate success, global_id 4491 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1450001620 con 0x7f146009ac90 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: dump: 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.138 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14500018d0 con 0x7f146009ac90 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 1 -- 172.21.15.63:0/3753889386 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1450001ca0 con 0x7f146009ac90 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 4 set_mon_vals no callback set 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f145dd9b640 10 monclient: dump: 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient: shutdown 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:03.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- 172.21.15.63:0/3753889386 >> v1:172.21.15.6:6789/0 conn(0x7f146009ac90 legacy=0x7f146009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- 172.21.15.63:0/3753889386 shutdown_connections 2024-09-16T05:36:03.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- 172.21.15.63:0/3753889386 >> 172.21.15.63:0/3753889386 conn(0x7f14600a5290 msgr2=0x7f14600a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:03.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- 172.21.15.63:0/3753889386 shutdown_connections 2024-09-16T05:36:03.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 1 -- 172.21.15.63:0/3753889386 wait complete. 2024-09-16T05:36:03.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient: build_initial_monmap 2024-09-16T05:36:03.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.129+0000 7f1464da8640 10 monclient: monmap: 2024-09-16T05:36:03.144 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:03.144 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:03.144 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 1 Processor -- start 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 1 -- start start 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: init 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient(hunting): picked mon.noname-c con 0x7f146009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:03.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient(hunting): picked mon.noname-b con 0x7f146009b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient(hunting): picked mon.noname-a con 0x7f14600a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f146009c0f0 con 0x7f14600a0280 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f146009e030 con 0x7f146009b780 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1460140310 con 0x7f146009ac90 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient(hunting): authenticate will time out at 1019.912669s 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1457fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f146009b780 legacy=0x7f146009cb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f145e59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f146009ac90 0x7f146009c480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53458/0 (socket says 172.21.15.63:53458) 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f145e59c640 1 -- 172.21.15.63:0/3658900250 learned_addr learned my addr 172.21.15.63:0/3658900250 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4190483140 0 0) 0x7f1460140310 con 0x7f146009ac90 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): my global_id is 4493 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1440003530 con 0x7f146009ac90 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108861679 0 0) 0x7f146009c0f0 con 0x7f14600a0280 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): my global_id is 4494 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1460140310 con 0x7f14600a0280 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 984161029 0 0) 0x7f1460140310 con 0x7f14600a0280 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f146009c0f0 con 0x7f14600a0280 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1458004340 con 0x7f14600a0280 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:03.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:03.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3635622177 0 0) 0x7f1440003530 con 0x7f146009ac90 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1460140310 con 0x7f146009ac90 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1450001c20 con 0x7f146009ac90 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.149 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3243337516 0 0) 0x7f146009c0f0 con 0x7f14600a0280 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 >> v1:172.21.15.6:6790/0 conn(0x7f146009ac90 legacy=0x7f146009c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 >> v1:172.21.15.63:6789/0 conn(0x7f146009b780 legacy=0x7f146009cb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 monclient: found mon.a 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146013fa70 con 0x7f14600a0280 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 5 monclient: authenticate success, global_id 4494 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: _renew_subs 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:03.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f146009d2a0 con 0x7f14600a0280 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: _renew_subs 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1460134380 con 0x7f14600a0280 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14580045f0 con 0x7f14600a0280 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14580049c0 con 0x7f14600a0280 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f145d59a640 4 set_mon_vals no callback set 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 10 monclient: dump: 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.151 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.152 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.152 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.152 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.152 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1444005090 con 0x7f14600a0280 2024-09-16T05:36:03.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1458004be0 con 0x7f14600a0280 2024-09-16T05:36:03.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3889+0+0 (unknown 3418402422 0 0) 0x7f14580444e0 con 0x7f14600a0280 2024-09-16T05:36:03.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.133+0000 7f145e59c640 10 monclient: get_auth_request con 0x7f14400369d0 auth_method 0 2024-09-16T05:36:03.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.145+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f14580155c0 con 0x7f14600a0280 2024-09-16T05:36:03.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.145+0000 7f1456ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.145+0000 7f1456ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:03.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.309+0000 7f1464da8640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:03.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.309+0000 7f1464da8640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:03.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.309+0000 7f1464da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:03.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.309+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) -- 0x7f1444005b90 con 0x7f14600a0280 2024-09-16T05:36:03.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.313+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v25) ==== 105+0+15 (unknown 1401130430 0 2995009362) 0x7f1458019270 con 0x7f14600a0280 2024-09-16T05:36:03.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.313+0000 7f1456ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:03.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.313+0000 7f1456ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:03.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:03.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:03.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:03.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) -- 0x7f1444005610 con 0x7f14600a0280 2024-09-16T05:36:03.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1456ffd640 1 -- 172.21.15.63:0/3658900250 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v25) ==== 105+0+15 (unknown 1401130430 0 2995009362) 0x7f145801db30 con 0x7f14600a0280 2024-09-16T05:36:03.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1456ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:03.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1456ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:03.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 >> v1:172.21.15.63:6812/3643207211 conn(0x7f14400369d0 legacy=0x7f1440038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 10 monclient: shutdown 2024-09-16T05:36:03.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:03.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 >> v1:172.21.15.6:6789/0 conn(0x7f14600a0280 legacy=0x7f14601d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 shutdown_connections 2024-09-16T05:36:03.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 >> 172.21.15.63:0/3658900250 conn(0x7f14600a5290 msgr2=0x7f146006e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:03.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 shutdown_connections 2024-09-16T05:36:03.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.409+0000 7f1464da8640 1 -- 172.21.15.63:0/3658900250 wait complete. 2024-09-16T05:36:03.431 INFO:tasks.workunit.client.0.smithi063.stdout:pg_num_max: 33 2024-09-16T05:36:03.431 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2024-09-16T05:36:03.432 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:03.432 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient: get_monmap_and_config 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient: build_initial_monmap 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient: monmap: 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:03.557569+0000 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:03.557569+0000 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.557 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:03.558 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:03.558 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:03.558 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 1 Processor -- start 2024-09-16T05:36:03.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 1 -- start start 2024-09-16T05:36:03.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient: init 2024-09-16T05:36:03.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:03.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:03.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient(hunting): picked mon.noname-a con 0x7f2bb0134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient(hunting): picked mon.noname-c con 0x7f2bb006c240 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient(hunting): picked mon.noname-b con 0x7f2bb013a830 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb0135420 con 0x7f2bb0134930 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb009e090 con 0x7f2bb006c240 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb009dad0 con 0x7f2bb013a830 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2bb5bc7640 10 monclient(hunting): authenticate will time out at 1020.335345s 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2baeffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2bb006c240 legacy=0x7f2bb0134d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2baf7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2bb0134930 0x7f2bb009e790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42454/0 (socket says 172.21.15.63:42454) 2024-09-16T05:36:03.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.553+0000 7f2baf7fe640 1 -- 172.21.15.63:0/732609945 learned_addr learned my addr 172.21.15.63:0/732609945 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:03.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322998958 0 0) 0x7f2bb0135420 con 0x7f2bb0134930 2024-09-16T05:36:03.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient(hunting): my global_id is 4497 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b98003530 con 0x7f2bb0134930 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3103562506 0 0) 0x7f2bb009dad0 con 0x7f2bb013a830 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient(hunting): my global_id is 4496 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bb0135420 con 0x7f2bb013a830 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2141727105 0 0) 0x7f2bb0135420 con 0x7f2bb013a830 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 >> v1:172.21.15.63:6789/0 conn(0x7f2bb006c240 legacy=0x7f2bb0134d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 >> v1:172.21.15.6:6789/0 conn(0x7f2bb0134930 legacy=0x7f2bb009e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 monclient: found mon.noname-b 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb0075de0 con 0x7f2bb013a830 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ba8001980 con 0x7f2bb013a830 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 5 monclient: authenticate success, global_id 4496 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: dump: 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:03.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ba8001be0 con 0x7f2bb013a830 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 4 set_mon_vals no callback set 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 1 -- 172.21.15.63:0/732609945 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ba8001ff0 con 0x7f2bb013a830 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bae7fc640 10 monclient: dump: 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.567 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient: shutdown 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/732609945 >> v1:172.21.15.6:6790/0 conn(0x7f2bb013a830 legacy=0x7f2bb013cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/732609945 shutdown_connections 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/732609945 >> 172.21.15.63:0/732609945 conn(0x7f2bb00a2f90 msgr2=0x7f2bb0075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/732609945 shutdown_connections 2024-09-16T05:36:03.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/732609945 wait complete. 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient: build_initial_monmap 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient: monmap: 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:03.569 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 Processor -- start 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- start start 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient: init 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient(hunting): picked mon.noname-a con 0x7f2bb006c240 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient(hunting): picked mon.noname-b con 0x7f2bb0134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient(hunting): picked mon.noname-c con 0x7f2bb013a830 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb011d3b0 con 0x7f2bb006c240 2024-09-16T05:36:03.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb009b880 con 0x7f2bb0134930 2024-09-16T05:36:03.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb00a0930 con 0x7f2bb013a830 2024-09-16T05:36:03.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:03.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2bb5bc7640 10 monclient(hunting): authenticate will time out at 1020.339093s 2024-09-16T05:36:03.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2baeffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2bb0134930 legacy=0x7f2bb009b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:03.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2baf7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2bb006c240 0x7f2bb009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42468/0 (socket says 172.21.15.63:42468) 2024-09-16T05:36:03.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2baffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2bb013a830 0x7f2bb01d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53472/0 (socket says 172.21.15.63:53472) 2024-09-16T05:36:03.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2baf7fe640 1 -- 172.21.15.63:0/2102294903 learned_addr learned my addr 172.21.15.63:0/2102294903 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:03.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259772537 0 0) 0x7f2bb00a0930 con 0x7f2bb013a830 2024-09-16T05:36:03.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 10 monclient(hunting): my global_id is 4499 2024-09-16T05:36:03.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b9c003530 con 0x7f2bb013a830 2024-09-16T05:36:03.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075573765 0 0) 0x7f2bb011d3b0 con 0x7f2bb006c240 2024-09-16T05:36:03.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 10 monclient(hunting): my global_id is 4500 2024-09-16T05:36:03.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.557+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bb00a0930 con 0x7f2bb006c240 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3345662023 0 0) 0x7f2b9c003530 con 0x7f2bb013a830 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bb011d3b0 con 0x7f2bb013a830 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ba8001f30 con 0x7f2bb013a830 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient(hunting): dump: 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.576 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2732059035 0 0) 0x7f2bb00a0930 con 0x7f2bb006c240 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b9c003530 con 0x7f2bb006c240 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ba00018e0 con 0x7f2bb006c240 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient(hunting): dump: 2024-09-16T05:36:03.577 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4048409265 0 0) 0x7f2bb011d3b0 con 0x7f2bb013a830 2024-09-16T05:36:03.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 >> v1:172.21.15.63:6789/0 conn(0x7f2bb0134930 legacy=0x7f2bb009b170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 >> v1:172.21.15.6:6789/0 conn(0x7f2bb006c240 legacy=0x7f2bb009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 monclient: found mon.c 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb013df40 con 0x7f2bb013a830 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient: _finish_auth 0 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 5 monclient: authenticate success, global_id 4499 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 10 monclient: _renew_subs 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb011d0d0 con 0x7f2bb013a830 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 10 monclient: _renew_subs 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2bb0139fc0 con 0x7f2bb013a830 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ba8001be0 con 0x7f2bb013a830 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ba8003f80 con 0x7f2bb013a830 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2badffb640 4 set_mon_vals no callback set 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 10 monclient: dump: 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.580 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b90005090 con 0x7f2bb013a830 2024-09-16T05:36:03.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2ba80041a0 con 0x7f2bb013a830 2024-09-16T05:36:03.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3889+0+0 (unknown 3418402422 0 0) 0x7f2ba80445a0 con 0x7f2bb013a830 2024-09-16T05:36:03.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.561+0000 7f2baf7fe640 10 monclient: get_auth_request con 0x7f2b9c036a20 auth_method 0 2024-09-16T05:36:03.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.573+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2ba80157a0 con 0x7f2bb013a830 2024-09-16T05:36:03.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.573+0000 7f2b97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.573+0000 7f2b97fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:03.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.745+0000 7f2bb5bc7640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T05:36:03.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.745+0000 7f2bb5bc7640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T05:36:03.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.745+0000 7f2bb5bc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.745+0000 7f2bb5bc7640 1 -- 172.21.15.63:0/2102294903 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"} v 0) -- 0x7f2b90005380 con 0x7f2bb013a830 2024-09-16T05:36:03.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.749+0000 7f2b97fff640 1 -- 172.21.15.63:0/2102294903 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}]=-22 specified pg_num_min 9 > pg_num 8 v25) ==== 150+0+0 (unknown 4273668204 0 0) 0x7f2ba801dbf0 con 0x7f2bb013a830 2024-09-16T05:36:03.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.749+0000 7f2b97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T05:36:03.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.749+0000 7f2b97fff640 10 monclient: _finish_command 2 = mon:22 specified pg_num_min 9 > pg_num 8 2024-09-16T05:36:03.753 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2024-09-16T05:36:03.760 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:03.760 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2024-09-16T05:36:03.760 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:03.760 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2024-09-16T05:36:03.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient: get_monmap_and_config 2024-09-16T05:36:03.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient: build_initial_monmap 2024-09-16T05:36:03.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient: monmap: 2024-09-16T05:36:03.878 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:03.878 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.878 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:03.878224+0000 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:03.878224+0000 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 1 Processor -- start 2024-09-16T05:36:03.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 1 -- start start 2024-09-16T05:36:03.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient: init 2024-09-16T05:36:03.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:03.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:03.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient(hunting): picked mon.noname-b con 0x7f6e501339b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:03.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient(hunting): picked mon.noname-a con 0x7f6e5009fc50 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:03.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient(hunting): picked mon.noname-c con 0x7f6e5013d140 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:03.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e500a0710 con 0x7f6e5009fc50 2024-09-16T05:36:03.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e500a09f0 con 0x7f6e5013d140 2024-09-16T05:36:03.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e500a0cd0 con 0x7f6e501339b0 2024-09-16T05:36:03.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:03.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e56546640 10 monclient(hunting): authenticate will time out at 1020.655306s 2024-09-16T05:36:03.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e54ae3640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6e5013d140 legacy=0x7f6e5013f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:03.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e4ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6e501339b0 0x7f6e50133d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53482/0 (socket says 172.21.15.63:53482) 2024-09-16T05:36:03.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.873+0000 7f6e4ffff640 1 -- 172.21.15.63:0/3068915679 learned_addr learned my addr 172.21.15.63:0/3068915679 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:03.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2976500908 0 0) 0x7f6e500a0cd0 con 0x7f6e501339b0 2024-09-16T05:36:03.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient(hunting): my global_id is 4502 2024-09-16T05:36:03.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e38003530 con 0x7f6e501339b0 2024-09-16T05:36:03.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2139922735 0 0) 0x7f6e500a0710 con 0x7f6e5009fc50 2024-09-16T05:36:03.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient(hunting): my global_id is 4503 2024-09-16T05:36:03.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e500a0cd0 con 0x7f6e5009fc50 2024-09-16T05:36:03.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 163757490 0 0) 0x7f6e38003530 con 0x7f6e501339b0 2024-09-16T05:36:03.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 >> v1:172.21.15.63:6789/0 conn(0x7f6e5013d140 legacy=0x7f6e5013f530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:03.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 >> v1:172.21.15.6:6789/0 conn(0x7f6e5009fc50 legacy=0x7f6e5013ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:03.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 monclient: found mon.noname-b 2024-09-16T05:36:03.893 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:03.893 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.893 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e500a15b0 con 0x7f6e501339b0 2024-09-16T05:36:03.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: _finish_auth 0 2024-09-16T05:36:03.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:03.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:03.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e40001550 con 0x7f6e501339b0 2024-09-16T05:36:03.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:03.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: dump: 2024-09-16T05:36:03.895 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.895 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.895 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.895 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.895 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.896 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.896 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.896 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.896 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.896 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 5 monclient: authenticate success, global_id 4502 2024-09-16T05:36:03.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:03.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e400017b0 con 0x7f6e501339b0 2024-09-16T05:36:03.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:03.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 1 -- 172.21.15.63:0/3068915679 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e40001b80 con 0x7f6e501339b0 2024-09-16T05:36:03.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:03.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 4 set_mon_vals no callback set 2024-09-16T05:36:03.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:03.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4effd640 10 monclient: dump: 2024-09-16T05:36:03.898 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.898 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.898 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.898 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.899 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.899 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.899 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.899 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.899 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.899 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient: shutdown 2024-09-16T05:36:03.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:03.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- 172.21.15.63:0/3068915679 >> v1:172.21.15.6:6790/0 conn(0x7f6e501339b0 legacy=0x7f6e50133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- 172.21.15.63:0/3068915679 shutdown_connections 2024-09-16T05:36:03.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- 172.21.15.63:0/3068915679 >> 172.21.15.63:0/3068915679 conn(0x7f6e500a1e90 msgr2=0x7f6e50075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:03.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- 172.21.15.63:0/3068915679 shutdown_connections 2024-09-16T05:36:03.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- 172.21.15.63:0/3068915679 wait complete. 2024-09-16T05:36:03.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient: build_initial_monmap 2024-09-16T05:36:03.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient: monmap: 2024-09-16T05:36:03.901 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:03.901 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:03.901 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:03.901 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:03.902 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:03.902 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.902 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:03.902 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:03.902 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:03.902 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 Processor -- start 2024-09-16T05:36:03.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- start start 2024-09-16T05:36:03.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient: init 2024-09-16T05:36:03.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:03.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:03.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient(hunting): picked mon.noname-a con 0x7f6e5009fc50 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:03.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5013d140 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:03.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient(hunting): picked mon.noname-c con 0x7f6e501330b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:03.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e50133490 con 0x7f6e5009fc50 2024-09-16T05:36:03.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e5012f630 con 0x7f6e5013d140 2024-09-16T05:36:03.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e5013f5c0 con 0x7f6e501330b0 2024-09-16T05:36:03.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:03.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e56546640 10 monclient(hunting): authenticate will time out at 1020.659363s 2024-09-16T05:36:03.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6e5013d140 legacy=0x7f6e5013eeb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:03.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6e5009fc50 0x7f6e5013e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42490/0 (socket says 172.21.15.63:42490) 2024-09-16T05:36:03.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e4ffff640 1 -- 172.21.15.63:0/2953523583 learned_addr learned my addr 172.21.15.63:0/2953523583 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:03.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.877+0000 7f6e54ae3640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6e501330b0 0x7f6e501d3c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53494/0 (socket says 172.21.15.63:53494) 2024-09-16T05:36:03.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254389210 0 0) 0x7f6e5013f5c0 con 0x7f6e501330b0 2024-09-16T05:36:03.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): my global_id is 4505 2024-09-16T05:36:03.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e3c003530 con 0x7f6e501330b0 2024-09-16T05:36:03.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 756064589 0 0) 0x7f6e50133490 con 0x7f6e5009fc50 2024-09-16T05:36:03.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:03.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:03.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:03.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): my global_id is 4506 2024-09-16T05:36:03.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e5013f5c0 con 0x7f6e5009fc50 2024-09-16T05:36:03.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2967477168 0 0) 0x7f6e3c003530 con 0x7f6e501330b0 2024-09-16T05:36:03.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e50133490 con 0x7f6e501330b0 2024-09-16T05:36:03.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e480043d0 con 0x7f6e501330b0 2024-09-16T05:36:03.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:03.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:03.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): dump: 2024-09-16T05:36:03.908 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.908 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.908 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.908 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2445019279 0 0) 0x7f6e5013f5c0 con 0x7f6e5009fc50 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e3c003530 con 0x7f6e5009fc50 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e40001450 con 0x7f6e5009fc50 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:03.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient(hunting): dump: 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4215502021 0 0) 0x7f6e50133490 con 0x7f6e501330b0 2024-09-16T05:36:03.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 >> v1:172.21.15.63:6789/0 conn(0x7f6e5013d140 legacy=0x7f6e5013eeb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 >> v1:172.21.15.6:6789/0 conn(0x7f6e5009fc50 legacy=0x7f6e5013e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 monclient: found mon.c 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5012f910 con 0x7f6e501330b0 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 5 monclient: authenticate success, global_id 4505 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 10 monclient: _renew_subs 2024-09-16T05:36:03.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e5009c0f0 con 0x7f6e501330b0 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 10 monclient: _renew_subs 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e50140920 con 0x7f6e501330b0 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e48004630 con 0x7f6e501330b0 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e48004980 con 0x7f6e501330b0 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4e7fc640 4 set_mon_vals no callback set 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 10 monclient: dump: 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:03.912 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:03.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e56546640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e30005090 con 0x7f6e501330b0 2024-09-16T05:36:03.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6e48004b20 con 0x7f6e501330b0 2024-09-16T05:36:03.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3889+0+0 (unknown 3418402422 0 0) 0x7f6e480443d0 con 0x7f6e501330b0 2024-09-16T05:36:03.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.881+0000 7f6e4ffff640 10 monclient: get_auth_request con 0x7f6e3c036aa0 auth_method 0 2024-09-16T05:36:03.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.893+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6e48015510 con 0x7f6e501330b0 2024-09-16T05:36:03.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.893+0000 7f6e4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:03.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:03.893+0000 7f6e4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:04.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.065+0000 7f6e56546640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T05:36:04.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.065+0000 7f6e56546640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T05:36:04.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.065+0000 7f6e56546640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:04.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.065+0000 7f6e56546640 1 -- 172.21.15.63:0/2953523583 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"} v 0) -- 0x7f6e30005380 con 0x7f6e501330b0 2024-09-16T05:36:04.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.065+0000 7f6e4cff9640 1 -- 172.21.15.63:0/2953523583 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}]=-22 specified pg_num_max 7 < pg_num 8 v25) ==== 150+0+0 (unknown 4260544524 0 0) 0x7f6e48014e10 con 0x7f6e501330b0 2024-09-16T05:36:04.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.065+0000 7f6e4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T05:36:04.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.065+0000 7f6e4cff9640 10 monclient: _finish_command 2 = mon:22 specified pg_num_max 7 < pg_num 8 2024-09-16T05:36:04.071 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2024-09-16T05:36:04.078 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:04.078 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2024-09-16T05:36:04.079 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:04.079 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set testpool1234 pg_num 1 2024-09-16T05:36:04.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient: get_monmap_and_config 2024-09-16T05:36:04.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient: build_initial_monmap 2024-09-16T05:36:04.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient: monmap: 2024-09-16T05:36:04.202 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:04.202 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.202 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:04.202794+0000 2024-09-16T05:36:04.202 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:04.202794+0000 2024-09-16T05:36:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:04.203 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 1 Processor -- start 2024-09-16T05:36:04.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 1 -- start start 2024-09-16T05:36:04.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient: init 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient(hunting): picked mon.noname-c con 0x7f9bc8134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc80a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient(hunting): picked mon.noname-b con 0x7f9bc806dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bc8135420 con 0x7f9bc80a0810 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bc809e830 con 0x7f9bc8134930 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bc7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9bc8134930 legacy=0x7f9bc8134d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bc80a0dc0 con 0x7f9bc806dfa0 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.197+0000 7f9bce466640 10 monclient(hunting): authenticate will time out at 1020.979794s 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bcca03640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9bc806dfa0 0x7f9bc813cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53510/0 (socket says 172.21.15.63:53510) 2024-09-16T05:36:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bcca03640 1 -- 172.21.15.63:0/54263633 learned_addr learned my addr 172.21.15.63:0/54263633 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc77fe640 1 --1- 172.21.15.63:0/54263633 >> v1:172.21.15.6:6789/0 conn(0x7f9bc80a0810 0x7f9bc8139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42494/0 (socket says 172.21.15.63:42494) 2024-09-16T05:36:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197742716 0 0) 0x7f9bc8135420 con 0x7f9bc80a0810 2024-09-16T05:36:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient(hunting): my global_id is 4509 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bb0003530 con 0x7f9bc80a0810 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565254667 0 0) 0x7f9bc80a0dc0 con 0x7f9bc806dfa0 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient(hunting): my global_id is 4508 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bc8135420 con 0x7f9bc806dfa0 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4185188455 0 0) 0x7f9bb0003530 con 0x7f9bc80a0810 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 >> v1:172.21.15.6:6790/0 conn(0x7f9bc806dfa0 legacy=0x7f9bc813cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 >> v1:172.21.15.63:6789/0 conn(0x7f9bc8134930 legacy=0x7f9bc8134d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 monclient: found mon.noname-a 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:04.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc80a2500 con 0x7f9bc80a0810 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9bbc001620 con 0x7f9bc80a0810 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 5 monclient: authenticate success, global_id 4509 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: dump: 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.210 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9bbc001880 con 0x7f9bc80a0810 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 1 -- 172.21.15.63:0/54263633 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9bbc001c90 con 0x7f9bc80a0810 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 4 set_mon_vals no callback set 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:04.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bc6ffd640 10 monclient: dump: 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 10 monclient: shutdown 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 1 -- 172.21.15.63:0/54263633 >> v1:172.21.15.6:6789/0 conn(0x7f9bc80a0810 legacy=0x7f9bc8139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 1 -- 172.21.15.63:0/54263633 shutdown_connections 2024-09-16T05:36:04.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 1 -- 172.21.15.63:0/54263633 >> 172.21.15.63:0/54263633 conn(0x7f9bc80a1e90 msgr2=0x7f9bc8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 1 -- 172.21.15.63:0/54263633 shutdown_connections 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 1 -- 172.21.15.63:0/54263633 wait complete. 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 10 monclient: build_initial_monmap 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 10 monclient: monmap: 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:04.213 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 1 Processor -- start 2024-09-16T05:36:04.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 1 -- start start 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.201+0000 7f9bce466640 10 monclient: init 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient(hunting): picked mon.noname-c con 0x7f9bc806dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient(hunting): picked mon.noname-a con 0x7f9bc80a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient(hunting): picked mon.noname-b con 0x7f9bc8134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bc8071d60 con 0x7f9bc80a0810 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bc813e350 con 0x7f9bc8134930 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bc8139e10 con 0x7f9bc806dfa0 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bcca03640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9bc8134930 legacy=0x7f9bc81d3780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:04.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:04.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient(hunting): authenticate will time out at 1020.984169s 2024-09-16T05:36:04.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9bc80a0810 0x7f9bc809c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42498/0 (socket says 172.21.15.63:42498) 2024-09-16T05:36:04.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc77fe640 1 -- 172.21.15.63:0/2305471561 learned_addr learned my addr 172.21.15.63:0/2305471561 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:04.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 908467221 0 0) 0x7f9bc8139e10 con 0x7f9bc806dfa0 2024-09-16T05:36:04.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): my global_id is 4511 2024-09-16T05:36:04.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bb4003530 con 0x7f9bc806dfa0 2024-09-16T05:36:04.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743212365 0 0) 0x7f9bc80a2500 con 0x7f9bc80a0810 2024-09-16T05:36:04.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): my global_id is 4512 2024-09-16T05:36:04.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bc8139e10 con 0x7f9bc80a0810 2024-09-16T05:36:04.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 939954579 0 0) 0x7f9bc8139e10 con 0x7f9bc80a0810 2024-09-16T05:36:04.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9bc80a2500 con 0x7f9bc80a0810 2024-09-16T05:36:04.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9bbc001c10 con 0x7f9bc80a0810 2024-09-16T05:36:04.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:04.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:04.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:04.222 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.222 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.222 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.222 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.223 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.223 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.223 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.223 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.223 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.224 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:04.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2096306506 0 0) 0x7f9bb4003530 con 0x7f9bc806dfa0 2024-09-16T05:36:04.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9bc8139e10 con 0x7f9bc806dfa0 2024-09-16T05:36:04.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9bb80040d0 con 0x7f9bc806dfa0 2024-09-16T05:36:04.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:04.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:04.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:04.225 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.225 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.225 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.225 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.226 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.226 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.226 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.227 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.227 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.227 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3661257383 0 0) 0x7f9bc80a2500 con 0x7f9bc80a0810 2024-09-16T05:36:04.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 >> v1:172.21.15.6:6790/0 conn(0x7f9bc806dfa0 legacy=0x7f9bc809c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 >> v1:172.21.15.63:6789/0 conn(0x7f9bc8134930 legacy=0x7f9bc81d3780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:04.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:04.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 monclient: found mon.a 2024-09-16T05:36:04.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:04.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc813a0f0 con 0x7f9bc80a0810 2024-09-16T05:36:04.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:04.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:04.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:04.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 5 monclient: authenticate success, global_id 4512 2024-09-16T05:36:04.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient: _renew_subs 2024-09-16T05:36:04.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc813a3d0 con 0x7f9bc80a0810 2024-09-16T05:36:04.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient: _renew_subs 2024-09-16T05:36:04.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9bc8133d20 con 0x7f9bc80a0810 2024-09-16T05:36:04.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9bbc001880 con 0x7f9bc80a0810 2024-09-16T05:36:04.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:04.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9bbc004ce0 con 0x7f9bc80a0810 2024-09-16T05:36:04.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:04.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc5ffb640 4 set_mon_vals no callback set 2024-09-16T05:36:04.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 10 monclient: dump: 2024-09-16T05:36:04.231 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.231 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.232 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.232 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.232 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.232 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.232 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.233 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.233 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.233 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bce466640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba8005090 con 0x7f9bc80a0810 2024-09-16T05:36:04.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9bbc004e80 con 0x7f9bc80a0810 2024-09-16T05:36:04.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3889+0+0 (unknown 3418402422 0 0) 0x7f9bbc044210 con 0x7f9bc80a0810 2024-09-16T05:36:04.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.205+0000 7f9bc7fff640 10 monclient: get_auth_request con 0x7f9bb4036ac0 auth_method 0 2024-09-16T05:36:04.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.217+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9bc8071d60 con 0x7f9bc80a0810 2024-09-16T05:36:04.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.217+0000 7f9bc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.217+0000 7f9bc4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:04.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.381+0000 7f9bce466640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T05:36:04.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.381+0000 7f9bce466640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T05:36:04.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.381+0000 7f9bce466640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.381+0000 7f9bce466640 1 -- 172.21.15.63:0/2305471561 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"} v 0) -- 0x7f9ba8005380 con 0x7f9bc80a0810 2024-09-16T05:36:04.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.385+0000 7f9bc4ff9640 1 -- 172.21.15.63:0/2305471561 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}]=-22 specified pg_num 1 < pg_num_min 2 v25) ==== 146+0+0 (unknown 2160489427 0 0) 0x7f9bbc01d860 con 0x7f9bc80a0810 2024-09-16T05:36:04.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.385+0000 7f9bc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T05:36:04.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.385+0000 7f9bc4ff9640 10 monclient: _finish_command 2 = mon:22 specified pg_num 1 < pg_num_min 2 2024-09-16T05:36:04.389 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2024-09-16T05:36:04.393 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:04.393 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2024-09-16T05:36:04.393 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:04.393 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set testpool1234 pg_num 44 2024-09-16T05:36:04.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient: get_monmap_and_config 2024-09-16T05:36:04.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient: build_initial_monmap 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient: monmap: 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:04.521122+0000 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:04.521122+0000 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:04.521 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 1 Processor -- start 2024-09-16T05:36:04.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 1 -- start start 2024-09-16T05:36:04.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient: init 2024-09-16T05:36:04.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:04.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient(hunting): picked mon.noname-a con 0x7f1e7809c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient(hunting): picked mon.noname-c con 0x7f1e780a09c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient(hunting): picked mon.noname-b con 0x7f1e7809d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e780a0ed0 con 0x7f1e7809c300 2024-09-16T05:36:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7809d950 con 0x7f1e780a09c0 2024-09-16T05:36:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7809ebd0 con 0x7f1e7809d3a0 2024-09-16T05:36:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient(hunting): authenticate will time out at 1021.298069s 2024-09-16T05:36:04.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e6effd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f1e780a09c0 legacy=0x7f1e78138a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:04.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e77fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1e7809d3a0 0x7f1e7813c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46484/0 (socket says 172.21.15.63:46484) 2024-09-16T05:36:04.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e77fff640 1 -- 172.21.15.63:0/1137190749 learned_addr learned my addr 172.21.15.63:0/1137190749 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:04.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e777fe640 1 --1- 172.21.15.63:0/1137190749 >> v1:172.21.15.6:6789/0 conn(0x7f1e7809c300 0x7f1e7809c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52448/0 (socket says 172.21.15.63:52448) 2024-09-16T05:36:04.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292476921 0 0) 0x7f1e780a0ed0 con 0x7f1e7809c300 2024-09-16T05:36:04.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient(hunting): my global_id is 4515 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e64003530 con 0x7f1e7809c300 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2300189901 0 0) 0x7f1e7809ebd0 con 0x7f1e7809d3a0 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient(hunting): my global_id is 4514 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e780a0ed0 con 0x7f1e7809d3a0 2024-09-16T05:36:04.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1178329487 0 0) 0x7f1e64003530 con 0x7f1e7809c300 2024-09-16T05:36:04.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 >> v1:172.21.15.6:6790/0 conn(0x7f1e7809d3a0 legacy=0x7f1e7813c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 >> v1:172.21.15.63:6789/0 conn(0x7f1e780a09c0 legacy=0x7f1e78138a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:04.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:04.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 monclient: found mon.noname-a 2024-09-16T05:36:04.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:04.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e780a2470 con 0x7f1e7809c300 2024-09-16T05:36:04.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:04.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:04.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:04.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e68001620 con 0x7f1e7809c300 2024-09-16T05:36:04.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: dump: 2024-09-16T05:36:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.538 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 5 monclient: authenticate success, global_id 4515 2024-09-16T05:36:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1e68001940 con 0x7f1e7809c300 2024-09-16T05:36:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:04.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 1 -- 172.21.15.63:0/1137190749 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e68001d10 con 0x7f1e7809c300 2024-09-16T05:36:04.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:04.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e76ffd640 10 monclient: dump: 2024-09-16T05:36:04.539 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.539 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.539 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.539 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.517+0000 7f1e7daf2640 4 set_mon_vals no callback set 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient: shutdown 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:04.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- 172.21.15.63:0/1137190749 >> v1:172.21.15.6:6789/0 conn(0x7f1e7809c300 legacy=0x7f1e7809c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- 172.21.15.63:0/1137190749 shutdown_connections 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- 172.21.15.63:0/1137190749 >> 172.21.15.63:0/1137190749 conn(0x7f1e780a1e90 msgr2=0x7f1e78075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- 172.21.15.63:0/1137190749 shutdown_connections 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- 172.21.15.63:0/1137190749 wait complete. 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient: build_initial_monmap 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient: monmap: 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.541 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:04.542 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:04.542 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:04.542 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 Processor -- start 2024-09-16T05:36:04.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- start start 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient: init 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient(hunting): picked mon.noname-c con 0x7f1e7809c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient(hunting): picked mon.noname-b con 0x7f1e7809d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient(hunting): picked mon.noname-a con 0x7f1e78133ff0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e780a4320 con 0x7f1e78133ff0 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7811ffc0 con 0x7f1e7809d3a0 2024-09-16T05:36:04.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e78121cf0 con 0x7f1e7809c300 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient(hunting): authenticate will time out at 1021.302173s 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e6effd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f1e7809d3a0 legacy=0x7f1e781338e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e77fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1e78133ff0 0x7f1e781d10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52464/0 (socket says 172.21.15.63:52464) 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e77fff640 1 -- 172.21.15.63:0/3735124628 learned_addr learned my addr 172.21.15.63:0/3735124628 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696805076 0 0) 0x7f1e780a4320 con 0x7f1e78133ff0 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): my global_id is 4518 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e58003530 con 0x7f1e78133ff0 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3580276710 0 0) 0x7f1e78121cf0 con 0x7f1e7809c300 2024-09-16T05:36:04.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): my global_id is 4517 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e780a4320 con 0x7f1e7809c300 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1007303795 0 0) 0x7f1e58003530 con 0x7f1e78133ff0 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e78121cf0 con 0x7f1e78133ff0 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e70001c00 con 0x7f1e78133ff0 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.546 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3253207636 0 0) 0x7f1e780a4320 con 0x7f1e7809c300 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e58003530 con 0x7f1e7809c300 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e680015e0 con 0x7f1e7809c300 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:04.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3310430196 0 0) 0x7f1e78121cf0 con 0x7f1e78133ff0 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 >> v1:172.21.15.6:6790/0 conn(0x7f1e7809c300 legacy=0x7f1e781c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 >> v1:172.21.15.63:6789/0 conn(0x7f1e7809d3a0 legacy=0x7f1e781338e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 monclient: found mon.a 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e781222b0 con 0x7f1e78133ff0 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 5 monclient: authenticate success, global_id 4518 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient: _renew_subs 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.521+0000 7f1e7daf2640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1e781208a0 con 0x7f1e78133ff0 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e7daf2640 10 monclient: _renew_subs 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e7daf2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e7daf2640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1e781d2990 con 0x7f1e78133ff0 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1e70004b70 con 0x7f1e78133ff0 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e74ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e70004ec0 con 0x7f1e78133ff0 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e74ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e767fc640 4 set_mon_vals no callback set 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e74ff9640 10 monclient: dump: 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.550 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e7daf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e7daf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e7daf2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e7daf2640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e5c005090 con 0x7f1e78133ff0 2024-09-16T05:36:04.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1e7000fc60 con 0x7f1e78133ff0 2024-09-16T05:36:04.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3889+0+0 (unknown 3418402422 0 0) 0x7f1e70044530 con 0x7f1e78133ff0 2024-09-16T05:36:04.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.525+0000 7f1e777fe640 10 monclient: get_auth_request con 0x7f1e58036a50 auth_method 0 2024-09-16T05:36:04.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.537+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1e700192c0 con 0x7f1e78133ff0 2024-09-16T05:36:04.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.537+0000 7f1e74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.537+0000 7f1e74ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:04.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.701+0000 7f1e7daf2640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T05:36:04.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.701+0000 7f1e7daf2640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T05:36:04.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.701+0000 7f1e7daf2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.701+0000 7f1e7daf2640 1 -- 172.21.15.63:0/3735124628 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"} v 0) -- 0x7f1e5c005380 con 0x7f1e78133ff0 2024-09-16T05:36:04.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.701+0000 7f1e74ff9640 1 -- 172.21.15.63:0/3735124628 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}]=-22 specified pg_num 44 < pg_num_max 33 v25) ==== 149+0+0 (unknown 3720108747 0 0) 0x7f1e7001db80 con 0x7f1e78133ff0 2024-09-16T05:36:04.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.701+0000 7f1e74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T05:36:04.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.701+0000 7f1e74ff9640 10 monclient: _finish_command 2 = mon:22 specified pg_num 44 < pg_num_max 33 2024-09-16T05:36:04.707 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2024-09-16T05:36:04.712 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:04.712 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.817+0000 7ff8ae12f640 10 monclient: get_monmap_and_config 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.817+0000 7ff8ae12f640 10 monclient: build_initial_monmap 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient: monmap: 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:04.824517+0000 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:04.824517+0000 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:04.824 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:04.825 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:04.825 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 Processor -- start 2024-09-16T05:36:04.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- start start 2024-09-16T05:36:04.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient: init 2024-09-16T05:36:04.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a8134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a80a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a806dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a8135420 con 0x7ff8a80a0810 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a809e830 con 0x7ff8a8134930 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a80a0dc0 con 0x7ff8a806dfa0 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient(hunting): authenticate will time out at 1021.601532s 2024-09-16T05:36:04.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a77fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff8a8134930 legacy=0x7ff8a8134d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff8a806dfa0 0x7ff8a813cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46508/0 (socket says 172.21.15.63:46508) 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff89ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff8a80a0810 0x7ff8a8139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52476/0 (socket says 172.21.15.63:52476) 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a7fff640 1 -- 172.21.15.63:0/3331694175 learned_addr learned my addr 172.21.15.63:0/3331694175 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3639069382 0 0) 0x7ff8a80a0dc0 con 0x7ff8a806dfa0 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient(hunting): my global_id is 4520 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff894003530 con 0x7ff8a806dfa0 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3088707783 0 0) 0x7ff8a8135420 con 0x7ff8a80a0810 2024-09-16T05:36:04.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient(hunting): my global_id is 4521 2024-09-16T05:36:04.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8a80a0dc0 con 0x7ff8a80a0810 2024-09-16T05:36:04.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2272389542 0 0) 0x7ff8a80a0dc0 con 0x7ff8a80a0810 2024-09-16T05:36:04.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 >> v1:172.21.15.6:6790/0 conn(0x7ff8a806dfa0 legacy=0x7ff8a813cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 >> v1:172.21.15.63:6789/0 conn(0x7ff8a8134930 legacy=0x7ff8a8134d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:04.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:04.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 monclient: found mon.noname-a 2024-09-16T05:36:04.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:04.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:04.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a80a2500 con 0x7ff8a80a0810 2024-09-16T05:36:04.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:04.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:04.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:04.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 5 monclient: authenticate success, global_id 4521 2024-09-16T05:36:04.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff890001600 con 0x7ff8a80a0810 2024-09-16T05:36:04.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:04.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:04.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: dump: 2024-09-16T05:36:04.838 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.838 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.838 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.839 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.839 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.839 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.839 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.839 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.839 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.840 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff890001860 con 0x7ff8a80a0810 2024-09-16T05:36:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 1 -- 172.21.15.63:0/3331694175 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff890001cb0 con 0x7ff8a80a0810 2024-09-16T05:36:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:04.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 4 set_mon_vals no callback set 2024-09-16T05:36:04.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:04.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8a6ffd640 10 monclient: dump: 2024-09-16T05:36:04.841 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.842 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.842 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.842 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.842 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.842 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.843 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.843 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.843 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.843 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 10 monclient: shutdown 2024-09-16T05:36:04.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:04.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- 172.21.15.63:0/3331694175 >> v1:172.21.15.6:6789/0 conn(0x7ff8a80a0810 legacy=0x7ff8a8139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- 172.21.15.63:0/3331694175 shutdown_connections 2024-09-16T05:36:04.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- 172.21.15.63:0/3331694175 >> 172.21.15.63:0/3331694175 conn(0x7ff8a80a1e90 msgr2=0x7ff8a8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:04.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- 172.21.15.63:0/3331694175 shutdown_connections 2024-09-16T05:36:04.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.821+0000 7ff8ae12f640 1 -- 172.21.15.63:0/3331694175 wait complete. 2024-09-16T05:36:04.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: build_initial_monmap 2024-09-16T05:36:04.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: monmap: 2024-09-16T05:36:04.845 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:04.846 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:04.846 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:04.846 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:04.846 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:04.846 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.847 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:04.847 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:04.847 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:04.847 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 1 Processor -- start 2024-09-16T05:36:04.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 1 -- start start 2024-09-16T05:36:04.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: init 2024-09-16T05:36:04.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:04.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:04.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a806dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:04.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a80a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:04.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a8134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:04.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a8135b30 con 0x7ff8a806dfa0 2024-09-16T05:36:04.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a8138180 con 0x7ff8a8134930 2024-09-16T05:36:04.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a813e320 con 0x7ff8a80a0810 2024-09-16T05:36:04.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:04.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient(hunting): authenticate will time out at 1021.605085s 2024-09-16T05:36:04.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff8a8134930 legacy=0x7ff8a81d1760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:04.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff89ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff8a80a0810 0x7ff8a8135420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46510/0 (socket says 172.21.15.63:46510) 2024-09-16T05:36:04.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff89ffff640 1 -- 172.21.15.63:0/2720761202 learned_addr learned my addr 172.21.15.63:0/2720761202 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:04.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473611462 0 0) 0x7ff8a813e320 con 0x7ff8a80a0810 2024-09-16T05:36:04.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): my global_id is 4523 2024-09-16T05:36:04.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff888003530 con 0x7ff8a80a0810 2024-09-16T05:36:04.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2323486511 0 0) 0x7ff8a8135b30 con 0x7ff8a806dfa0 2024-09-16T05:36:04.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:04.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:04.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:04.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): my global_id is 4524 2024-09-16T05:36:04.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8a813e320 con 0x7ff8a806dfa0 2024-09-16T05:36:04.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1008154313 0 0) 0x7ff888003530 con 0x7ff8a80a0810 2024-09-16T05:36:04.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8a8135b30 con 0x7ff8a80a0810 2024-09-16T05:36:04.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff890001c30 con 0x7ff8a80a0810 2024-09-16T05:36:04.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:04.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:04.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:04.854 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.854 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.854 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.854 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.854 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.855 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.855 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.855 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.855 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.855 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:04.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 509791698 0 0) 0x7ff8a813e320 con 0x7ff8a806dfa0 2024-09-16T05:36:04.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff888003530 con 0x7ff8a806dfa0 2024-09-16T05:36:04.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff8980040d0 con 0x7ff8a806dfa0 2024-09-16T05:36:04.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:04.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:04.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:04.857 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.857 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.857 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.858 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.858 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.858 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.858 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.858 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.858 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.859 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 603581658 0 0) 0x7ff8a8135b30 con 0x7ff8a80a0810 2024-09-16T05:36:04.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 >> v1:172.21.15.63:6789/0 conn(0x7ff8a8134930 legacy=0x7ff8a81d1760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:04.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 >> v1:172.21.15.6:6789/0 conn(0x7ff8a806dfa0 legacy=0x7ff8a8134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:04.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:04.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 monclient: found mon.c 2024-09-16T05:36:04.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:04.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:04.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a809c0f0 con 0x7ff8a80a0810 2024-09-16T05:36:04.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:04.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:04.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:04.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 5 monclient: authenticate success, global_id 4523 2024-09-16T05:36:04.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: _renew_subs 2024-09-16T05:36:04.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:04.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a8071e20 con 0x7ff8a80a0810 2024-09-16T05:36:04.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: _renew_subs 2024-09-16T05:36:04.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:04.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff8a8137ea0 con 0x7ff8a80a0810 2024-09-16T05:36:04.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff890001860 con 0x7ff8a80a0810 2024-09-16T05:36:04.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:04.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff890004b40 con 0x7ff8a80a0810 2024-09-16T05:36:04.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:04.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:04.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a67fc640 4 set_mon_vals no callback set 2024-09-16T05:36:04.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8a4ff9640 10 monclient: dump: 2024-09-16T05:36:04.864 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:04.864 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:04.864 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.864 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:04.864 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:04.865 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:04.865 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:04.865 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:04.865 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:04.865 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:04.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:04.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.825+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88c005090 con 0x7ff8a80a0810 2024-09-16T05:36:04.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.829+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff890004da0 con 0x7ff8a80a0810 2024-09-16T05:36:04.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.829+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3889+0+0 (unknown 3418402422 0 0) 0x7ff8900442f0 con 0x7ff8a80a0810 2024-09-16T05:36:04.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.829+0000 7ff8a77fe640 10 monclient: get_auth_request con 0x7ff8880369d0 auth_method 0 2024-09-16T05:36:04.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.841+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff890015550 con 0x7ff8a80a0810 2024-09-16T05:36:04.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.841+0000 7ff8a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:04.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:04.841+0000 7ff8a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:05.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:05.001+0000 7ff8ae12f640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T05:36:05.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:05.001+0000 7ff8ae12f640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T05:36:05.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:05.001+0000 7ff8ae12f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:05.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:05.001+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) -- 0x7ff88c005380 con 0x7ff8a80a0810 2024-09-16T05:36:06.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:06.005+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v26) ==== 143+0+0 (unknown 4160523098 0 0) 0x7ff890014e50 con 0x7ff8a80a0810 2024-09-16T05:36:06.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:06.005+0000 7ff8a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T05:36:06.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:06.005+0000 7ff8a4ff9640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 0 2024-09-16T05:36:06.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:06.100+0000 7ff8ae12f640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T05:36:06.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:06.100+0000 7ff8ae12f640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T05:36:06.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:06.100+0000 7ff8ae12f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:06.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:06.100+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) -- 0x7ff88c0010a0 con 0x7ff8a80a0810 2024-09-16T05:36:07.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.012+0000 7ff8a4ff9640 1 -- 172.21.15.63:0/2720761202 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v27) ==== 143+0+0 (unknown 3385735541 0 0) 0x7ff8900402a0 con 0x7ff8a80a0810 2024-09-16T05:36:07.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.012+0000 7ff8a4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T05:36:07.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.012+0000 7ff8a4ff9640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 0 2024-09-16T05:36:07.019 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 3 pg_num_min to 0 2024-09-16T05:36:07.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.016+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff8880369d0 legacy=0x7ff888038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:07.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.016+0000 7ff8ae12f640 10 monclient: shutdown 2024-09-16T05:36:07.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.016+0000 7ff8ae12f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:07.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.016+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 >> v1:172.21.15.6:6790/0 conn(0x7ff8a80a0810 legacy=0x7ff8a8135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:07.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.016+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 shutdown_connections 2024-09-16T05:36:07.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.016+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 >> 172.21.15.63:0/2720761202 conn(0x7ff8a80a1e90 msgr2=0x7ff8a809f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:07.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.016+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 shutdown_connections 2024-09-16T05:36:07.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.016+0000 7ff8ae12f640 1 -- 172.21.15.63:0/2720761202 wait complete. 2024-09-16T05:36:07.044 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2024-09-16T05:36:07.044 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:07.044 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.164+0000 7fd542565640 10 monclient: get_monmap_and_config 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.164+0000 7fd542565640 10 monclient: build_initial_monmap 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.164+0000 7fd542565640 10 monclient: monmap: 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:07.171529+0000 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:07.171529+0000 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:07.171 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.172 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:07.172 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:07.172 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:07.172 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 Processor -- start 2024-09-16T05:36:07.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- start start 2024-09-16T05:36:07.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient: init 2024-09-16T05:36:07.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:07.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:07.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient(hunting): picked mon.noname-a con 0x7fd53c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:07.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient(hunting): picked mon.noname-b con 0x7fd53c09b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient(hunting): picked mon.noname-c con 0x7fd53c139fc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53c0a14f0 con 0x7fd53c09ac90 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53c09c680 con 0x7fd53c139fc0 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53c09e930 con 0x7fd53c09b780 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient(hunting): authenticate will time out at 1023.949002s 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd540b02640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd53c139fc0 legacy=0x7fd53c13c3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd53c09b780 0x7fd53c1388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46512/0 (socket says 172.21.15.63:46512) 2024-09-16T05:36:07.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd53c09ac90 0x7fd53c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52484/0 (socket says 172.21.15.63:52484) 2024-09-16T05:36:07.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53b7fe640 1 -- 172.21.15.63:0/781076506 learned_addr learned my addr 172.21.15.63:0/781076506 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:07.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639142330 0 0) 0x7fd53c0a14f0 con 0x7fd53c09ac90 2024-09-16T05:36:07.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:07.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient(hunting): my global_id is 4527 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd524003530 con 0x7fd53c09ac90 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2833153026 0 0) 0x7fd53c09e930 con 0x7fd53c09b780 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient(hunting): my global_id is 4526 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd53c0a14f0 con 0x7fd53c09b780 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 387580563 0 0) 0x7fd524003530 con 0x7fd53c09ac90 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 >> v1:172.21.15.6:6790/0 conn(0x7fd53c09b780 legacy=0x7fd53c1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:07.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 >> v1:172.21.15.63:6789/0 conn(0x7fd53c139fc0 legacy=0x7fd53c13c3b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:07.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:07.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 monclient: found mon.noname-a 2024-09-16T05:36:07.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:07.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:07.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53c09c400 con 0x7fd53c09ac90 2024-09-16T05:36:07.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: _finish_auth 0 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd52c001620 con 0x7fd53c09ac90 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: dump: 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.181 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 5 monclient: authenticate success, global_id 4527 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd52c0018d0 con 0x7fd53c09ac90 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 1 -- 172.21.15.63:0/781076506 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd52c001ce0 con 0x7fd53c09ac90 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 4 set_mon_vals no callback set 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:07.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd53affd640 10 monclient: dump: 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.183 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 10 monclient: shutdown 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- 172.21.15.63:0/781076506 >> v1:172.21.15.6:6789/0 conn(0x7fd53c09ac90 legacy=0x7fd53c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- 172.21.15.63:0/781076506 shutdown_connections 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- 172.21.15.63:0/781076506 >> 172.21.15.63:0/781076506 conn(0x7fd53c0a5290 msgr2=0x7fd53c0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- 172.21.15.63:0/781076506 shutdown_connections 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.168+0000 7fd542565640 1 -- 172.21.15.63:0/781076506 wait complete. 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: build_initial_monmap 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: monmap: 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:07.184 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 1 Processor -- start 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 1 -- start start 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: init 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:07.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient(hunting): picked mon.noname-c con 0x7fd53c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient(hunting): picked mon.noname-b con 0x7fd53c09b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient(hunting): picked mon.noname-a con 0x7fd53c139fc0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53c121f10 con 0x7fd53c139fc0 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53c1221f0 con 0x7fd53c09b780 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53c0a3fc0 con 0x7fd53c09ac90 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient(hunting): authenticate will time out at 1023.953049s 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd53b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd53c09b780 legacy=0x7fd53c1d0600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd540b02640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd53c139fc0 0x7fd53c1d3d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52486/0 (socket says 172.21.15.63:52486) 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd53bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd53c09ac90 0x7fd53c133800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46520/0 (socket says 172.21.15.63:46520) 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd540b02640 1 -- 172.21.15.63:0/899897831 learned_addr learned my addr 172.21.15.63:0/899897831 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:07.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042552701 0 0) 0x7fd53c0a3fc0 con 0x7fd53c09ac90 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): my global_id is 4529 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd528003530 con 0x7fd53c09ac90 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2327012706 0 0) 0x7fd53c121f10 con 0x7fd53c139fc0 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:07.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): my global_id is 4530 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd53c0a3fc0 con 0x7fd53c139fc0 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 579639424 0 0) 0x7fd53c0a3fc0 con 0x7fd53c139fc0 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd53c121f10 con 0x7fd53c139fc0 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1179583070 0 0) 0x7fd528003530 con 0x7fd53c09ac90 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd53c0a3fc0 con 0x7fd53c09ac90 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd534004300 con 0x7fd53c139fc0 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.188 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd52c001c20 con 0x7fd53c09ac90 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.189 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.190 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.190 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.190 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.190 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.190 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.190 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.190 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.190 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1316272020 0 0) 0x7fd53c0a3fc0 con 0x7fd53c09ac90 2024-09-16T05:36:07.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 >> v1:172.21.15.63:6789/0 conn(0x7fd53c09b780 legacy=0x7fd53c1d0600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:07.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 >> v1:172.21.15.6:6789/0 conn(0x7fd53c139fc0 legacy=0x7fd53c1d3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:07.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:07.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 monclient: found mon.c 2024-09-16T05:36:07.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:07.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53c09c0f0 con 0x7fd53c09ac90 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 5 monclient: authenticate success, global_id 4529 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: _renew_subs 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd53c139a80 con 0x7fd53c09ac90 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: _renew_subs 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd542565640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd53c1207e0 con 0x7fd53c09ac90 2024-09-16T05:36:07.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd52c0018d0 con 0x7fd53c09ac90 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd52c004d20 con 0x7fd53c09ac90 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd538ff9640 10 monclient: dump: 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.194 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.172+0000 7fd53a7fc640 4 set_mon_vals no callback set 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.176+0000 7fd542565640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.176+0000 7fd542565640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.176+0000 7fd542565640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.176+0000 7fd542565640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd51c005090 con 0x7fd53c09ac90 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.176+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd52c004f40 con 0x7fd53c09ac90 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.176+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 3873+0+0 (unknown 2401824728 0 0) 0x7fd52c044350 con 0x7fd53c09ac90 2024-09-16T05:36:07.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.176+0000 7fd53b7fe640 10 monclient: get_auth_request con 0x7fd5280369a0 auth_method 0 2024-09-16T05:36:07.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.188+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd52c0155c0 con 0x7fd53c09ac90 2024-09-16T05:36:07.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.188+0000 7fd538ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:07.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.188+0000 7fd538ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:07.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.352+0000 7fd542565640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:07.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.352+0000 7fd542565640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:07.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.352+0000 7fd542565640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:07.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.352+0000 7fd542565640 1 -- 172.21.15.63:0/899897831 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) -- 0x7fd51c005b90 con 0x7fd53c09ac90 2024-09-16T05:36:07.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.352+0000 7fd538ff9640 1 -- 172.21.15.63:0/899897831 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=-2 option 'pg_num_min' is not set on pool 'testpool1234' v27) ==== 158+0+0 (unknown 1736350970 0 0) 0x7fd52c040310 con 0x7fd53c09ac90 2024-09-16T05:36:07.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.352+0000 7fd538ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T05:36:07.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.352+0000 7fd538ff9640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-16T05:36:07.358 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-16T05:36:07.364 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:07.364 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2024-09-16T05:36:07.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.484+0000 7f051bfb9640 10 monclient: get_monmap_and_config 2024-09-16T05:36:07.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.484+0000 7f051bfb9640 10 monclient: build_initial_monmap 2024-09-16T05:36:07.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.484+0000 7f051bfb9640 10 monclient: monmap: 2024-09-16T05:36:07.490 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:07.490 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.490 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:07.490739+0000 2024-09-16T05:36:07.491 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:07.490739+0000 2024-09-16T05:36:07.491 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:07.491 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.491 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:07.491 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:07.491 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:07.491 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.484+0000 7f051bfb9640 1 Processor -- start 2024-09-16T05:36:07.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.484+0000 7f051bfb9640 1 -- start start 2024-09-16T05:36:07.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.484+0000 7f051bfb9640 10 monclient: init 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient(hunting): picked mon.noname-c con 0x7f05141339b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient(hunting): picked mon.noname-a con 0x7f051409fc50 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient(hunting): picked mon.noname-b con 0x7f051413d140 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05140a0710 con 0x7f051409fc50 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05140a09f0 con 0x7f05141339b0 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05140a0cd0 con 0x7f051413d140 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient(hunting): authenticate will time out at 1024.268311s 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0519d55640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f05141339b0 legacy=0x7f0514133d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0519554640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f051409fc50 0x7f051413ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52496/0 (socket says 172.21.15.63:52496) 2024-09-16T05:36:07.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051a556640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f051413d140 0x7f051413f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46528/0 (socket says 172.21.15.63:46528) 2024-09-16T05:36:07.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0519554640 1 -- 172.21.15.63:0/1677453141 learned_addr learned my addr 172.21.15.63:0/1677453141 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:07.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317182918 0 0) 0x7f05140a0cd0 con 0x7f051413d140 2024-09-16T05:36:07.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient(hunting): _negotiate 2024-09-16T05:36:07.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient(hunting): my global_id is 4532 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0500003530 con 0x7f051413d140 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795818009 0 0) 0x7f05140a0710 con 0x7f051409fc50 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient(hunting): _negotiate 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient(hunting): my global_id is 4533 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05140a0cd0 con 0x7f051409fc50 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1939094365 0 0) 0x7f0500003530 con 0x7f051413d140 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 >> v1:172.21.15.63:6789/0 conn(0x7f05141339b0 legacy=0x7f0514133d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 >> v1:172.21.15.6:6789/0 conn(0x7f051409fc50 legacy=0x7f051413ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:07.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: _finish_hunting 0 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 monclient: found mon.noname-b 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05140a15b0 con 0x7f051413d140 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: _finish_auth 0 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: _check_auth_tickets 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f050c0019a0 con 0x7f051413d140 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: dump: 2024-09-16T05:36:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 5 monclient: authenticate success, global_id 4532 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f050c001c00 con 0x7f051413d140 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 1 -- 172.21.15.63:0/1677453141 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f050c002010 con 0x7f051413d140 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f0518d53640 10 monclient: dump: 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.500 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 4 set_mon_vals no callback set 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient: shutdown 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- 172.21.15.63:0/1677453141 >> v1:172.21.15.6:6790/0 conn(0x7f051413d140 legacy=0x7f051413f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- 172.21.15.63:0/1677453141 shutdown_connections 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- 172.21.15.63:0/1677453141 >> 172.21.15.63:0/1677453141 conn(0x7f05140a1e90 msgr2=0x7f0514075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- 172.21.15.63:0/1677453141 shutdown_connections 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- 172.21.15.63:0/1677453141 wait complete. 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient: build_initial_monmap 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient: monmap: 2024-09-16T05:36:07.501 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 Processor -- start 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 1 -- start start 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.488+0000 7f051bfb9640 10 monclient: init 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:07.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient(hunting): picked mon.noname-c con 0x7f051409fc50 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient(hunting): picked mon.noname-a con 0x7f05141339b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient(hunting): picked mon.noname-b con 0x7f051413d140 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f051412f640 con 0x7f05141339b0 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f051413f3c0 con 0x7f051413d140 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0514140770 con 0x7f051409fc50 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient(hunting): authenticate will time out at 1024.272435s 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051a556640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f051413d140 legacy=0x7f05141d3c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f0519554640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f05141339b0 0x7f051413ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52500/0 (socket says 172.21.15.63:52500) 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f0519554640 1 -- 172.21.15.63:0/3304426665 learned_addr learned my addr 172.21.15.63:0/3304426665 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:07.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f0519d55640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f051409fc50 0x7f051413e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46530/0 (socket says 172.21.15.63:46530) 2024-09-16T05:36:07.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4075945960 0 0) 0x7f0514140770 con 0x7f051409fc50 2024-09-16T05:36:07.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:07.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:07.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:07.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): my global_id is 4535 2024-09-16T05:36:07.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04f8003530 con 0x7f051409fc50 2024-09-16T05:36:07.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331176953 0 0) 0x7f051412f640 con 0x7f05141339b0 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): my global_id is 4536 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0514140770 con 0x7f05141339b0 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2109398050 0 0) 0x7f0514140770 con 0x7f05141339b0 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f051412f640 con 0x7f05141339b0 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 480129244 0 0) 0x7f04f8003530 con 0x7f051409fc50 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0514140770 con 0x7f051409fc50 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0510004000 con 0x7f051409fc50 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:07.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): dump: 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f05040019e0 con 0x7f05141339b0 2024-09-16T05:36:07.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient(hunting): dump: 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1886654357 0 0) 0x7f051412f640 con 0x7f05141339b0 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 >> v1:172.21.15.6:6790/0 conn(0x7f051409fc50 legacy=0x7f051413e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 >> v1:172.21.15.63:6789/0 conn(0x7f051413d140 legacy=0x7f05141d3c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 monclient: found mon.a 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051413c8f0 con 0x7f05141339b0 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 5 monclient: authenticate success, global_id 4536 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient: _renew_subs 2024-09-16T05:36:07.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f051412e990 con 0x7f05141339b0 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient: _renew_subs 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f051412f360 con 0x7f05141339b0 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f05040047f0 con 0x7f05141339b0 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0504004bc0 con 0x7f05141339b0 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050bfff640 4 set_mon_vals no callback set 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 10 monclient: dump: 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:07.511 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:07.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04fc005090 con 0x7f05141339b0 2024-09-16T05:36:07.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.492+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f050400f960 con 0x7f05141339b0 2024-09-16T05:36:07.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.496+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3873+0+0 (unknown 2401824728 0 0) 0x7f0504044130 con 0x7f05141339b0 2024-09-16T05:36:07.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.496+0000 7f0519d55640 10 monclient: get_auth_request con 0x7f04f80369c0 auth_method 0 2024-09-16T05:36:07.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.508+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0504015420 con 0x7f05141339b0 2024-09-16T05:36:07.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.508+0000 7f050a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:07.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.508+0000 7f050a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:07.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.668+0000 7f051bfb9640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T05:36:07.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.668+0000 7f051bfb9640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T05:36:07.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.668+0000 7f051bfb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:07.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:07.668+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) -- 0x7f04fc005380 con 0x7f05141339b0 2024-09-16T05:36:08.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:08.012+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v28) ==== 143+0+0 (unknown 820237781 0 0) 0x7f0504014d20 con 0x7f05141339b0 2024-09-16T05:36:08.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:08.012+0000 7f050a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T05:36:08.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:08.012+0000 7f050a7fc640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 0 2024-09-16T05:36:08.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:08.116+0000 7f051bfb9640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T05:36:08.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:08.116+0000 7f051bfb9640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T05:36:08.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:08.116+0000 7f051bfb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:08.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:08.116+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) -- 0x7f04fc0010a0 con 0x7f05141339b0 2024-09-16T05:36:09.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f050a7fc640 1 -- 172.21.15.63:0/3304426665 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v29) ==== 143+0+0 (unknown 248604666 0 0) 0x7f050401d8d0 con 0x7f05141339b0 2024-09-16T05:36:09.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f050a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T05:36:09.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f050a7fc640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 0 2024-09-16T05:36:09.022 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 3 pg_num_max to 0 2024-09-16T05:36:09.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 >> v1:172.21.15.63:6812/3643207211 conn(0x7f04f80369c0 legacy=0x7f04f8038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:09.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f051bfb9640 10 monclient: shutdown 2024-09-16T05:36:09.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f051bfb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:09.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 >> v1:172.21.15.6:6789/0 conn(0x7f05141339b0 legacy=0x7f051413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:09.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 shutdown_connections 2024-09-16T05:36:09.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 >> 172.21.15.63:0/3304426665 conn(0x7f05140a1e90 msgr2=0x7f051412c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:09.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 shutdown_connections 2024-09-16T05:36:09.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.020+0000 7f051bfb9640 1 -- 172.21.15.63:0/3304426665 wait complete. 2024-09-16T05:36:09.047 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2024-09-16T05:36:09.047 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:09.047 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-16T05:36:09.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.164+0000 7f0193f26640 10 monclient: get_monmap_and_config 2024-09-16T05:36:09.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.164+0000 7f0193f26640 10 monclient: build_initial_monmap 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.164+0000 7f0193f26640 10 monclient: monmap: 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:09.171965+0000 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:09.171965+0000 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:09.172 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 Processor -- start 2024-09-16T05:36:09.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- start start 2024-09-16T05:36:09.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient: init 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient(hunting): picked mon.noname-a con 0x7f018c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient(hunting): picked mon.noname-c con 0x7f018c09b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient(hunting): picked mon.noname-b con 0x7f018c139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c0a14f0 con 0x7f018c09ac90 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c09c680 con 0x7f018c09b780 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c09e930 con 0x7f018c139fc0 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient(hunting): authenticate will time out at 1025.949359s 2024-09-16T05:36:09.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f01914c1640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f018c09b780 legacy=0x7f018c1388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0191cc2640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f018c09ac90 0x7f018c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52514/0 (socket says 172.21.15.63:52514) 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0191cc2640 1 -- 172.21.15.63:0/2748183001 learned_addr learned my addr 172.21.15.63:0/2748183001 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f01924c3640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f018c139fc0 0x7f018c13c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46538/0 (socket says 172.21.15.63:46538) 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131180855 0 0) 0x7f018c09e930 con 0x7f018c139fc0 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient(hunting): _negotiate 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient(hunting): my global_id is 4538 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0178003530 con 0x7f018c139fc0 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999728197 0 0) 0x7f018c0a14f0 con 0x7f018c09ac90 2024-09-16T05:36:09.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient(hunting): _negotiate 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient(hunting): my global_id is 4539 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f018c09e930 con 0x7f018c09ac90 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3762121022 0 0) 0x7f0178003530 con 0x7f018c139fc0 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 >> v1:172.21.15.63:6789/0 conn(0x7f018c09b780 legacy=0x7f018c1388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 >> v1:172.21.15.6:6789/0 conn(0x7f018c09ac90 legacy=0x7f018c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: _finish_hunting 0 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 monclient: found mon.noname-b 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c09c400 con 0x7f018c139fc0 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: _finish_auth 0 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: _check_auth_tickets 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01840019a0 con 0x7f018c139fc0 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: dump: 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.179 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 5 monclient: authenticate success, global_id 4538 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0184001c50 con 0x7f018c139fc0 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 1 -- 172.21.15.63:0/2748183001 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0184002020 con 0x7f018c139fc0 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:09.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 4 set_mon_vals no callback set 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0190cc0640 10 monclient: dump: 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.181 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 10 monclient: shutdown 2024-09-16T05:36:09.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:09.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- 172.21.15.63:0/2748183001 >> v1:172.21.15.6:6790/0 conn(0x7f018c139fc0 legacy=0x7f018c13c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:09.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- 172.21.15.63:0/2748183001 shutdown_connections 2024-09-16T05:36:09.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- 172.21.15.63:0/2748183001 >> 172.21.15.63:0/2748183001 conn(0x7f018c0a5290 msgr2=0x7f018c0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:09.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- 172.21.15.63:0/2748183001 shutdown_connections 2024-09-16T05:36:09.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.168+0000 7f0193f26640 1 -- 172.21.15.63:0/2748183001 wait complete. 2024-09-16T05:36:09.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: build_initial_monmap 2024-09-16T05:36:09.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: monmap: 2024-09-16T05:36:09.186 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:09.186 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:09.186 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:09.186 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:09.186 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 1 Processor -- start 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 1 -- start start 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: init 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient(hunting): picked mon.noname-b con 0x7f018c09ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient(hunting): picked mon.noname-a con 0x7f018c09b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:09.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient(hunting): picked mon.noname-c con 0x7f018c139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c11d410 con 0x7f018c09b780 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c11d6f0 con 0x7f018c09ac90 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c11c3f0 con 0x7f018c139fc0 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient(hunting): authenticate will time out at 1025.953251s 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0191cc2640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f018c09ac90 legacy=0x7f018c138130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01914c1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f018c09b780 0x7f018c138840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52524/0 (socket says 172.21.15.63:52524) 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01914c1640 1 -- 172.21.15.63:0/832979683 learned_addr learned my addr 172.21.15.63:0/832979683 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3160576083 0 0) 0x7f018c11c3f0 con 0x7f018c139fc0 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:09.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): my global_id is 4541 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0170003530 con 0x7f018c139fc0 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2822216956 0 0) 0x7f018c11d410 con 0x7f018c09b780 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): my global_id is 4542 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f018c11c3f0 con 0x7f018c09b780 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3237595167 0 0) 0x7f0170003530 con 0x7f018c139fc0 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f018c11d410 con 0x7f018c139fc0 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0184001fa0 con 0x7f018c139fc0 2024-09-16T05:36:09.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): dump: 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.190 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 552876010 0 0) 0x7f018c11c3f0 con 0x7f018c09b780 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0170003530 con 0x7f018c09b780 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f017c004120 con 0x7f018c09b780 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient(hunting): dump: 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.191 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2751232455 0 0) 0x7f018c11d410 con 0x7f018c139fc0 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 >> v1:172.21.15.63:6789/0 conn(0x7f018c09ac90 legacy=0x7f018c138130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 >> v1:172.21.15.6:6789/0 conn(0x7f018c09b780 legacy=0x7f018c138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 monclient: found mon.c 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:09.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c11b670 con 0x7f018c139fc0 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 5 monclient: authenticate success, global_id 4541 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: _renew_subs 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f018c11d9d0 con 0x7f018c139fc0 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: _renew_subs 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f0193f26640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f018c139120 con 0x7f018c139fc0 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0184001c50 con 0x7f018c139fc0 2024-09-16T05:36:09.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0184004f10 con 0x7f018c139fc0 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01837fe640 4 set_mon_vals no callback set 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.172+0000 7f01827fc640 10 monclient: dump: 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.194 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.176+0000 7f0193f26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.176+0000 7f0193f26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.176+0000 7f0193f26640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.176+0000 7f0193f26640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0174005090 con 0x7f018c139fc0 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.176+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f018400fcb0 con 0x7f018c139fc0 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.176+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(29..29 src has 1..29) ==== 3857+0+0 (unknown 600541090 0 0) 0x7f0184044630 con 0x7f018c139fc0 2024-09-16T05:36:09.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.176+0000 7f0191cc2640 10 monclient: get_auth_request con 0x7f0170036a50 auth_method 0 2024-09-16T05:36:09.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.188+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f01840157f0 con 0x7f018c139fc0 2024-09-16T05:36:09.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.188+0000 7f01827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:09.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.188+0000 7f01827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:09.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.352+0000 7f0193f26640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:09.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.352+0000 7f0193f26640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:09.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.352+0000 7f0193f26640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.352+0000 7f0193f26640 1 -- 172.21.15.63:0/832979683 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) -- 0x7f0174005b90 con 0x7f018c139fc0 2024-09-16T05:36:09.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.352+0000 7f01827fc640 1 -- 172.21.15.63:0/832979683 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=-2 option 'pg_num_max' is not set on pool 'testpool1234' v29) ==== 158+0+0 (unknown 1359812769 0 0) 0x7f01840150f0 con 0x7f018c139fc0 2024-09-16T05:36:09.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.352+0000 7f01827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T05:36:09.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.352+0000 7f01827fc640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-16T05:36:09.359 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-16T05:36:09.364 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:09.364 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2024-09-16T05:36:09.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient: get_monmap_and_config 2024-09-16T05:36:09.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient: build_initial_monmap 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient: monmap: 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:09.490121+0000 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:09.490121+0000 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:09.490 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 1 Processor -- start 2024-09-16T05:36:09.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 1 -- start start 2024-09-16T05:36:09.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient: init 2024-09-16T05:36:09.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca40a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient(hunting): picked mon.noname-b con 0x7f6ca40a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca40a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca40a51e0 con 0x7f6ca40a4790 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca40a0d60 con 0x7f6ca40a3ca0 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca40a1040 con 0x7f6ca40a4d90 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:09.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6ca3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6ca40a3ca0 legacy=0x7f6ca40a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.484+0000 7f6caa2e3640 10 monclient(hunting): authenticate will time out at 1026.267926s 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca37fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6ca40a4d90 0x7f6ca413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46554/0 (socket says 172.21.15.63:46554) 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca37fe640 1 -- 172.21.15.63:0/3082258792 learned_addr learned my addr 172.21.15.63:0/3082258792 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca8880640 1 --1- 172.21.15.63:0/3082258792 >> v1:172.21.15.6:6789/0 conn(0x7f6ca40a4790 0x7f6ca413f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52540/0 (socket says 172.21.15.63:52540) 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1397693383 0 0) 0x7f6ca40a1040 con 0x7f6ca40a4d90 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient(hunting): my global_id is 4544 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c8c003530 con 0x7f6ca40a4d90 2024-09-16T05:36:09.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 460750022 0 0) 0x7f6ca40a51e0 con 0x7f6ca40a4790 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient(hunting): my global_id is 4545 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ca40a1040 con 0x7f6ca40a4790 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1082231297 0 0) 0x7f6c8c003530 con 0x7f6ca40a4d90 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 >> v1:172.21.15.63:6789/0 conn(0x7f6ca40a3ca0 legacy=0x7f6ca40a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 >> v1:172.21.15.6:6789/0 conn(0x7f6ca40a4790 legacy=0x7f6ca413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 monclient: found mon.noname-b 2024-09-16T05:36:09.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca40a0810 con 0x7f6ca40a4d90 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c98001620 con 0x7f6ca40a4d90 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: dump: 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.498 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 5 monclient: authenticate success, global_id 4544 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c980018d0 con 0x7f6ca40a4d90 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 1 -- 172.21.15.63:0/3082258792 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c98001ca0 con 0x7f6ca40a4d90 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:09.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca2ffd640 10 monclient: dump: 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 4 set_mon_vals no callback set 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient: shutdown 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- 172.21.15.63:0/3082258792 >> v1:172.21.15.6:6790/0 conn(0x7f6ca40a4d90 legacy=0x7f6ca413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- 172.21.15.63:0/3082258792 shutdown_connections 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- 172.21.15.63:0/3082258792 >> 172.21.15.63:0/3082258792 conn(0x7f6ca4075470 msgr2=0x7f6ca4075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- 172.21.15.63:0/3082258792 shutdown_connections 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- 172.21.15.63:0/3082258792 wait complete. 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient: build_initial_monmap 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient: monmap: 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:09.501 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:09.502 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:09.502 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:09.502 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.502 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:09.502 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:09.502 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:09.502 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 Processor -- start 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- start start 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient: init 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient(hunting): picked mon.noname-b con 0x7f6ca40a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca40a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca40a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca4136b50 con 0x7f6ca40a4790 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6ca3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6ca40a3ca0 legacy=0x7f6ca4134eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:09.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca4135cd0 con 0x7f6ca40a3ca0 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca40a0500 con 0x7f6ca40a4d90 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.488+0000 7f6caa2e3640 10 monclient(hunting): authenticate will time out at 1026.271782s 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6ca40a4790 0x7f6ca41355c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52546/0 (socket says 172.21.15.63:52546) 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca37fe640 1 -- 172.21.15.63:0/228542218 learned_addr learned my addr 172.21.15.63:0/228542218 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371867946 0 0) 0x7f6ca4136b50 con 0x7f6ca40a4790 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): my global_id is 4548 2024-09-16T05:36:09.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c90003530 con 0x7f6ca40a4790 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2663253657 0 0) 0x7f6ca40a0500 con 0x7f6ca40a4d90 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): my global_id is 4547 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ca4136b50 con 0x7f6ca40a4d90 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2396432205 0 0) 0x7f6ca4136b50 con 0x7f6ca40a4d90 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ca40a0500 con 0x7f6ca40a4d90 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c9c001bc0 con 0x7f6ca40a4d90 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:09.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3876838935 0 0) 0x7f6c90003530 con 0x7f6ca40a4790 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ca4136b50 con 0x7f6ca40a4790 2024-09-16T05:36:09.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c98001c20 con 0x7f6ca40a4790 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 472093785 0 0) 0x7f6ca40a0500 con 0x7f6ca40a4d90 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 >> v1:172.21.15.63:6789/0 conn(0x7f6ca40a3ca0 legacy=0x7f6ca4134eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 >> v1:172.21.15.6:6789/0 conn(0x7f6ca40a4790 legacy=0x7f6ca41355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 monclient: found mon.c 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca4139080 con 0x7f6ca40a4d90 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:09.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 5 monclient: authenticate success, global_id 4547 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 10 monclient: _renew_subs 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca41376b0 con 0x7f6ca40a4d90 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 10 monclient: _renew_subs 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ca41d4a80 con 0x7f6ca40a4d90 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c9c004b70 con 0x7f6ca40a4d90 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c9c004d20 con 0x7f6ca40a4d90 2024-09-16T05:36:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca27fc640 4 set_mon_vals no callback set 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 10 monclient: dump: 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:09.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c84005090 con 0x7f6ca40a4d90 2024-09-16T05:36:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6c9c004f40 con 0x7f6ca40a4d90 2024-09-16T05:36:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(29..29 src has 1..29) ==== 3857+0+0 (unknown 600541090 0 0) 0x7f6c9c044440 con 0x7f6ca40a4d90 2024-09-16T05:36:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.492+0000 7f6ca3fff640 10 monclient: get_auth_request con 0x7f6c90036a00 auth_method 0 2024-09-16T05:36:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.504+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6c9c0155a0 con 0x7f6ca40a4d90 2024-09-16T05:36:09.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.504+0000 7f6ca0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:09.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.504+0000 7f6ca0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.668+0000 7f6caa2e3640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.668+0000 7f6caa2e3640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.668+0000 7f6caa2e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:09.668+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6c84005380 con 0x7f6ca40a4d90 2024-09-16T05:36:10.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.040+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' removed v30) ==== 176+0+0 (unknown 2084555871 0 0) 0x7f6c9c014ea0 con 0x7f6ca40a4d90 2024-09-16T05:36:10.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.040+0000 7f6ca0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:10.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.040+0000 7f6ca0ff9640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' removed 2024-09-16T05:36:10.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.136+0000 7f6caa2e3640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:10.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.136+0000 7f6caa2e3640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:10.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.136+0000 7f6caa2e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:10.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.136+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6c84005ac0 con 0x7f6ca40a4d90 2024-09-16T05:36:10.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.140+0000 7f6ca0ff9640 1 -- 172.21.15.63:0/228542218 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v30) ==== 183+0+0 (unknown 2931140000 0 0) 0x7f6c9c0191f0 con 0x7f6ca40a4d90 2024-09-16T05:36:10.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.140+0000 7f6ca0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:10.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.140+0000 7f6ca0ff9640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' does not exist 2024-09-16T05:36:10.146 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'testpool1234' does not exist 2024-09-16T05:36:10.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.144+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6c90036a00 legacy=0x7f6c90038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:10.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.144+0000 7f6caa2e3640 10 monclient: shutdown 2024-09-16T05:36:10.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.144+0000 7f6caa2e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:10.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.144+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 >> v1:172.21.15.6:6790/0 conn(0x7f6ca40a4d90 legacy=0x7f6ca41d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:10.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.144+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 shutdown_connections 2024-09-16T05:36:10.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.144+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 >> 172.21.15.63:0/228542218 conn(0x7f6ca4075470 msgr2=0x7f6ca40a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:10.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.144+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 shutdown_connections 2024-09-16T05:36:10.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.144+0000 7f6caa2e3640 1 -- 172.21.15.63:0/228542218 wait complete. 2024-09-16T05:36:10.164 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-16T05:36:10.164 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:10.165 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-16T05:36:10.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient: get_monmap_and_config 2024-09-16T05:36:10.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient: build_initial_monmap 2024-09-16T05:36:10.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient: monmap: 2024-09-16T05:36:10.296 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:10.296 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.297 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:10.296868+0000 2024-09-16T05:36:10.297 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:10.296868+0000 2024-09-16T05:36:10.297 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:10.297 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.297 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:10.297 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:10.297 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:10.297 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 1 Processor -- start 2024-09-16T05:36:10.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 1 -- start start 2024-09-16T05:36:10.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient: init 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient(hunting): picked mon.noname-c con 0x7fbab809db30 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient(hunting): picked mon.noname-a con 0x7fbab809f7e0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient(hunting): picked mon.noname-b con 0x7fbab809e620 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab809ed90 con 0x7fbab809f7e0 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabd0d6640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbab809db30 legacy=0x7fbab809df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab809ea50 con 0x7fbab809db30 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab8099520 con 0x7fbab809e620 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:10.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabf33a640 10 monclient(hunting): authenticate will time out at 1027.074827s 2024-09-16T05:36:10.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabd8d7640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbab809e620 0x7fbab813c2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46576/0 (socket says 172.21.15.63:46576) 2024-09-16T05:36:10.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabd8d7640 1 -- 172.21.15.63:0/2961521811 learned_addr learned my addr 172.21.15.63:0/2961521811 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:10.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbabc8d5640 1 --1- 172.21.15.63:0/2961521811 >> v1:172.21.15.6:6789/0 conn(0x7fbab809f7e0 0x7fbab8138ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52558/0 (socket says 172.21.15.63:52558) 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229949120 0 0) 0x7fbab809ed90 con 0x7fbab809f7e0 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 10 monclient(hunting): my global_id is 4551 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaa4003530 con 0x7fbab809f7e0 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3527018630 0 0) 0x7fbab8099520 con 0x7fbab809e620 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 10 monclient(hunting): my global_id is 4550 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.292+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbab809ed90 con 0x7fbab809e620 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3295528869 0 0) 0x7fbaa4003530 con 0x7fbab809f7e0 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 >> v1:172.21.15.6:6790/0 conn(0x7fbab809e620 legacy=0x7fbab813c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 >> v1:172.21.15.63:6789/0 conn(0x7fbab809db30 legacy=0x7fbab809df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 1 monclient: found mon.noname-a 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab80a1440 con 0x7fbab809f7e0 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: _finish_auth 0 2024-09-16T05:36:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbaa8001620 con 0x7fbab809f7e0 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 5 monclient: authenticate success, global_id 4551 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: dump: 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbaa8001880 con 0x7fbab809f7e0 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 1 -- 172.21.15.63:0/2961521811 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbaa8001c90 con 0x7fbab809f7e0 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaaffff640 10 monclient: dump: 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.305 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 4 set_mon_vals no callback set 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient: shutdown 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- 172.21.15.63:0/2961521811 >> v1:172.21.15.6:6789/0 conn(0x7fbab809f7e0 legacy=0x7fbab8138ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- 172.21.15.63:0/2961521811 shutdown_connections 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- 172.21.15.63:0/2961521811 >> 172.21.15.63:0/2961521811 conn(0x7fbab80a2420 msgr2=0x7fbab80a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- 172.21.15.63:0/2961521811 shutdown_connections 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- 172.21.15.63:0/2961521811 wait complete. 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient: build_initial_monmap 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient: monmap: 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:10.308 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:10.308 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 Processor -- start 2024-09-16T05:36:10.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- start start 2024-09-16T05:36:10.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient: init 2024-09-16T05:36:10.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:10.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:10.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient(hunting): picked mon.noname-a con 0x7fbab809db30 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:10.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient(hunting): picked mon.noname-c con 0x7fbab809e620 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:10.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient(hunting): picked mon.noname-b con 0x7fbab809f7e0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab813ba60 con 0x7fbab809db30 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab81c99f0 con 0x7fbab809f7e0 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab813b780 con 0x7fbab809e620 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabf33a640 10 monclient(hunting): authenticate will time out at 1027.078957s 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabd8d7640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbab809f7e0 legacy=0x7fbab81d1190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabd0d6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbab809db30 0x7fbab81c8bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52570/0 (socket says 172.21.15.63:52570) 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbabd0d6640 1 -- 172.21.15.63:0/1608504902 learned_addr learned my addr 172.21.15.63:0/1608504902 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009172198 0 0) 0x7fbab813ba60 con 0x7fbab809db30 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:10.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 10 monclient(hunting): my global_id is 4554 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba9c003530 con 0x7fbab809db30 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2361221955 0 0) 0x7fbab813b780 con 0x7fbab809e620 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 10 monclient(hunting): my global_id is 4553 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.296+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbab813ba60 con 0x7fbab809e620 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 148502925 0 0) 0x7fba9c003530 con 0x7fbab809db30 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbab813b780 con 0x7fbab809db30 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbab4004160 con 0x7fbab809db30 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient(hunting): dump: 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2570557781 0 0) 0x7fbab813ba60 con 0x7fbab809e620 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba9c003530 con 0x7fbab809e620 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbaa8001bd0 con 0x7fbab809e620 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient(hunting): dump: 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3641328122 0 0) 0x7fbab813b780 con 0x7fbab809db30 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 >> v1:172.21.15.6:6790/0 conn(0x7fbab809e620 legacy=0x7fbab81c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 >> v1:172.21.15.63:6789/0 conn(0x7fbab809f7e0 legacy=0x7fbab81d1190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 monclient: found mon.a 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab813a5b0 con 0x7fbab809db30 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 5 monclient: authenticate success, global_id 4554 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 10 monclient: _renew_subs 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 1 -- 172.21.15.63:0/1608504902 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbab813c4c0 con 0x7fbab809db30 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 10 monclient: _renew_subs 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 1 -- 172.21.15.63:0/1608504902 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbab81d2a80 con 0x7fbab809db30 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbab40043c0 con 0x7fbab809db30 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbab4004790 con 0x7fbab809db30 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaaf7fe640 4 set_mon_vals no callback set 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 10 monclient: dump: 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabf33a640 1 -- 172.21.15.63:0/1608504902 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba98005090 con 0x7fbab809db30 2024-09-16T05:36:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbab40049b0 con 0x7fbab809db30 2024-09-16T05:36:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(30..30 src has 1..30) ==== 3506+0+0 (unknown 3500684082 0 0) 0x7fbab40440b0 con 0x7fbab809db30 2024-09-16T05:36:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.300+0000 7fbabc8d5640 10 monclient: get_auth_request con 0x7fba9c036a20 auth_method 0 2024-09-16T05:36:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.312+0000 7fbaadffb640 1 -- 172.21.15.63:0/1608504902 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbab4015430 con 0x7fbab809db30 2024-09-16T05:36:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.312+0000 7fbaadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.312+0000 7fbaadffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:10.485 INFO:tasks.workunit.client.0.smithi063.stderr:foo-rule not valid: foo-rule not one of 'true', 'false' 2024-09-16T05:36:10.485 INFO:tasks.workunit.client.0.smithi063.stderr:Invalid command: unused arguments: ['foo-rule'] 2024-09-16T05:36:10.486 INFO:tasks.workunit.client.0.smithi063.stderr:osd pool create [] [] [] [] [] [] [] [] [] [] [--bulk] [] [] [--yes-i-really-mean-it] : create pool 2024-09-16T05:36:10.486 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T05:36:10.490 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:10.490 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create foo 123 123 replicated 2024-09-16T05:36:10.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.608+0000 7f829017e640 10 monclient: get_monmap_and_config 2024-09-16T05:36:10.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.608+0000 7f829017e640 10 monclient: build_initial_monmap 2024-09-16T05:36:10.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.608+0000 7f829017e640 10 monclient: monmap: 2024-09-16T05:36:10.614 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:10.614909+0000 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:10.614909+0000 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:10.615 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.608+0000 7f829017e640 1 Processor -- start 2024-09-16T05:36:10.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.608+0000 7f829017e640 1 -- start start 2024-09-16T05:36:10.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.608+0000 7f829017e640 10 monclient: init 2024-09-16T05:36:10.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:10.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:10.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient(hunting): picked mon.noname-c con 0x7f82880a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:10.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient(hunting): picked mon.noname-b con 0x7f82880a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:10.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient(hunting): picked mon.noname-a con 0x7f82880a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:10.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82880a53b0 con 0x7f82880a4600 2024-09-16T05:36:10.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82880a0290 con 0x7f82880a3b10 2024-09-16T05:36:10.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82880a0dc0 con 0x7f82880a07a0 2024-09-16T05:36:10.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:10.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient(hunting): authenticate will time out at 1027.392661s 2024-09-16T05:36:10.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828df1a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f82880a3b10 legacy=0x7f82880a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:10.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828e71b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f82880a4600 0x7f828813f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52584/0 (socket says 172.21.15.63:52584) 2024-09-16T05:36:10.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828d719640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f82880a07a0 0x7f828813be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46604/0 (socket says 172.21.15.63:46604) 2024-09-16T05:36:10.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828e71b640 1 -- 172.21.15.63:0/3474658380 learned_addr learned my addr 172.21.15.63:0/3474658380 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3968328041 0 0) 0x7f82880a53b0 con 0x7f82880a4600 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient(hunting): _negotiate 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient(hunting): my global_id is 4557 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8274003530 con 0x7f82880a4600 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455926904 0 0) 0x7f82880a0dc0 con 0x7f82880a07a0 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient(hunting): _negotiate 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient(hunting): my global_id is 4556 2024-09-16T05:36:10.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82880a53b0 con 0x7f82880a07a0 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3589976079 0 0) 0x7f8274003530 con 0x7f82880a4600 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 >> v1:172.21.15.6:6790/0 conn(0x7f82880a07a0 legacy=0x7f828813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 >> v1:172.21.15.63:6789/0 conn(0x7f82880a3b10 legacy=0x7f82880a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: _finish_hunting 0 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 monclient: found mon.noname-a 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82880a10a0 con 0x7f82880a4600 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: _finish_auth 0 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: _check_auth_tickets 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f82800019a0 con 0x7f82880a4600 2024-09-16T05:36:10.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: dump: 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 5 monclient: authenticate success, global_id 4557 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8280001c50 con 0x7f82880a4600 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 1 -- 172.21.15.63:0/3474658380 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8280002060 con 0x7f82880a4600 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f828cf18640 10 monclient: dump: 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.628 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 4 set_mon_vals no callback set 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient: shutdown 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 1 -- 172.21.15.63:0/3474658380 >> v1:172.21.15.6:6789/0 conn(0x7f82880a4600 legacy=0x7f828813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 1 -- 172.21.15.63:0/3474658380 shutdown_connections 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 1 -- 172.21.15.63:0/3474658380 >> 172.21.15.63:0/3474658380 conn(0x7f8288075470 msgr2=0x7f8288075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 1 -- 172.21.15.63:0/3474658380 shutdown_connections 2024-09-16T05:36:10.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 1 -- 172.21.15.63:0/3474658380 wait complete. 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient: build_initial_monmap 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.612+0000 7f829017e640 10 monclient: monmap: 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:10.630 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 1 Processor -- start 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 1 -- start start 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: init 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient(hunting): picked mon.noname-c con 0x7f82880a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient(hunting): picked mon.noname-a con 0x7f82880a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient(hunting): picked mon.noname-b con 0x7f82880a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8288127760 con 0x7f82880a3b10 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82880a87e0 con 0x7f82880a4600 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82880a82b0 con 0x7f82880a07a0 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient(hunting): authenticate will time out at 1027.397060s 2024-09-16T05:36:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f828e71b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f82880a4600 legacy=0x7f82881d3900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f828df1a640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f82880a07a0 0x7f8288136d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46612/0 (socket says 172.21.15.63:46612) 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f828df1a640 1 -- 172.21.15.63:0/717845030 learned_addr learned my addr 172.21.15.63:0/717845030 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4144108941 0 0) 0x7f8288127760 con 0x7f82880a3b10 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): my global_id is 4560 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f826c003530 con 0x7f82880a3b10 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440476696 0 0) 0x7f82880a82b0 con 0x7f82880a07a0 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): my global_id is 4559 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8288127760 con 0x7f82880a07a0 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2894960535 0 0) 0x7f826c003530 con 0x7f82880a3b10 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82880a82b0 con 0x7f82880a3b10 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8278001a80 con 0x7f82880a3b10 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): dump: 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.633 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1521533432 0 0) 0x7f8288127760 con 0x7f82880a07a0 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f826c003530 con 0x7f82880a07a0 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8284004160 con 0x7f82880a07a0 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient(hunting): dump: 2024-09-16T05:36:10.634 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3562771910 0 0) 0x7f82880a82b0 con 0x7f82880a3b10 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 >> v1:172.21.15.6:6790/0 conn(0x7f82880a07a0 legacy=0x7f8288136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 >> v1:172.21.15.63:6789/0 conn(0x7f82880a4600 legacy=0x7f82881d3900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:10.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 monclient: found mon.a 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82881271a0 con 0x7f82880a3b10 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 5 monclient: authenticate success, global_id 4560 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: _renew_subs 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8288127480 con 0x7f82880a3b10 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: _renew_subs 2024-09-16T05:36:10.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8288137480 con 0x7f82880a3b10 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8278001ce0 con 0x7f82880a3b10 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8278004c40 con 0x7f82880a3b10 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827ffff640 4 set_mon_vals no callback set 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f827e7fc640 10 monclient: dump: 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:10.637 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.616+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8270005090 con 0x7f82880a3b10 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.620+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f827800f9e0 con 0x7f82880a3b10 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.620+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(30..30 src has 1..30) ==== 3506+0+0 (unknown 3500684082 0 0) 0x7f8278044080 con 0x7f82880a3b10 2024-09-16T05:36:10.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.620+0000 7f828df1a640 10 monclient: get_auth_request con 0x7f826c036a70 auth_method 0 2024-09-16T05:36:10.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.632+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8278018f90 con 0x7f82880a3b10 2024-09-16T05:36:10.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.632+0000 7f827e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:10.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.632+0000 7f827e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:10.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.792+0000 7f829017e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T05:36:10.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.792+0000 7f829017e640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T05:36:10.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.792+0000 7f829017e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:10.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:10.792+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) -- 0x7f8270005380 con 0x7f82880a3b10 2024-09-16T05:36:11.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.048+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' created v31) ==== 154+0+0 (unknown 2502566099 0 0) 0x7f827801d850 con 0x7f82880a3b10 2024-09-16T05:36:11.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.048+0000 7f827e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T05:36:11.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.048+0000 7f827e7fc640 10 monclient: _finish_command 2 = system:0 pool 'foo' created 2024-09-16T05:36:11.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.156+0000 7f829017e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T05:36:11.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.156+0000 7f829017e640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T05:36:11.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.156+0000 7f829017e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:11.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.156+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) -- 0x7f8270001150 con 0x7f82880a3b10 2024-09-16T05:36:11.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.156+0000 7f827e7fc640 1 -- 172.21.15.63:0/717845030 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' already exists v31) ==== 161+0+0 (unknown 817130500 0 0) 0x7f827803ff90 con 0x7f82880a3b10 2024-09-16T05:36:11.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.156+0000 7f827e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T05:36:11.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.156+0000 7f827e7fc640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-16T05:36:11.162 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'foo' already exists 2024-09-16T05:36:11.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.160+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 >> v1:172.21.15.63:6812/3643207211 conn(0x7f826c036a70 legacy=0x7f826c038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:11.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.160+0000 7f829017e640 10 monclient: shutdown 2024-09-16T05:36:11.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.160+0000 7f829017e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:11.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.160+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 >> v1:172.21.15.6:6789/0 conn(0x7f82880a3b10 legacy=0x7f82881d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:11.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.160+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 shutdown_connections 2024-09-16T05:36:11.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.160+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 >> 172.21.15.63:0/717845030 conn(0x7f8288075470 msgr2=0x7f82880a78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:11.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.160+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 shutdown_connections 2024-09-16T05:36:11.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.160+0000 7f829017e640 1 -- 172.21.15.63:0/717845030 wait complete. 2024-09-16T05:36:11.178 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create fooo 123 123 erasure default 2024-09-16T05:36:11.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.296+0000 7f2543ed1640 10 monclient: get_monmap_and_config 2024-09-16T05:36:11.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.296+0000 7f2543ed1640 10 monclient: build_initial_monmap 2024-09-16T05:36:11.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.296+0000 7f2543ed1640 10 monclient: monmap: 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:11.303970+0000 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:11.303970+0000 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:11.304 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:11.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 Processor -- start 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 -- start start 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient: init 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient(hunting): picked mon.noname-c con 0x7f253c0a3c10 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient(hunting): picked mon.noname-a con 0x7f253c0a4700 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient(hunting): picked mon.noname-b con 0x7f253c13d330 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c0a0ed0 con 0x7f253c0a4700 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c0761f0 con 0x7f253c0a3c10 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c0a08a0 con 0x7f253c13d330 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient(hunting): authenticate will time out at 1028.081528s 2024-09-16T05:36:11.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2541c6d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f253c0a3c10 legacy=0x7f253c0a3ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:11.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f254246e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f253c13d330 0x7f253c13f720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46626/0 (socket says 172.21.15.63:46626) 2024-09-16T05:36:11.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f254246e640 1 -- 172.21.15.63:0/1250679643 learned_addr learned my addr 172.21.15.63:0/1250679643 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:11.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2560504043 0 0) 0x7f253c0a0ed0 con 0x7f253c0a4700 2024-09-16T05:36:11.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:11.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:11.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:11.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient(hunting): my global_id is 4563 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2528003530 con 0x7f253c0a4700 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 570423332 0 0) 0x7f253c0a08a0 con 0x7f253c13d330 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient(hunting): my global_id is 4562 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f253c0a0ed0 con 0x7f253c13d330 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3469283972 0 0) 0x7f253c0a0ed0 con 0x7f253c13d330 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 >> v1:172.21.15.63:6789/0 conn(0x7f253c0a3c10 legacy=0x7f253c0a3ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 >> v1:172.21.15.6:6789/0 conn(0x7f253c0a4700 legacy=0x7f253c13bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:11.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 monclient: found mon.noname-b 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c0a5400 con 0x7f253c13d330 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: _finish_auth 0 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2534001980 con 0x7f253c13d330 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: dump: 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:11.311 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 5 monclient: authenticate success, global_id 4562 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2534001be0 con 0x7f253c13d330 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:11.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 1 -- 172.21.15.63:0/1250679643 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2534002030 con 0x7f253c13d330 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2540c6b640 10 monclient: dump: 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:11.313 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 4 set_mon_vals no callback set 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 10 monclient: shutdown 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 -- 172.21.15.63:0/1250679643 >> v1:172.21.15.6:6790/0 conn(0x7f253c13d330 legacy=0x7f253c13f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 -- 172.21.15.63:0/1250679643 shutdown_connections 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 -- 172.21.15.63:0/1250679643 >> 172.21.15.63:0/1250679643 conn(0x7f253c075470 msgr2=0x7f253c075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.300+0000 7f2543ed1640 1 -- 172.21.15.63:0/1250679643 shutdown_connections 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 1 -- 172.21.15.63:0/1250679643 wait complete. 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: build_initial_monmap 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: monmap: 2024-09-16T05:36:11.314 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 1 Processor -- start 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 1 -- start start 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: init 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:11.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient(hunting): picked mon.noname-b con 0x7f253c0a3c10 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient(hunting): picked mon.noname-a con 0x7f253c0a4700 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient(hunting): picked mon.noname-c con 0x7f253c13d330 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c1cc2d0 con 0x7f253c0a4700 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c1228b0 con 0x7f253c0a3c10 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253c0a8970 con 0x7f253c13d330 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient(hunting): authenticate will time out at 1028.085447s 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2541c6d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f253c0a3c10 legacy=0x7f253c1cb4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f254246e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f253c13d330 0x7f253c1d3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46640/0 (socket says 172.21.15.63:46640) 2024-09-16T05:36:11.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f254246e640 1 -- 172.21.15.63:0/215979924 learned_addr learned my addr 172.21.15.63:0/215979924 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1215953564 0 0) 0x7f253c0a8970 con 0x7f253c13d330 2024-09-16T05:36:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): my global_id is 4565 2024-09-16T05:36:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2520003530 con 0x7f253c13d330 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 306637950 0 0) 0x7f253c1cc2d0 con 0x7f253c0a4700 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): my global_id is 4566 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f253c0a8970 con 0x7f253c0a4700 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3501314214 0 0) 0x7f2520003530 con 0x7f253c13d330 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f253c1cc2d0 con 0x7f253c13d330 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2534001f70 con 0x7f253c13d330 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): dump: 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 526856729 0 0) 0x7f253c0a8970 con 0x7f253c0a4700 2024-09-16T05:36:11.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2520003530 con 0x7f253c0a4700 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f252c001a40 con 0x7f253c0a4700 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient(hunting): dump: 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:11.322 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 825995519 0 0) 0x7f253c1cc2d0 con 0x7f253c13d330 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 >> v1:172.21.15.63:6789/0 conn(0x7f253c0a3c10 legacy=0x7f253c1cb4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 >> v1:172.21.15.6:6789/0 conn(0x7f253c0a4700 legacy=0x7f253c1cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 monclient: found mon.c 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c0a8420 con 0x7f253c13d330 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:11.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 5 monclient: authenticate success, global_id 4565 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: _renew_subs 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f253c123a80 con 0x7f253c13d330 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: _renew_subs 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f253c1237a0 con 0x7f253c13d330 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2534001be0 con 0x7f253c13d330 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2534005120 con 0x7f253c13d330 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f2533fff640 4 set_mon_vals no callback set 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.304+0000 7f25327fc640 10 monclient: dump: 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.308+0000 7f2543ed1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.308+0000 7f2543ed1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.308+0000 7f2543ed1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.308+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2524005090 con 0x7f253c13d330 2024-09-16T05:36:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.308+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2534005340 con 0x7f253c13d330 2024-09-16T05:36:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.308+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(31..31 src has 1..31) ==== 3848+0+0 (unknown 3490091569 0 0) 0x7f2534044620 con 0x7f253c13d330 2024-09-16T05:36:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.308+0000 7f2541c6d640 10 monclient: get_auth_request con 0x7f2520036a20 auth_method 0 2024-09-16T05:36:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.320+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2534015910 con 0x7f253c13d330 2024-09-16T05:36:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.320+0000 7f25327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.320+0000 7f25327fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:11.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.488+0000 7f2543ed1640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T05:36:11.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.488+0000 7f2543ed1640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T05:36:11.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.488+0000 7f2543ed1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:11.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:11.488+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) -- 0x7f2524005380 con 0x7f253c13d330 2024-09-16T05:36:13.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.076+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' created v33) ==== 188+0+0 (unknown 3742009765 0 0) 0x7f2534015210 con 0x7f253c13d330 2024-09-16T05:36:13.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.076+0000 7f25327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T05:36:13.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.076+0000 7f25327fc640 10 monclient: _finish_command 2 = system:0 pool 'fooo' created 2024-09-16T05:36:13.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.200+0000 7f2543ed1640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T05:36:13.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.200+0000 7f2543ed1640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T05:36:13.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.200+0000 7f2543ed1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:13.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.200+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) -- 0x7f2524001170 con 0x7f253c13d330 2024-09-16T05:36:13.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f25327fc640 1 -- 172.21.15.63:0/215979924 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' already exists v33) ==== 195+0+0 (unknown 4106932475 0 0) 0x7f2534042300 con 0x7f253c13d330 2024-09-16T05:36:13.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f25327fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T05:36:13.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f25327fc640 10 monclient: _finish_command 3 = system:0 pool 'fooo' already exists 2024-09-16T05:36:13.208 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'fooo' already exists 2024-09-16T05:36:13.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2520036a20 legacy=0x7f2520038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:13.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f2543ed1640 10 monclient: shutdown 2024-09-16T05:36:13.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f2543ed1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:13.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 >> v1:172.21.15.6:6790/0 conn(0x7f253c13d330 legacy=0x7f253c1d3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:13.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 shutdown_connections 2024-09-16T05:36:13.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 >> 172.21.15.63:0/215979924 conn(0x7f253c075470 msgr2=0x7f253c072700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:13.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 shutdown_connections 2024-09-16T05:36:13.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.204+0000 7f2543ed1640 1 -- 172.21.15.63:0/215979924 wait complete. 2024-09-16T05:36:13.226 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create foooo 123 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.340+0000 7fbc2702e640 10 monclient: get_monmap_and_config 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.340+0000 7fbc2702e640 10 monclient: build_initial_monmap 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.340+0000 7fbc2702e640 10 monclient: monmap: 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:13.348320+0000 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:13.348320+0000 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:13.348 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:13.349 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:13.349 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:13.349 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:13.349 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:13.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 Processor -- start 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- start start 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient: init 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc2009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc200a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc2009b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc20074200 con 0x7fbc2009b780 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2009f900 con 0x7fbc200a0280 2024-09-16T05:36:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2009e870 con 0x7fbc2009ac90 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient(hunting): authenticate will time out at 1030.125306s 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbc200a0280 legacy=0x7fbc201398b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc24dca640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbc2009ac90 0x7fbc2009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:46656/0 (socket says 172.21.15.63:46656) 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc24dca640 1 -- 172.21.15.63:0/3284145355 learned_addr learned my addr 172.21.15.63:0/3284145355 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724045920 0 0) 0x7fbc2009e870 con 0x7fbc2009ac90 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient(hunting): my global_id is 4568 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc08003530 con 0x7fbc2009ac90 2024-09-16T05:36:13.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 699480231 0 0) 0x7fbc20074200 con 0x7fbc2009b780 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient(hunting): my global_id is 4569 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc2009e870 con 0x7fbc2009b780 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2228926795 0 0) 0x7fbc08003530 con 0x7fbc2009ac90 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 >> v1:172.21.15.63:6789/0 conn(0x7fbc200a0280 legacy=0x7fbc201398b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 >> v1:172.21.15.6:6789/0 conn(0x7fbc2009b780 legacy=0x7fbc2013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 monclient: found mon.noname-b 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc20141730 con 0x7fbc2009ac90 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc14001620 con 0x7fbc2009ac90 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: dump: 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 5 monclient: authenticate success, global_id 4568 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc140018d0 con 0x7fbc2009ac90 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 1 -- 172.21.15.63:0/3284145355 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc14001ca0 con 0x7fbc2009ac90 2024-09-16T05:36:13.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 4 set_mon_vals no callback set 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc1f7fe640 10 monclient: dump: 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:13.361 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 10 monclient: shutdown 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- 172.21.15.63:0/3284145355 >> v1:172.21.15.6:6790/0 conn(0x7fbc2009ac90 legacy=0x7fbc2009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- 172.21.15.63:0/3284145355 shutdown_connections 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- 172.21.15.63:0/3284145355 >> 172.21.15.63:0/3284145355 conn(0x7fbc200a5290 msgr2=0x7fbc200a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:13.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- 172.21.15.63:0/3284145355 shutdown_connections 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.344+0000 7fbc2702e640 1 -- 172.21.15.63:0/3284145355 wait complete. 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: build_initial_monmap 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: monmap: 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:13.363 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 1 Processor -- start 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 1 -- start start 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: init 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc2009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc2009b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc200a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:13.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2009c0f0 con 0x7fbc2009ac90 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc20140330 con 0x7fbc200a0280 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2013d4a0 con 0x7fbc2009b780 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient(hunting): authenticate will time out at 1030.128911s 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc255cb640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbc200a0280 legacy=0x7fbc201d36b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc24dca640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbc2009ac90 0x7fbc2013c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:52632/0 (socket says 172.21.15.63:52632) 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc24dca640 1 -- 172.21.15.63:0/166806843 learned_addr learned my addr 172.21.15.63:0/166806843 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927973549 0 0) 0x7fbc20141730 con 0x7fbc2009ac90 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:13.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): my global_id is 4572 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc0c003530 con 0x7fbc2009ac90 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575630636 0 0) 0x7fbc2013d4a0 con 0x7fbc2009b780 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): my global_id is 4571 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc20141730 con 0x7fbc2009b780 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1297309173 0 0) 0x7fbc0c003530 con 0x7fbc2009ac90 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc2013d4a0 con 0x7fbc2009ac90 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1767781741 0 0) 0x7fbc20141730 con 0x7fbc2009b780 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc0c003530 con 0x7fbc2009b780 2024-09-16T05:36:13.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc14001c20 con 0x7fbc2009ac90 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): dump: 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:13.367 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc100040d0 con 0x7fbc2009b780 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient(hunting): dump: 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:13.368 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2484624527 0 0) 0x7fbc2013d4a0 con 0x7fbc2009ac90 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 >> v1:172.21.15.6:6790/0 conn(0x7fbc2009b780 legacy=0x7fbc2013cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 >> v1:172.21.15.63:6789/0 conn(0x7fbc200a0280 legacy=0x7fbc201d36b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 monclient: found mon.a 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2013f0a0 con 0x7fbc2009ac90 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:36:13.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 5 monclient: authenticate success, global_id 4572 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: _renew_subs 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc20140050 con 0x7fbc2009ac90 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: _renew_subs 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc20140d90 con 0x7fbc2009ac90 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc14004c50 con 0x7fbc2009ac90 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc14005020 con 0x7fbc2009ac90 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1effd640 4 set_mon_vals no callback set 2024-09-16T05:36:13.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc1d7fa640 10 monclient: dump: 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:13.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:13.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:13.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.348+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc00005090 con 0x7fbc2009ac90 2024-09-16T05:36:13.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.352+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbc14005240 con 0x7fbc2009ac90 2024-09-16T05:36:13.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.352+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(33..33 src has 1..33) ==== 4290+0+0 (unknown 367616029 0 0) 0x7fbc140445b0 con 0x7fbc2009ac90 2024-09-16T05:36:13.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.352+0000 7fbc1ffff640 10 monclient: get_auth_request con 0x7fbc0c0369c0 auth_method 0 2024-09-16T05:36:13.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.364+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbc2009c0f0 con 0x7fbc2009ac90 2024-09-16T05:36:13.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.364+0000 7fbc1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:13.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.364+0000 7fbc1d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:13.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.528+0000 7fbc2702e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T05:36:13.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.528+0000 7fbc2702e640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T05:36:13.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.528+0000 7fbc2702e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:13.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:13.528+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) -- 0x7fbc00005b90 con 0x7fbc2009ac90 2024-09-16T05:36:14.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.080+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' created v34) ==== 115+0+0 (unknown 1597986720 0 0) 0x7fbc14015780 con 0x7fbc2009ac90 2024-09-16T05:36:14.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.080+0000 7fbc1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T05:36:14.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.080+0000 7fbc1d7fa640 10 monclient: _finish_command 2 = system:0 pool 'foooo' created 2024-09-16T05:36:14.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.248+0000 7fbc2702e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T05:36:14.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.248+0000 7fbc2702e640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T05:36:14.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.248+0000 7fbc2702e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.248+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) -- 0x7fbc00005610 con 0x7fbc2009ac90 2024-09-16T05:36:14.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.248+0000 7fbc1d7fa640 1 -- 172.21.15.63:0/166806843 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' already exists v34) ==== 122+0+0 (unknown 1981446773 0 0) 0x7fbc14015080 con 0x7fbc2009ac90 2024-09-16T05:36:14.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.248+0000 7fbc1d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T05:36:14.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.248+0000 7fbc1d7fa640 10 monclient: _finish_command 3 = system:0 pool 'foooo' already exists 2024-09-16T05:36:14.255 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'foooo' already exists 2024-09-16T05:36:14.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.252+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbc0c0369c0 legacy=0x7fbc0c038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.252+0000 7fbc2702e640 10 monclient: shutdown 2024-09-16T05:36:14.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.252+0000 7fbc2702e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:14.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.252+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 >> v1:172.21.15.6:6789/0 conn(0x7fbc2009ac90 legacy=0x7fbc2013c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.252+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 shutdown_connections 2024-09-16T05:36:14.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.252+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 >> 172.21.15.63:0/166806843 conn(0x7fbc200a5290 msgr2=0x7fbc200a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:14.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.252+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 shutdown_connections 2024-09-16T05:36:14.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.252+0000 7fbc2702e640 1 -- 172.21.15.63:0/166806843 wait complete. 2024-09-16T05:36:14.272 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create foo 123 2024-09-16T05:36:14.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient: get_monmap_and_config 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient: build_initial_monmap 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient: monmap: 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:14.393053+0000 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:14.393053+0000 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:14.393 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:14.394 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:14.394 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 1 Processor -- start 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 1 -- start start 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient: init 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient(hunting): picked mon.noname-b con 0x7fd74409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient(hunting): picked mon.noname-a con 0x7fd7440a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient(hunting): picked mon.noname-c con 0x7fd74409b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:14.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd744074200 con 0x7fd7440a0280 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74409f900 con 0x7fd74409b780 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74409e870 con 0x7fd74409ac90 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient(hunting): authenticate will time out at 1031.170078s 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd748d3c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd74409b780 legacy=0x7fd74413cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd743fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd74409ac90 0x7fd74409b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43172/0 (socket says 172.21.15.63:43172) 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd743fff640 1 -- 172.21.15.63:0/2363348929 learned_addr learned my addr 172.21.15.63:0/2363348929 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd7437fe640 1 --1- 172.21.15.63:0/2363348929 >> v1:172.21.15.6:6789/0 conn(0x7fd7440a0280 0x7fd7441398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58686/0 (socket says 172.21.15.63:58686) 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110561515 0 0) 0x7fd74409e870 con 0x7fd74409ac90 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient(hunting): my global_id is 4574 2024-09-16T05:36:14.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd730003530 con 0x7fd74409ac90 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525921434 0 0) 0x7fd744074200 con 0x7fd7440a0280 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient(hunting): my global_id is 4575 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd74409e870 con 0x7fd7440a0280 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3391269372 0 0) 0x7fd730003530 con 0x7fd74409ac90 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 >> v1:172.21.15.63:6789/0 conn(0x7fd74409b780 legacy=0x7fd74413cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 >> v1:172.21.15.6:6789/0 conn(0x7fd7440a0280 legacy=0x7fd7441398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 monclient: found mon.noname-b 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd744141730 con 0x7fd74409ac90 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd734001620 con 0x7fd74409ac90 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: dump: 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.400 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 5 monclient: authenticate success, global_id 4574 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7340018d0 con 0x7fd74409ac90 2024-09-16T05:36:14.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 1 -- 172.21.15.63:0/2363348929 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd734001ca0 con 0x7fd74409ac90 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd74a79f640 4 set_mon_vals no callback set 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.388+0000 7fd742ffd640 10 monclient: dump: 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.402 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: shutdown 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- 172.21.15.63:0/2363348929 >> v1:172.21.15.6:6790/0 conn(0x7fd74409ac90 legacy=0x7fd74409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- 172.21.15.63:0/2363348929 shutdown_connections 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- 172.21.15.63:0/2363348929 >> 172.21.15.63:0/2363348929 conn(0x7fd7440a5290 msgr2=0x7fd7440a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- 172.21.15.63:0/2363348929 shutdown_connections 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- 172.21.15.63:0/2363348929 wait complete. 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: build_initial_monmap 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: monmap: 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:14.403 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 Processor -- start 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- start start 2024-09-16T05:36:14.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: init 2024-09-16T05:36:14.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:14.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:14.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient(hunting): picked mon.noname-a con 0x7fd74409ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:14.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient(hunting): picked mon.noname-b con 0x7fd74409b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:14.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient(hunting): picked mon.noname-c con 0x7fd7440a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:14.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd744140900 con 0x7fd74409ac90 2024-09-16T05:36:14.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74413fc50 con 0x7fd74409b780 2024-09-16T05:36:14.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74409e000 con 0x7fd7440a0280 2024-09-16T05:36:14.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:14.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient(hunting): authenticate will time out at 1031.174002s 2024-09-16T05:36:14.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd7437fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd74409b780 legacy=0x7fd74409cb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:14.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd748d3c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd7440a0280 0x7fd7441d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43184/0 (socket says 172.21.15.63:43184) 2024-09-16T05:36:14.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd748d3c640 1 -- 172.21.15.63:0/3580058074 learned_addr learned my addr 172.21.15.63:0/3580058074 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:14.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776573483 0 0) 0x7fd744140900 con 0x7fd74409ac90 2024-09-16T05:36:14.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:14.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:14.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:14.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): my global_id is 4578 2024-09-16T05:36:14.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd724003530 con 0x7fd74409ac90 2024-09-16T05:36:14.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1990276387 0 0) 0x7fd74409e000 con 0x7fd7440a0280 2024-09-16T05:36:14.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:14.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:14.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:14.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): my global_id is 4577 2024-09-16T05:36:14.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd744140900 con 0x7fd7440a0280 2024-09-16T05:36:14.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 525686867 0 0) 0x7fd724003530 con 0x7fd74409ac90 2024-09-16T05:36:14.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd74409e000 con 0x7fd74409ac90 2024-09-16T05:36:14.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd734001c20 con 0x7fd74409ac90 2024-09-16T05:36:14.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:14.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:14.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:14.417 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.417 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.417 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.417 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.418 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.418 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.418 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.418 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.418 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.418 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:14.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1433151096 0 0) 0x7fd744140900 con 0x7fd7440a0280 2024-09-16T05:36:14.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd724003530 con 0x7fd7440a0280 2024-09-16T05:36:14.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd73c0043c0 con 0x7fd7440a0280 2024-09-16T05:36:14.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:14.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:14.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:14.420 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.420 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.420 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.421 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.421 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.421 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.421 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.421 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.422 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.422 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1460391129 0 0) 0x7fd74409e000 con 0x7fd74409ac90 2024-09-16T05:36:14.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 >> v1:172.21.15.6:6790/0 conn(0x7fd7440a0280 legacy=0x7fd7441d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 >> v1:172.21.15.63:6789/0 conn(0x7fd74409b780 legacy=0x7fd74409cb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:14.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:14.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 monclient: found mon.a 2024-09-16T05:36:14.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:14.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74409d220 con 0x7fd74409ac90 2024-09-16T05:36:14.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:14.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:14.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd740ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:14.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 5 monclient: authenticate success, global_id 4578 2024-09-16T05:36:14.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: _renew_subs 2024-09-16T05:36:14.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd744141360 con 0x7fd74409ac90 2024-09-16T05:36:14.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: _renew_subs 2024-09-16T05:36:14.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.392+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd74413f620 con 0x7fd74409ac90 2024-09-16T05:36:14.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7340018d0 con 0x7fd74409ac90 2024-09-16T05:36:14.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd740ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:14.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd734004c50 con 0x7fd74409ac90 2024-09-16T05:36:14.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd740ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:14.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd740ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:14.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd7427fc640 4 set_mon_vals no callback set 2024-09-16T05:36:14.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd740ff9640 10 monclient: dump: 2024-09-16T05:36:14.426 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.426 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.426 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.426 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.426 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.427 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.427 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.427 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.427 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.427 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd74a79f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:14.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd74a79f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:14.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd74a79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd728005090 con 0x7fd74409ac90 2024-09-16T05:36:14.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd73400f9f0 con 0x7fd74409ac90 2024-09-16T05:36:14.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(34..34 src has 1..34) ==== 4634+0+0 (unknown 1845097811 0 0) 0x7fd734044550 con 0x7fd74409ac90 2024-09-16T05:36:14.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.396+0000 7fd7437fe640 10 monclient: get_auth_request con 0x7fd7240369d0 auth_method 0 2024-09-16T05:36:14.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.408+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd734015470 con 0x7fd74409ac90 2024-09-16T05:36:14.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.408+0000 7fd740ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:14.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.408+0000 7fd740ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:14.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.576+0000 7fd74a79f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T05:36:14.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.576+0000 7fd74a79f640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T05:36:14.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.576+0000 7fd74a79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.576+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) -- 0x7fd728005b90 con 0x7fd74409ac90 2024-09-16T05:36:14.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.576+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v34) ==== 118+0+0 (unknown 3490867400 0 0) 0x7fd734014d70 con 0x7fd74409ac90 2024-09-16T05:36:14.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.576+0000 7fd740ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T05:36:14.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.576+0000 7fd740ff9640 10 monclient: _finish_command 2 = system:0 pool 'foo' already exists 2024-09-16T05:36:14.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.676+0000 7fd74a79f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T05:36:14.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.676+0000 7fd74a79f640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T05:36:14.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.676+0000 7fd74a79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.676+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) -- 0x7fd728005610 con 0x7fd74409ac90 2024-09-16T05:36:14.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.676+0000 7fd740ff9640 1 -- 172.21.15.63:0/3580058074 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v34) ==== 118+0+0 (unknown 3490867400 0 0) 0x7fd73401d920 con 0x7fd74409ac90 2024-09-16T05:36:14.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.676+0000 7fd740ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T05:36:14.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.676+0000 7fd740ff9640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-16T05:36:14.684 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'foo' already exists 2024-09-16T05:36:14.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.680+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd7240369d0 legacy=0x7fd724038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.680+0000 7fd74a79f640 10 monclient: shutdown 2024-09-16T05:36:14.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.680+0000 7fd74a79f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:14.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.680+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 >> v1:172.21.15.6:6789/0 conn(0x7fd74409ac90 legacy=0x7fd74409c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.680+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 shutdown_connections 2024-09-16T05:36:14.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.680+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 >> 172.21.15.63:0/3580058074 conn(0x7fd7440a5290 msgr2=0x7fd7440a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:14.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.680+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 shutdown_connections 2024-09-16T05:36:14.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.680+0000 7fd74a79f640 1 -- 172.21.15.63:0/3580058074 wait complete. 2024-09-16T05:36:14.701 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.820+0000 7f49dc89c640 10 monclient: get_monmap_and_config 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.820+0000 7f49dc89c640 10 monclient: build_initial_monmap 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.820+0000 7f49dc89c640 10 monclient: monmap: 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:14.827054+0000 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:14.827054+0000 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.827 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:14.828 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:14.828 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:14.828 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.820+0000 7f49dc89c640 1 Processor -- start 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.820+0000 7f49dc89c640 1 -- start start 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.820+0000 7f49dc89c640 10 monclient: init 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient(hunting): picked mon.noname-a con 0x7f49d80a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient(hunting): picked mon.noname-b con 0x7f49d80a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient(hunting): picked mon.noname-c con 0x7f49d80a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d80a53b0 con 0x7f49d80a3b10 2024-09-16T05:36:14.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d80a0290 con 0x7f49d80a4600 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d80a0dc0 con 0x7f49d80a07a0 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d6d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f49d80a4600 legacy=0x7f49d813f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient(hunting): authenticate will time out at 1031.604279s 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d5d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f49d80a07a0 0x7f49d813be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43198/0 (socket says 172.21.15.63:43198) 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d5d9b640 1 -- 172.21.15.63:0/845364228 learned_addr learned my addr 172.21.15.63:0/845364228 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092264609 0 0) 0x7f49d80a53b0 con 0x7f49d80a3b10 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:14.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient(hunting): my global_id is 4581 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49c0003530 con 0x7f49d80a3b10 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1068950026 0 0) 0x7f49d80a0dc0 con 0x7f49d80a07a0 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient(hunting): my global_id is 4580 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49d80a53b0 con 0x7f49d80a07a0 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3513777292 0 0) 0x7f49c0003530 con 0x7f49d80a3b10 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 >> v1:172.21.15.6:6790/0 conn(0x7f49d80a07a0 legacy=0x7f49d813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 >> v1:172.21.15.63:6789/0 conn(0x7f49d80a4600 legacy=0x7f49d813f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:14.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 monclient: found mon.noname-a 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d80a10a0 con 0x7f49d80a3b10 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: _finish_auth 0 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f49c8001620 con 0x7f49d80a3b10 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 5 monclient: authenticate success, global_id 4581 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: dump: 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.840 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:14.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f49c8001880 con 0x7f49d80a3b10 2024-09-16T05:36:14.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:14.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 1 -- 172.21.15.63:0/845364228 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f49c8001bd0 con 0x7f49d80a3b10 2024-09-16T05:36:14.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:14.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:14.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49d559a640 10 monclient: dump: 2024-09-16T05:36:14.841 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 4 set_mon_vals no callback set 2024-09-16T05:36:14.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient: shutdown 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- 172.21.15.63:0/845364228 >> v1:172.21.15.6:6789/0 conn(0x7f49d80a3b10 legacy=0x7f49d80a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- 172.21.15.63:0/845364228 shutdown_connections 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- 172.21.15.63:0/845364228 >> 172.21.15.63:0/845364228 conn(0x7f49d8075470 msgr2=0x7f49d8075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- 172.21.15.63:0/845364228 shutdown_connections 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- 172.21.15.63:0/845364228 wait complete. 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient: build_initial_monmap 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient: monmap: 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:14.843 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 Processor -- start 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 1 -- start start 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.824+0000 7f49dc89c640 10 monclient: init 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:14.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient(hunting): picked mon.noname-b con 0x7f49d80a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient(hunting): picked mon.noname-a con 0x7f49d80a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient(hunting): picked mon.noname-c con 0x7f49d80a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d8121510 con 0x7f49d80a3b10 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d8123800 con 0x7f49d80a07a0 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d80a0500 con 0x7f49d80a4600 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient(hunting): authenticate will time out at 1031.608109s 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49d659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f49d80a07a0 legacy=0x7f49d8136050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:14.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49d6d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f49d80a4600 0x7f49d81d3db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43206/0 (socket says 172.21.15.63:43206) 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49d5d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f49d80a3b10 0x7f49d81d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58700/0 (socket says 172.21.15.63:58700) 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49d6d9d640 1 -- 172.21.15.63:0/3394867299 learned_addr learned my addr 172.21.15.63:0/3394867299 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828186179 0 0) 0x7f49d80a0500 con 0x7f49d80a4600 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): my global_id is 4583 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b8003530 con 0x7f49d80a4600 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1873665762 0 0) 0x7f49d8121510 con 0x7f49d80a3b10 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:14.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): my global_id is 4584 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49d80a0500 con 0x7f49d80a3b10 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 361576182 0 0) 0x7f49d80a0500 con 0x7f49d80a3b10 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49d8121510 con 0x7f49d80a3b10 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f49cc0018c0 con 0x7f49d80a3b10 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.847 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2183349466 0 0) 0x7f49b8003530 con 0x7f49d80a4600 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49d80a0500 con 0x7f49d80a4600 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f49d00043c0 con 0x7f49d80a4600 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:14.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 437439168 0 0) 0x7f49d8121510 con 0x7f49d80a3b10 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 >> v1:172.21.15.6:6790/0 conn(0x7f49d80a4600 legacy=0x7f49d81d3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:14.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 >> v1:172.21.15.63:6789/0 conn(0x7f49d80a07a0 legacy=0x7f49d8136050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 monclient: found mon.a 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d81cba60 con 0x7f49d80a3b10 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 5 monclient: authenticate success, global_id 4584 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient: _renew_subs 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49d8123520 con 0x7f49d80a3b10 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient: _renew_subs 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f49d8136760 con 0x7f49d80a3b10 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f49cc0047f0 con 0x7f49d80a3b10 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49d4d99640 4 set_mon_vals no callback set 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f49cc004b40 con 0x7f49d80a3b10 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 10 monclient: dump: 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:14.851 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:14.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49bc005090 con 0x7f49d80a3b10 2024-09-16T05:36:14.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f49cc00f8e0 con 0x7f49d80a3b10 2024-09-16T05:36:14.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(34..34 src has 1..34) ==== 4634+0+0 (unknown 1845097811 0 0) 0x7f49cc0444a0 con 0x7f49d80a3b10 2024-09-16T05:36:14.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.828+0000 7f49d659c640 10 monclient: get_auth_request con 0x7f49b8036a20 auth_method 0 2024-09-16T05:36:14.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.840+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f49cc015380 con 0x7f49d80a3b10 2024-09-16T05:36:14.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.844+0000 7f49c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:14.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:14.844+0000 7f49c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:15.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.008+0000 7f49dc89c640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T05:36:15.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.008+0000 7f49dc89c640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T05:36:15.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.008+0000 7f49dc89c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:15.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.008+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) -- 0x7f49bc005380 con 0x7f49d80a3b10 2024-09-16T05:36:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.088+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v35) ==== 152+0+0 (unknown 1997080733 0 0) 0x7f49cc014c80 con 0x7f49d80a3b10 2024-09-16T05:36:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.088+0000 7f49c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T05:36:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.088+0000 7f49c6ffd640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 1 2024-09-16T05:36:15.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.236+0000 7f49dc89c640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T05:36:15.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.236+0000 7f49dc89c640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T05:36:15.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.236+0000 7f49dc89c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:15.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:15.236+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) -- 0x7f49bc001150 con 0x7f49d80a3b10 2024-09-16T05:36:16.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.096+0000 7f49c6ffd640 1 -- 172.21.15.63:0/3394867299 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v36) ==== 152+0+0 (unknown 4018695931 0 0) 0x7f49cc01d890 con 0x7f49d80a3b10 2024-09-16T05:36:16.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.096+0000 7f49c6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T05:36:16.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.096+0000 7f49c6ffd640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 1 2024-09-16T05:36:16.104 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 4 size to 1 2024-09-16T05:36:16.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.100+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 >> v1:172.21.15.63:6812/3643207211 conn(0x7f49b8036a20 legacy=0x7f49b8038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:16.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.100+0000 7f49dc89c640 10 monclient: shutdown 2024-09-16T05:36:16.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.100+0000 7f49dc89c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:16.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.100+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 >> v1:172.21.15.6:6789/0 conn(0x7f49d80a3b10 legacy=0x7f49d81d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:16.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.100+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 shutdown_connections 2024-09-16T05:36:16.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.100+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 >> 172.21.15.63:0/3394867299 conn(0x7f49d8075470 msgr2=0x7f49d81beba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:16.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.100+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 shutdown_connections 2024-09-16T05:36:16.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.100+0000 7f49dc89c640 1 -- 172.21.15.63:0/3394867299 wait complete. 2024-09-16T05:36:16.139 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_config_value foo min_size 1 2024-09-16T05:36:16.139 INFO:tasks.workunit.client.0.smithi063.stderr:+ local pool_name config_opt expected_val val 2024-09-16T05:36:16.139 INFO:tasks.workunit.client.0.smithi063.stderr:+ pool_name=foo 2024-09-16T05:36:16.139 INFO:tasks.workunit.client.0.smithi063.stderr:+ config_opt=min_size 2024-09-16T05:36:16.139 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_val=1 2024-09-16T05:36:16.140 INFO:tasks.workunit.client.0.smithi063.stderr:++ get_config_value_or_die foo min_size 2024-09-16T05:36:16.141 INFO:tasks.workunit.client.0.smithi063.stderr:++ local pool_name config_opt raw val 2024-09-16T05:36:16.141 INFO:tasks.workunit.client.0.smithi063.stderr:++ pool_name=foo 2024-09-16T05:36:16.141 INFO:tasks.workunit.client.0.smithi063.stderr:++ config_opt=min_size 2024-09-16T05:36:16.141 INFO:tasks.workunit.client.0.smithi063.stderr:+++ ceph osd pool get foo min_size 2024-09-16T05:36:16.568 INFO:tasks.workunit.client.0.smithi063.stderr:++ raw='min_size: 1' 2024-09-16T05:36:16.569 INFO:tasks.workunit.client.0.smithi063.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T05:36:16.569 INFO:tasks.workunit.client.0.smithi063.stderr:+++ echo min_size: 1 2024-09-16T05:36:16.569 INFO:tasks.workunit.client.0.smithi063.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T05:36:16.572 INFO:tasks.workunit.client.0.smithi063.stderr:++ raw=min_size:1 2024-09-16T05:36:16.572 INFO:tasks.workunit.client.0.smithi063.stderr:+++ echo min_size:1 2024-09-16T05:36:16.572 INFO:tasks.workunit.client.0.smithi063.stderr:+++ cut -f2 -d: 2024-09-16T05:36:16.573 INFO:tasks.workunit.client.0.smithi063.stderr:++ val=1 2024-09-16T05:36:16.574 INFO:tasks.workunit.client.0.smithi063.stderr:++ echo 1 2024-09-16T05:36:16.574 INFO:tasks.workunit.client.0.smithi063.stderr:++ return 0 2024-09-16T05:36:16.574 INFO:tasks.workunit.client.0.smithi063.stderr:+ val=1 2024-09-16T05:36:16.574 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 1 != \1 ]] 2024-09-16T05:36:16.574 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set foo size 4 2024-09-16T05:36:16.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.688+0000 7f733e51c640 10 monclient: get_monmap_and_config 2024-09-16T05:36:16.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.688+0000 7f733e51c640 10 monclient: build_initial_monmap 2024-09-16T05:36:16.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.688+0000 7f733e51c640 10 monclient: monmap: 2024-09-16T05:36:16.697 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:16.697 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:16.697 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:16.696292+0000 2024-09-16T05:36:16.697 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:16.696292+0000 2024-09-16T05:36:16.697 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:16.698 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:16.698 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:16.698 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:16.698 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:16.698 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:16.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 Processor -- start 2024-09-16T05:36:16.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- start start 2024-09-16T05:36:16.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient: init 2024-09-16T05:36:16.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:16.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient(hunting): picked mon.noname-c con 0x7f7338136d20 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient(hunting): picked mon.noname-a con 0x7f73380a4b20 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient(hunting): picked mon.noname-b con 0x7f73380a1d20 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73380762a0 con 0x7f73380a4b20 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73380a4ff0 con 0x7f7338136d20 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7338137810 con 0x7f73380a1d20 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient(hunting): authenticate will time out at 1033.473246s 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7337fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7338136d20 legacy=0x7f7338137100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733cab9640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f73380a1d20 0x7f733813f3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43254/0 (socket says 172.21.15.63:43254) 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733cab9640 1 -- 172.21.15.63:0/4038097618 learned_addr learned my addr 172.21.15.63:0/4038097618 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:16.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 55035565 0 0) 0x7f7338137810 con 0x7f73380a1d20 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient(hunting): my global_id is 4592 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7320003530 con 0x7f73380a1d20 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2074950970 0 0) 0x7f73380762a0 con 0x7f73380a4b20 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient(hunting): my global_id is 4593 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7338137810 con 0x7f73380a4b20 2024-09-16T05:36:16.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2253468555 0 0) 0x7f7338137810 con 0x7f73380a4b20 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 >> v1:172.21.15.6:6790/0 conn(0x7f73380a1d20 legacy=0x7f733813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 >> v1:172.21.15.63:6789/0 conn(0x7f7338136d20 legacy=0x7f7338137100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 monclient: found mon.noname-a 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73380a6980 con 0x7f73380a4b20 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f732c001600 con 0x7f73380a4b20 2024-09-16T05:36:16.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 5 monclient: authenticate success, global_id 4593 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: dump: 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:16.705 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f732c001860 con 0x7f73380a4b20 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 4 set_mon_vals no callback set 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 1 -- 172.21.15.63:0/4038097618 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f732c001cb0 con 0x7f73380a4b20 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f7336ffd640 10 monclient: dump: 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:16.706 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 10 monclient: shutdown 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- 172.21.15.63:0/4038097618 >> v1:172.21.15.6:6789/0 conn(0x7f73380a4b20 legacy=0x7f733813bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- 172.21.15.63:0/4038097618 shutdown_connections 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- 172.21.15.63:0/4038097618 >> 172.21.15.63:0/4038097618 conn(0x7f73380a6310 msgr2=0x7f7338075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:16.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- 172.21.15.63:0/4038097618 shutdown_connections 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.692+0000 7f733e51c640 1 -- 172.21.15.63:0/4038097618 wait complete. 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: build_initial_monmap 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: monmap: 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:16.708 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 1 Processor -- start 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 1 -- start start 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: init 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient(hunting): picked mon.noname-b con 0x7f73380a1d20 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient(hunting): picked mon.noname-c con 0x7f73380a4b20 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient(hunting): picked mon.noname-a con 0x7f7338136d20 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7337fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f73380a1d20 legacy=0x7f73381342e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7338135ce0 con 0x7f7338136d20 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73380a0500 con 0x7f73380a1d20 2024-09-16T05:36:16.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7338140740 con 0x7f73380a4b20 2024-09-16T05:36:16.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:16.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient(hunting): authenticate will time out at 1033.477382s 2024-09-16T05:36:16.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f73377fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f73380a4b20 0x7f73381349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43260/0 (socket says 172.21.15.63:43260) 2024-09-16T05:36:16.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733cab9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7338136d20 0x7f73381d3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58748/0 (socket says 172.21.15.63:58748) 2024-09-16T05:36:16.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f73377fe640 1 -- 172.21.15.63:0/2038636651 learned_addr learned my addr 172.21.15.63:0/2038636651 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:16.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124439039 0 0) 0x7f7338140740 con 0x7f73380a4b20 2024-09-16T05:36:16.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:16.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:16.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): my global_id is 4595 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7324003530 con 0x7f73380a4b20 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1148415769 0 0) 0x7f7338135ce0 con 0x7f7338136d20 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): my global_id is 4596 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7338140740 con 0x7f7338136d20 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 245702526 0 0) 0x7f7338140740 con 0x7f7338136d20 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7338135ce0 con 0x7f7338136d20 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7330001b80 con 0x7f7338136d20 2024-09-16T05:36:16.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:16.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3041280032 0 0) 0x7f7324003530 con 0x7f73380a4b20 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7338140740 con 0x7f73380a4b20 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f732c0036c0 con 0x7f73380a4b20 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:16.715 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1394109524 0 0) 0x7f7338135ce0 con 0x7f7338136d20 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 >> v1:172.21.15.6:6790/0 conn(0x7f73380a4b20 legacy=0x7f73381349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 >> v1:172.21.15.63:6789/0 conn(0x7f73380a1d20 legacy=0x7f73381342e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 monclient: found mon.a 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7338138370 con 0x7f7338136d20 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:16.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 5 monclient: authenticate success, global_id 4596 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: _renew_subs 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7338138650 con 0x7f7338136d20 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: _renew_subs 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7338135100 con 0x7f7338136d20 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7330002090 con 0x7f7338136d20 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f73367fc640 4 set_mon_vals no callback set 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7330004d10 con 0x7f7338136d20 2024-09-16T05:36:16.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.696+0000 7f7334ff9640 10 monclient: dump: 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:16.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.700+0000 7f733e51c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:16.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.700+0000 7f733e51c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:16.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.700+0000 7f733e51c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:16.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.700+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7318005090 con 0x7f7338136d20 2024-09-16T05:36:16.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.700+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7330004f30 con 0x7f7338136d20 2024-09-16T05:36:16.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.700+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(36..36 src has 1..36) ==== 4634+0+0 (unknown 4119616521 0 0) 0x7f73300446c0 con 0x7f7338136d20 2024-09-16T05:36:16.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.700+0000 7f7337fff640 10 monclient: get_auth_request con 0x7f73240369d0 auth_method 0 2024-09-16T05:36:16.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.712+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7330015520 con 0x7f7338136d20 2024-09-16T05:36:16.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.712+0000 7f7334ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:16.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.712+0000 7f7334ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:16.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.876+0000 7f733e51c640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T05:36:16.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.876+0000 7f733e51c640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T05:36:16.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.876+0000 7f733e51c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:16.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:16.876+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) -- 0x7f7318005b90 con 0x7f7338136d20 2024-09-16T05:36:17.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:17.104+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v37) ==== 122+0+0 (unknown 1743069073 0 0) 0x7f73300191d0 con 0x7f7338136d20 2024-09-16T05:36:17.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:17.104+0000 7f7334ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T05:36:17.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:17.104+0000 7f7334ff9640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 4 2024-09-16T05:36:17.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:17.228+0000 7f733e51c640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T05:36:17.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:17.228+0000 7f733e51c640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T05:36:17.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:17.228+0000 7f733e51c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:17.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:17.228+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) -- 0x7f7318005610 con 0x7f7338136d20 2024-09-16T05:36:18.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.112+0000 7f7334ff9640 1 -- 172.21.15.63:0/2038636651 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v38) ==== 122+0+0 (unknown 3850864433 0 0) 0x7f733001da90 con 0x7f7338136d20 2024-09-16T05:36:18.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.112+0000 7f7334ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T05:36:18.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.112+0000 7f7334ff9640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 4 2024-09-16T05:36:18.119 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 4 size to 4 2024-09-16T05:36:18.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.116+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 >> v1:172.21.15.63:6812/3643207211 conn(0x7f73240369d0 legacy=0x7f7324038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:18.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.116+0000 7f733e51c640 10 monclient: shutdown 2024-09-16T05:36:18.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.116+0000 7f733e51c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:18.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.116+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 >> v1:172.21.15.6:6789/0 conn(0x7f7338136d20 legacy=0x7f73381d3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:18.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.116+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 shutdown_connections 2024-09-16T05:36:18.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.116+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 >> 172.21.15.63:0/2038636651 conn(0x7f73380a6310 msgr2=0x7f73380a7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:18.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.116+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 shutdown_connections 2024-09-16T05:36:18.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.116+0000 7f733e51c640 1 -- 172.21.15.63:0/2038636651 wait complete. 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_config_value foo min_size 2 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:+ local pool_name config_opt expected_val val 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:+ pool_name=foo 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:+ config_opt=min_size 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_val=2 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:++ get_config_value_or_die foo min_size 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:++ local pool_name config_opt raw val 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:++ pool_name=foo 2024-09-16T05:36:18.147 INFO:tasks.workunit.client.0.smithi063.stderr:++ config_opt=min_size 2024-09-16T05:36:18.148 INFO:tasks.workunit.client.0.smithi063.stderr:+++ ceph osd pool get foo min_size 2024-09-16T05:36:18.590 INFO:tasks.workunit.client.0.smithi063.stderr:++ raw='min_size: 2' 2024-09-16T05:36:18.590 INFO:tasks.workunit.client.0.smithi063.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T05:36:18.591 INFO:tasks.workunit.client.0.smithi063.stderr:+++ echo min_size: 2 2024-09-16T05:36:18.591 INFO:tasks.workunit.client.0.smithi063.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T05:36:18.592 INFO:tasks.workunit.client.0.smithi063.stderr:++ raw=min_size:2 2024-09-16T05:36:18.593 INFO:tasks.workunit.client.0.smithi063.stderr:+++ echo min_size:2 2024-09-16T05:36:18.593 INFO:tasks.workunit.client.0.smithi063.stderr:+++ cut -f2 -d: 2024-09-16T05:36:18.594 INFO:tasks.workunit.client.0.smithi063.stderr:++ val=2 2024-09-16T05:36:18.594 INFO:tasks.workunit.client.0.smithi063.stderr:++ echo 2 2024-09-16T05:36:18.594 INFO:tasks.workunit.client.0.smithi063.stderr:++ return 0 2024-09-16T05:36:18.594 INFO:tasks.workunit.client.0.smithi063.stderr:+ val=2 2024-09-16T05:36:18.594 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 2 != \2 ]] 2024-09-16T05:36:18.594 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set foo size 10 2024-09-16T05:36:18.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.712+0000 7fc1ee862640 10 monclient: get_monmap_and_config 2024-09-16T05:36:18.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.712+0000 7fc1ee862640 10 monclient: build_initial_monmap 2024-09-16T05:36:18.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.712+0000 7fc1ee862640 10 monclient: monmap: 2024-09-16T05:36:18.718 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:18.718 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:18.718 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:18.718517+0000 2024-09-16T05:36:18.719 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:18.718517+0000 2024-09-16T05:36:18.719 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:18.719 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:18.719 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:18.719 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:18.719 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:18.719 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:18.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.712+0000 7fc1ee862640 1 Processor -- start 2024-09-16T05:36:18.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.712+0000 7fc1ee862640 1 -- start start 2024-09-16T05:36:18.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.712+0000 7fc1ee862640 10 monclient: init 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient(hunting): picked mon.noname-a con 0x7fc1e8134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient(hunting): picked mon.noname-b con 0x7fc1e80a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient(hunting): picked mon.noname-c con 0x7fc1e806dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e8135420 con 0x7fc1e8134930 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e809e830 con 0x7fc1e806dfa0 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e80a0dc0 con 0x7fc1e80a0810 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient(hunting): authenticate will time out at 1035.496241s 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ecdff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc1e806dfa0 legacy=0x7fc1e813cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e77fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc1e80a0810 0x7fc1e8139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43290/0 (socket says 172.21.15.63:43290) 2024-09-16T05:36:18.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e77fe640 1 -- 172.21.15.63:0/2071286771 learned_addr learned my addr 172.21.15.63:0/2071286771 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243014146 0 0) 0x7fc1e8135420 con 0x7fc1e8134930 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient(hunting): my global_id is 4605 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d0003530 con 0x7fc1e8134930 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1751217494 0 0) 0x7fc1e80a0dc0 con 0x7fc1e80a0810 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:18.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient(hunting): my global_id is 4604 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1e8135420 con 0x7fc1e80a0810 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 637856884 0 0) 0x7fc1d0003530 con 0x7fc1e8134930 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 >> v1:172.21.15.6:6790/0 conn(0x7fc1e80a0810 legacy=0x7fc1e8139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 >> v1:172.21.15.63:6789/0 conn(0x7fc1e806dfa0 legacy=0x7fc1e813cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 monclient: found mon.noname-a 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e80a2500 con 0x7fc1e8134930 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:18.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1d8001550 con 0x7fc1e8134930 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: dump: 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:18.726 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 5 monclient: authenticate success, global_id 4605 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1d80017b0 con 0x7fc1e8134930 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 1 -- 172.21.15.63:0/2071286771 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1d8001b80 con 0x7fc1e8134930 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1e6ffd640 10 monclient: dump: 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.727 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 4 set_mon_vals no callback set 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient: shutdown 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- 172.21.15.63:0/2071286771 >> v1:172.21.15.6:6789/0 conn(0x7fc1e8134930 legacy=0x7fc1e8134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:18.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- 172.21.15.63:0/2071286771 shutdown_connections 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- 172.21.15.63:0/2071286771 >> 172.21.15.63:0/2071286771 conn(0x7fc1e80a1e90 msgr2=0x7fc1e8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- 172.21.15.63:0/2071286771 shutdown_connections 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- 172.21.15.63:0/2071286771 wait complete. 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient: build_initial_monmap 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient: monmap: 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:18.729 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:18.730 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:18.730 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:18.730 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:18.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 Processor -- start 2024-09-16T05:36:18.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 1 -- start start 2024-09-16T05:36:18.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.716+0000 7fc1ee862640 10 monclient: init 2024-09-16T05:36:18.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:18.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:18.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient(hunting): picked mon.noname-c con 0x7fc1e806dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:18.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient(hunting): picked mon.noname-a con 0x7fc1e80a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:18.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient(hunting): picked mon.noname-b con 0x7fc1e8134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e811eec0 con 0x7fc1e80a0810 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e81208d0 con 0x7fc1e8134930 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e80a3fb0 con 0x7fc1e806dfa0 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ecdff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc1e8134930 legacy=0x7fc1e81cf420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient(hunting): authenticate will time out at 1035.500715s 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc1e80a0810 0x7fc1e811d2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58784/0 (socket says 172.21.15.63:58784) 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e77fe640 1 -- 172.21.15.63:0/2458163801 learned_addr learned my addr 172.21.15.63:0/2458163801 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692502455 0 0) 0x7fc1e811eec0 con 0x7fc1e80a0810 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:18.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): my global_id is 4608 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d4003560 con 0x7fc1e80a0810 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279256558 0 0) 0x7fc1e80a3fb0 con 0x7fc1e806dfa0 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): my global_id is 4607 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1e811eec0 con 0x7fc1e806dfa0 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2781121030 0 0) 0x7fc1d4003560 con 0x7fc1e80a0810 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1e80a3fb0 con 0x7fc1e80a0810 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1dc001840 con 0x7fc1e80a0810 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:18.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:18.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1502249758 0 0) 0x7fc1e811eec0 con 0x7fc1e806dfa0 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1d4003560 con 0x7fc1e806dfa0 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1d8001450 con 0x7fc1e806dfa0 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:18.738 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 476832267 0 0) 0x7fc1e80a3fb0 con 0x7fc1e80a0810 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 >> v1:172.21.15.6:6790/0 conn(0x7fc1e806dfa0 legacy=0x7fc1e811cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 >> v1:172.21.15.63:6789/0 conn(0x7fc1e8134930 legacy=0x7fc1e81cf420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 monclient: found mon.a 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e809dc30 con 0x7fc1e80a0810 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:18.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 5 monclient: authenticate success, global_id 4608 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient: _renew_subs 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e809c0f0 con 0x7fc1e80a0810 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient: _renew_subs 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1e81205f0 con 0x7fc1e80a0810 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1dc001c70 con 0x7fc1e80a0810 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1dc004c40 con 0x7fc1e80a0810 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:18.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e67fc640 4 set_mon_vals no callback set 2024-09-16T05:36:18.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 10 monclient: dump: 2024-09-16T05:36:18.741 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:18.741 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:18.741 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.741 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:18.741 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:18.741 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:18.741 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c8005090 con 0x7fc1e80a0810 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.720+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc1dc004e60 con 0x7fc1e80a0810 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.724+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(38..38 src has 1..38) ==== 4634+0+0 (unknown 2187279167 0 0) 0x7fc1dc044620 con 0x7fc1e80a0810 2024-09-16T05:36:18.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.724+0000 7fc1e7fff640 10 monclient: get_auth_request con 0x7fc1d4036a00 auth_method 0 2024-09-16T05:36:18.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.732+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc1dc015480 con 0x7fc1e80a0810 2024-09-16T05:36:18.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.732+0000 7fc1e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:18.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.732+0000 7fc1e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:18.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.896+0000 7fc1ee862640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T05:36:18.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.896+0000 7fc1ee862640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T05:36:18.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.896+0000 7fc1ee862640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:18.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:18.896+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) -- 0x7fc1c8005b90 con 0x7fc1e80a0810 2024-09-16T05:36:19.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:19.120+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v39) ==== 124+0+0 (unknown 3564249015 0 0) 0x7fc1dc014d80 con 0x7fc1e80a0810 2024-09-16T05:36:19.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:19.120+0000 7fc1e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T05:36:19.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:19.120+0000 7fc1e4ff9640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 10 2024-09-16T05:36:19.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:19.260+0000 7fc1ee862640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T05:36:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:19.260+0000 7fc1ee862640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T05:36:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:19.260+0000 7fc1ee862640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:19.260+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) -- 0x7fc1c8005610 con 0x7fc1e80a0810 2024-09-16T05:36:20.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.148+0000 7fc1e4ff9640 1 -- 172.21.15.63:0/2458163801 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v40) ==== 124+0+0 (unknown 863449104 0 0) 0x7fc1dc0190d0 con 0x7fc1e80a0810 2024-09-16T05:36:20.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.148+0000 7fc1e4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T05:36:20.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.148+0000 7fc1e4ff9640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 10 2024-09-16T05:36:20.153 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 4 size to 10 2024-09-16T05:36:20.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.152+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc1d4036a00 legacy=0x7fc1d4038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:20.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.152+0000 7fc1ee862640 10 monclient: shutdown 2024-09-16T05:36:20.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.152+0000 7fc1ee862640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:20.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.152+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 >> v1:172.21.15.6:6789/0 conn(0x7fc1e80a0810 legacy=0x7fc1e811d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:20.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.152+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 shutdown_connections 2024-09-16T05:36:20.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.152+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 >> 172.21.15.63:0/2458163801 conn(0x7fc1e80a1e90 msgr2=0x7fc1e8133d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:20.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.152+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 shutdown_connections 2024-09-16T05:36:20.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.152+0000 7fc1ee862640 1 -- 172.21.15.63:0/2458163801 wait complete. 2024-09-16T05:36:20.199 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_config_value foo min_size 5 2024-09-16T05:36:20.199 INFO:tasks.workunit.client.0.smithi063.stderr:+ local pool_name config_opt expected_val val 2024-09-16T05:36:20.199 INFO:tasks.workunit.client.0.smithi063.stderr:+ pool_name=foo 2024-09-16T05:36:20.200 INFO:tasks.workunit.client.0.smithi063.stderr:+ config_opt=min_size 2024-09-16T05:36:20.200 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_val=5 2024-09-16T05:36:20.200 INFO:tasks.workunit.client.0.smithi063.stderr:++ get_config_value_or_die foo min_size 2024-09-16T05:36:20.200 INFO:tasks.workunit.client.0.smithi063.stderr:++ local pool_name config_opt raw val 2024-09-16T05:36:20.200 INFO:tasks.workunit.client.0.smithi063.stderr:++ pool_name=foo 2024-09-16T05:36:20.201 INFO:tasks.workunit.client.0.smithi063.stderr:++ config_opt=min_size 2024-09-16T05:36:20.201 INFO:tasks.workunit.client.0.smithi063.stderr:+++ ceph osd pool get foo min_size 2024-09-16T05:36:20.659 INFO:tasks.workunit.client.0.smithi063.stderr:++ raw='min_size: 5' 2024-09-16T05:36:20.659 INFO:tasks.workunit.client.0.smithi063.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T05:36:20.659 INFO:tasks.workunit.client.0.smithi063.stderr:+++ echo min_size: 5 2024-09-16T05:36:20.659 INFO:tasks.workunit.client.0.smithi063.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T05:36:20.661 INFO:tasks.workunit.client.0.smithi063.stderr:++ raw=min_size:5 2024-09-16T05:36:20.661 INFO:tasks.workunit.client.0.smithi063.stderr:+++ echo min_size:5 2024-09-16T05:36:20.661 INFO:tasks.workunit.client.0.smithi063.stderr:+++ cut -f2 -d: 2024-09-16T05:36:20.663 INFO:tasks.workunit.client.0.smithi063.stderr:++ val=5 2024-09-16T05:36:20.663 INFO:tasks.workunit.client.0.smithi063.stderr:++ echo 5 2024-09-16T05:36:20.663 INFO:tasks.workunit.client.0.smithi063.stderr:++ return 0 2024-09-16T05:36:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:+ val=5 2024-09-16T05:36:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 5 != \5 ]] 2024-09-16T05:36:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool set foo size 0 2024-09-16T05:36:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set foo size 0 2024-09-16T05:36:20.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient: get_monmap_and_config 2024-09-16T05:36:20.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient: build_initial_monmap 2024-09-16T05:36:20.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient: monmap: 2024-09-16T05:36:20.789 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:20.789 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:20.789 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:20.788650+0000 2024-09-16T05:36:20.789 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:20.788650+0000 2024-09-16T05:36:20.790 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:20.790 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:20.790 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:20.790 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:20.790 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:20.790 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:20.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 Processor -- start 2024-09-16T05:36:20.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- start start 2024-09-16T05:36:20.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient: init 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f19740a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f19740a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f19740a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19740a51e0 con 0x7f19740a3ca0 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19740a0d60 con 0x7f19740a4790 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19740a1040 con 0x7f19740a4d90 2024-09-16T05:36:20.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient(hunting): authenticate will time out at 1037.565379s 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1972d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f19740a4790 legacy=0x7f197413f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f19740a3ca0 0x7f19740a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58808/0 (socket says 172.21.15.63:58808) 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197259c640 1 -- 172.21.15.63:0/3538424757 learned_addr learned my addr 172.21.15.63:0/3538424757 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1971d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f19740a4d90 0x7f197413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43310/0 (socket says 172.21.15.63:43310) 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2398329189 0 0) 0x7f19740a51e0 con 0x7f19740a3ca0 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient(hunting): my global_id is 4617 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f195c003530 con 0x7f19740a3ca0 2024-09-16T05:36:20.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 154124427 0 0) 0x7f19740a1040 con 0x7f19740a4d90 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient(hunting): my global_id is 4616 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19740a51e0 con 0x7f19740a4d90 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1956012473 0 0) 0x7f195c003530 con 0x7f19740a3ca0 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 >> v1:172.21.15.6:6790/0 conn(0x7f19740a4d90 legacy=0x7f197413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 >> v1:172.21.15.63:6789/0 conn(0x7f19740a4790 legacy=0x7f197413f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 monclient: found mon.noname-a 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:20.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19740a0810 con 0x7f19740a3ca0 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: _finish_auth 0 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f196c0019a0 con 0x7f19740a3ca0 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: dump: 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:20.802 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 5 monclient: authenticate success, global_id 4617 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f196c001c00 con 0x7f19740a3ca0 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 4 set_mon_vals no callback set 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 1 -- 172.21.15.63:0/3538424757 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f196c001f50 con 0x7f19740a3ca0 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:20.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f197159a640 10 monclient: dump: 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 10 monclient: shutdown 2024-09-16T05:36:20.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- 172.21.15.63:0/3538424757 >> v1:172.21.15.6:6789/0 conn(0x7f19740a3ca0 legacy=0x7f19740a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- 172.21.15.63:0/3538424757 shutdown_connections 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- 172.21.15.63:0/3538424757 >> 172.21.15.63:0/3538424757 conn(0x7f1974075470 msgr2=0x7f1974075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- 172.21.15.63:0/3538424757 shutdown_connections 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.784+0000 7f1978ecf640 1 -- 172.21.15.63:0/3538424757 wait complete. 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: build_initial_monmap 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: monmap: 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 1 Processor -- start 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 1 -- start start 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: init 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f19740a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f19740a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f19740a4d90 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1974138cc0 con 0x7f19740a4790 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1974135f60 con 0x7f19740a4d90 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1974139280 con 0x7f19740a3ca0 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient(hunting): authenticate will time out at 1037.569232s 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1972d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f19740a4d90 legacy=0x7f19741d3170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f197259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f19740a3ca0 0x7f1974135140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43320/0 (socket says 172.21.15.63:43320) 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f197259c640 1 -- 172.21.15.63:0/2811166023 learned_addr learned my addr 172.21.15.63:0/2811166023 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2863035395 0 0) 0x7f1974138cc0 con 0x7f19740a4790 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): my global_id is 4620 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1954003530 con 0x7f19740a4790 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3764840336 0 0) 0x7f1974139280 con 0x7f19740a3ca0 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): my global_id is 4619 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1974138cc0 con 0x7f19740a3ca0 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3438734126 0 0) 0x7f1954003530 con 0x7f19740a4790 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1974139280 con 0x7f19740a4790 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1968001a80 con 0x7f19740a4790 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:20.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3799366460 0 0) 0x7f1974138cc0 con 0x7f19740a3ca0 2024-09-16T05:36:20.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1954003530 con 0x7f19740a3ca0 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f196c0018e0 con 0x7f19740a3ca0 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4017036460 0 0) 0x7f1974139280 con 0x7f19740a4790 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 >> v1:172.21.15.6:6790/0 conn(0x7f19740a3ca0 legacy=0x7f1974135140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 >> v1:172.21.15.63:6789/0 conn(0x7f19740a4d90 legacy=0x7f19741d3170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 monclient: found mon.a 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19741389e0 con 0x7f19740a4790 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 5 monclient: authenticate success, global_id 4620 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: _renew_subs 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1974138fa0 con 0x7f19740a4790 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: _renew_subs 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f19741d4a60 con 0x7f19740a4790 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1968001ce0 con 0x7f19740a4790 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1968004c40 con 0x7f19740a4790 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1970d99640 4 set_mon_vals no callback set 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1962ffd640 10 monclient: dump: 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.788+0000 7f1978ecf640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1958005090 con 0x7f19740a4790 2024-09-16T05:36:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.792+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f196800f9e0 con 0x7f19740a4790 2024-09-16T05:36:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.792+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(40..40 src has 1..40) ==== 4634+0+0 (unknown 3230365772 0 0) 0x7f1968044630 con 0x7f19740a4790 2024-09-16T05:36:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.792+0000 7f197259c640 10 monclient: get_auth_request con 0x7f19540369d0 auth_method 0 2024-09-16T05:36:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.804+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1968015430 con 0x7f19740a4790 2024-09-16T05:36:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.804+0000 7f1962ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.804+0000 7f1962ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:20.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.968+0000 7f1978ecf640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T05:36:20.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.968+0000 7f1978ecf640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T05:36:20.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.968+0000 7f1978ecf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:20.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.968+0000 7f1978ecf640 1 -- 172.21.15.63:0/2811166023 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"} v 0) -- 0x7f1958005b90 con 0x7f19740a4790 2024-09-16T05:36:20.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.976+0000 7f1962ffd640 1 -- 172.21.15.63:0/2811166023 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}]=-22 pool size must be between 1 and 10 v40) ==== 136+0+0 (unknown 1338229041 0 0) 0x7f19680190e0 con 0x7f19740a4790 2024-09-16T05:36:20.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.976+0000 7f1962ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T05:36:20.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:20.976+0000 7f1962ffd640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-16T05:36:20.977 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-16T05:36:20.981 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:20.981 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool set foo size 20 2024-09-16T05:36:20.981 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:20.981 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set foo size 20 2024-09-16T05:36:21.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient: get_monmap_and_config 2024-09-16T05:36:21.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient: build_initial_monmap 2024-09-16T05:36:21.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient: monmap: 2024-09-16T05:36:21.106 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:21.106680+0000 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:21.106680+0000 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:21.107 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 1 Processor -- start 2024-09-16T05:36:21.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 1 -- start start 2024-09-16T05:36:21.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient: init 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient(hunting): picked mon.noname-b con 0x7f6a68134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient(hunting): picked mon.noname-c con 0x7f6a680a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient(hunting): picked mon.noname-a con 0x7f6a6806dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a68135420 con 0x7f6a6806dfa0 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a6809e830 con 0x7f6a680a0810 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a680a0dc0 con 0x7f6a68134930 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a6df72640 10 monclient(hunting): authenticate will time out at 1037.883662s 2024-09-16T05:36:21.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.100+0000 7f6a5ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6a680a0810 legacy=0x7f6a68139820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a67fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a6806dfa0 0x7f6a6813cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58822/0 (socket says 172.21.15.63:58822) 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a67fff640 1 -- 172.21.15.63:0/3715057815 learned_addr learned my addr 172.21.15.63:0/3715057815 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 276721386 0 0) 0x7f6a68135420 con 0x7f6a6806dfa0 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient(hunting): my global_id is 4623 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a54003530 con 0x7f6a6806dfa0 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2678929611 0 0) 0x7f6a680a0dc0 con 0x7f6a68134930 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient(hunting): my global_id is 4622 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a68135420 con 0x7f6a68134930 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2796621075 0 0) 0x7f6a54003530 con 0x7f6a6806dfa0 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 >> v1:172.21.15.6:6790/0 conn(0x7f6a68134930 legacy=0x7f6a68134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 >> v1:172.21.15.63:6789/0 conn(0x7f6a680a0810 legacy=0x7f6a68139820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 monclient: found mon.noname-a 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a680a2500 con 0x7f6a6806dfa0 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:21.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a600019a0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: dump: 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.113 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 5 monclient: authenticate success, global_id 4623 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a60001c00 con 0x7f6a6806dfa0 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 1 -- 172.21.15.63:0/3715057815 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a60001fd0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a66ffd640 10 monclient: dump: 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.114 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.115 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 4 set_mon_vals no callback set 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 10 monclient: shutdown 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 1 -- 172.21.15.63:0/3715057815 >> v1:172.21.15.6:6789/0 conn(0x7f6a6806dfa0 legacy=0x7f6a6813cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 1 -- 172.21.15.63:0/3715057815 shutdown_connections 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 1 -- 172.21.15.63:0/3715057815 >> 172.21.15.63:0/3715057815 conn(0x7f6a680a1e90 msgr2=0x7f6a68075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 1 -- 172.21.15.63:0/3715057815 shutdown_connections 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 1 -- 172.21.15.63:0/3715057815 wait complete. 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 10 monclient: build_initial_monmap 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 10 monclient: monmap: 2024-09-16T05:36:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 1 Processor -- start 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 1 -- start start 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.104+0000 7f6a6df72640 10 monclient: init 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:21.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient(hunting): picked mon.noname-a con 0x7f6a6806dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient(hunting): picked mon.noname-b con 0x7f6a680a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient(hunting): picked mon.noname-c con 0x7f6a68134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a68123270 con 0x7f6a6806dfa0 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a680a4810 con 0x7f6a680a0810 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a5ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6a680a0810 legacy=0x7f6a681cbfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a6809c0f0 con 0x7f6a68134930 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient(hunting): authenticate will time out at 1037.888538s 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a677fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a6806dfa0 0x7f6a68133890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58836/0 (socket says 172.21.15.63:58836) 2024-09-16T05:36:21.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a677fe640 1 -- 172.21.15.63:0/3694096915 learned_addr learned my addr 172.21.15.63:0/3694096915 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:21.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2778567712 0 0) 0x7f6a68123270 con 0x7f6a6806dfa0 2024-09-16T05:36:21.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): my global_id is 4626 2024-09-16T05:36:21.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a48003530 con 0x7f6a6806dfa0 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036619236 0 0) 0x7f6a6809c0f0 con 0x7f6a68134930 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): my global_id is 4625 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a68123270 con 0x7f6a68134930 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3043089052 0 0) 0x7f6a48003530 con 0x7f6a6806dfa0 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a6809c0f0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1202041873 0 0) 0x7f6a68123270 con 0x7f6a68134930 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a48003530 con 0x7f6a68134930 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a580019a0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:21.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a60001db0 con 0x7f6a68134930 2024-09-16T05:36:21.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2943515328 0 0) 0x7f6a6809c0f0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 >> v1:172.21.15.6:6790/0 conn(0x7f6a68134930 legacy=0x7f6a681cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 >> v1:172.21.15.63:6789/0 conn(0x7f6a680a0810 legacy=0x7f6a681cbfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 monclient: found mon.a 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a680732d0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 5 monclient: authenticate success, global_id 4626 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient: _renew_subs 2024-09-16T05:36:21.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a68122cb0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient: _renew_subs 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a681217f0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a580038b0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a58003c00 con 0x7f6a6806dfa0 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a667fc640 4 set_mon_vals no callback set 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 10 monclient: dump: 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.129 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a6df72640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a4c005090 con 0x7f6a6806dfa0 2024-09-16T05:36:21.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.108+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6a58003e20 con 0x7f6a6806dfa0 2024-09-16T05:36:21.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.112+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(40..40 src has 1..40) ==== 4634+0+0 (unknown 3230365772 0 0) 0x7f6a580445f0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.112+0000 7f6a5ffff640 10 monclient: get_auth_request con 0x7f6a480369c0 auth_method 0 2024-09-16T05:36:21.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.128+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6a580153f0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.128+0000 7f6a64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.128+0000 7f6a64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.284+0000 7f6a6df72640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T05:36:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.288+0000 7f6a6df72640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T05:36:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.288+0000 7f6a6df72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.288+0000 7f6a6df72640 1 -- 172.21.15.63:0/3694096915 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"} v 0) -- 0x7f6a4c005b90 con 0x7f6a6806dfa0 2024-09-16T05:36:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.288+0000 7f6a64ff9640 1 -- 172.21.15.63:0/3694096915 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}]=-22 pool size must be between 1 and 10 v40) ==== 137+0+0 (unknown 524740766 0 0) 0x7f6a580190a0 con 0x7f6a6806dfa0 2024-09-16T05:36:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.288+0000 7f6a64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T05:36:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.288+0000 7f6a64ff9640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-16T05:36:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-16T05:36:21.300 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:21.300 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool delete foo 2024-09-16T05:36:21.300 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:21.300 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete foo 2024-09-16T05:36:21.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: get_monmap_and_config 2024-09-16T05:36:21.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: build_initial_monmap 2024-09-16T05:36:21.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: monmap: 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:21.422264+0000 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:21.422264+0000 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:21.423 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 Processor -- start 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- start start 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: init 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): picked mon.noname-c con 0x7f580c0a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): picked mon.noname-a con 0x7f580c0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): picked mon.noname-b con 0x7f580c0a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c0a4fd0 con 0x7f580c0a08a0 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c0a5b40 con 0x7f580c0a3ca0 2024-09-16T05:36:21.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c0a1a70 con 0x7f580c0a4790 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): authenticate will time out at 1038.199347s 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f580ad9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f580c0a3ca0 legacy=0x7f580c0a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f580a59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f580c0a08a0 0x7f580c13beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58840/0 (socket says 172.21.15.63:58840) 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f580a59c640 1 -- 172.21.15.63:0/116575506 learned_addr learned my addr 172.21.15.63:0/116575506 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310386778 0 0) 0x7f580c0a4fd0 con 0x7f580c0a08a0 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient(hunting): my global_id is 4629 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57f4003530 con 0x7f580c0a08a0 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860545212 0 0) 0x7f580c0a1a70 con 0x7f580c0a4790 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient(hunting): my global_id is 4628 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f580c0a4fd0 con 0x7f580c0a4790 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1435059411 0 0) 0x7f580c0a4fd0 con 0x7f580c0a4790 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 >> v1:172.21.15.63:6789/0 conn(0x7f580c0a3ca0 legacy=0x7f580c0a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 >> v1:172.21.15.6:6789/0 conn(0x7f580c0a08a0 legacy=0x7f580c13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 monclient: found mon.noname-b 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580c0a53e0 con 0x7f580c0a4790 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: _finish_auth 0 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5804001900 con 0x7f580c0a4790 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: dump: 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 5 monclient: authenticate success, global_id 4628 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5804001bf0 con 0x7f580c0a4790 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 1 -- 172.21.15.63:0/116575506 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5804002000 con 0x7f580c0a4790 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f5809d9b640 10 monclient: dump: 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.434 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 4 set_mon_vals no callback set 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: shutdown 2024-09-16T05:36:21.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- 172.21.15.63:0/116575506 >> v1:172.21.15.6:6790/0 conn(0x7f580c0a4790 legacy=0x7f580c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- 172.21.15.63:0/116575506 shutdown_connections 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- 172.21.15.63:0/116575506 >> 172.21.15.63:0/116575506 conn(0x7f580c0a67c0 msgr2=0x7f580c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- 172.21.15.63:0/116575506 shutdown_connections 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- 172.21.15.63:0/116575506 wait complete. 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: build_initial_monmap 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: monmap: 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:21.436 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:21.437 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:21.437 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:21.437 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:21.437 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.437 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:21.437 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:21.438 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:21.438 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 Processor -- start 2024-09-16T05:36:21.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- start start 2024-09-16T05:36:21.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: init 2024-09-16T05:36:21.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:21.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): picked mon.noname-c con 0x7f580c0a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): picked mon.noname-b con 0x7f580c0a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): picked mon.noname-a con 0x7f580c0a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c1248e0 con 0x7f580c0a4790 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c1220d0 con 0x7f580c0a3ca0 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f580a59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f580c0a3ca0 legacy=0x7f580c1d00e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580c0a80b0 con 0x7f580c0a08a0 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:21.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.420+0000 7f58111b6640 10 monclient(hunting): authenticate will time out at 1038.203823s 2024-09-16T05:36:21.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f580ad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f580c0a08a0 0x7f580c1361f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43358/0 (socket says 172.21.15.63:43358) 2024-09-16T05:36:21.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f580ad9d640 1 -- 172.21.15.63:0/558277858 learned_addr learned my addr 172.21.15.63:0/558277858 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:21.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f580b59e640 1 --1- 172.21.15.63:0/558277858 >> v1:172.21.15.6:6789/0 conn(0x7f580c0a4790 0x7f580c1d3810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58854/0 (socket says 172.21.15.63:58854) 2024-09-16T05:36:21.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901462285 0 0) 0x7f580c0a80b0 con 0x7f580c0a08a0 2024-09-16T05:36:21.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): my global_id is 4631 2024-09-16T05:36:21.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57f8003530 con 0x7f580c0a08a0 2024-09-16T05:36:21.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700782259 0 0) 0x7f580c1248e0 con 0x7f580c0a4790 2024-09-16T05:36:21.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): my global_id is 4632 2024-09-16T05:36:21.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f580c0a80b0 con 0x7f580c0a4790 2024-09-16T05:36:21.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1957753609 0 0) 0x7f580c0a80b0 con 0x7f580c0a4790 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f580c1248e0 con 0x7f580c0a4790 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1305989528 0 0) 0x7f57f8003530 con 0x7f580c0a08a0 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f580c0a80b0 con 0x7f580c0a08a0 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5804001f40 con 0x7f580c0a4790 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): dump: 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.445 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f57fc0040e0 con 0x7f580c0a08a0 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient(hunting): dump: 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4014609465 0 0) 0x7f580c1248e0 con 0x7f580c0a4790 2024-09-16T05:36:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 >> v1:172.21.15.6:6790/0 conn(0x7f580c0a08a0 legacy=0x7f580c1361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 >> v1:172.21.15.63:6789/0 conn(0x7f580c0a3ca0 legacy=0x7f580c1d00e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:21.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:21.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 monclient: found mon.a 2024-09-16T05:36:21.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:21.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580c124600 con 0x7f580c0a4790 2024-09-16T05:36:21.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:21.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:21.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:21.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 5 monclient: authenticate success, global_id 4632 2024-09-16T05:36:21.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 10 monclient: _renew_subs 2024-09-16T05:36:21.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f580c1231b0 con 0x7f580c0a4790 2024-09-16T05:36:21.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 10 monclient: _renew_subs 2024-09-16T05:36:21.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f580c136900 con 0x7f580c0a4790 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5804001bf0 con 0x7f580c0a4790 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5804003d80 con 0x7f580c0a4790 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 10 monclient: dump: 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.450 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f580959a640 4 set_mon_vals no callback set 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f58111b6640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57ec005090 con 0x7f580c0a4790 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5804003fe0 con 0x7f580c0a4790 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(40..40 src has 1..40) ==== 4634+0+0 (unknown 3230365772 0 0) 0x7f5804044770 con 0x7f580c0a4790 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.424+0000 7f580ad9d640 10 monclient: get_auth_request con 0x7f57f80369c0 auth_method 0 2024-09-16T05:36:21.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.440+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f580401dae0 con 0x7f580c0a4790 2024-09-16T05:36:21.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.440+0000 7f57f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.440+0000 7f57f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:21.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.608+0000 7f58111b6640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T05:36:21.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.608+0000 7f58111b6640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T05:36:21.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.608+0000 7f58111b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:21.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.608+0000 7f58111b6640 1 -- 172.21.15.63:0/558277858 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo"} v 0) -- 0x7f57ec005380 con 0x7f580c0a4790 2024-09-16T05:36:21.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.608+0000 7f57f37fe640 1 -- 172.21.15.63:0/558277858 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v40) ==== 274+0+0 (unknown 1108762609 0 0) 0x7f5804040220 con 0x7f580c0a4790 2024-09-16T05:36:21.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.608+0000 7f57f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T05:36:21.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.608+0000 7f57f37fe640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T05:36:21.610 INFO:tasks.workunit.client.0.smithi063.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T05:36:21.616 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:21.617 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool delete foo foo 2024-09-16T05:36:21.617 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:21.617 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete foo foo 2024-09-16T05:36:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient: get_monmap_and_config 2024-09-16T05:36:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient: build_initial_monmap 2024-09-16T05:36:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient: monmap: 2024-09-16T05:36:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:21.733382+0000 2024-09-16T05:36:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:21.733382+0000 2024-09-16T05:36:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:21.734 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 1 Processor -- start 2024-09-16T05:36:21.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 1 -- start start 2024-09-16T05:36:21.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient: init 2024-09-16T05:36:21.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc00009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc0000a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc00009b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc000074200 con 0x7fc00009b780 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00009f900 con 0x7fc0000a0280 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00009e870 con 0x7fc00009ac90 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 10 monclient(hunting): authenticate will time out at 1038.510340s 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbfff7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc0000a0280 legacy=0x7fc0001398b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc004e4b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc00009b780 0x7fc00013cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58858/0 (socket says 172.21.15.63:58858) 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc00009ac90 0x7fc00009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43368/0 (socket says 172.21.15.63:43368) 2024-09-16T05:36:21.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc004e4b640 1 -- 172.21.15.63:0/1444683757 learned_addr learned my addr 172.21.15.63:0/1444683757 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:21.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1372478278 0 0) 0x7fc00009e870 con 0x7fc00009ac90 2024-09-16T05:36:21.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient(hunting): my global_id is 4634 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe8003530 con 0x7fc00009ac90 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2342013551 0 0) 0x7fc000074200 con 0x7fc00009b780 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient(hunting): my global_id is 4635 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc00009e870 con 0x7fc00009b780 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2627163557 0 0) 0x7fbfe8003530 con 0x7fc00009ac90 2024-09-16T05:36:21.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 >> v1:172.21.15.63:6789/0 conn(0x7fc0000a0280 legacy=0x7fc0001398b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 >> v1:172.21.15.6:6789/0 conn(0x7fc00009b780 legacy=0x7fc00013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 monclient: found mon.noname-b 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc000141730 con 0x7fc00009ac90 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbff4001620 con 0x7fc00009ac90 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:21.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fbffeffd640 10 monclient: dump: 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.743 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 5 monclient: authenticate success, global_id 4634 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.732+0000 7fc0068ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbff4001880 con 0x7fc00009ac90 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffeffd640 1 -- 172.21.15.63:0/1444683757 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbff4001bd0 con 0x7fc00009ac90 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffeffd640 10 monclient: dump: 2024-09-16T05:36:21.744 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 4 set_mon_vals no callback set 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient: shutdown 2024-09-16T05:36:21.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- 172.21.15.63:0/1444683757 >> v1:172.21.15.6:6790/0 conn(0x7fc00009ac90 legacy=0x7fc00009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- 172.21.15.63:0/1444683757 shutdown_connections 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- 172.21.15.63:0/1444683757 >> 172.21.15.63:0/1444683757 conn(0x7fc0000a5290 msgr2=0x7fc0000a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- 172.21.15.63:0/1444683757 shutdown_connections 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- 172.21.15.63:0/1444683757 wait complete. 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient: build_initial_monmap 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient: monmap: 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:21.746 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.747 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:21.747 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:21.747 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:21.747 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 Processor -- start 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- start start 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient: init 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc00009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc00009b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc0000a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00013ee30 con 0x7fc00009ac90 2024-09-16T05:36:21.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00013d000 con 0x7fc00009b780 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbfff7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc00009b780 legacy=0x7fc0001ce880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00006e0f0 con 0x7fc0000a0280 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc0068ae640 10 monclient(hunting): authenticate will time out at 1038.514629s 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc004e4b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc0000a0280 0x7fc0001d1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43384/0 (socket says 172.21.15.63:43384) 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fc004e4b640 1 -- 172.21.15.63:0/3876804359 learned_addr learned my addr 172.21.15.63:0/3876804359 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240001412 0 0) 0x7fc00006e0f0 con 0x7fc0000a0280 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): my global_id is 4637 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfec003530 con 0x7fc0000a0280 2024-09-16T05:36:21.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3721775090 0 0) 0x7fc000141730 con 0x7fc00009ac90 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): my global_id is 4638 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc00006e0f0 con 0x7fc00009ac90 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2208835699 0 0) 0x7fbfec003530 con 0x7fc0000a0280 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc000141730 con 0x7fc0000a0280 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbff8001bc0 con 0x7fc0000a0280 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): dump: 2024-09-16T05:36:21.754 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.736+0000 7fbffcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1930861810 0 0) 0x7fc00006e0f0 con 0x7fc00009ac90 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfec003530 con 0x7fc00009ac90 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbff4001560 con 0x7fc00009ac90 2024-09-16T05:36:21.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient(hunting): dump: 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4094154960 0 0) 0x7fc000141730 con 0x7fc0000a0280 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 >> v1:172.21.15.63:6789/0 conn(0x7fc00009b780 legacy=0x7fc0001ce880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 >> v1:172.21.15.6:6789/0 conn(0x7fc00009ac90 legacy=0x7fc0001c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 monclient: found mon.c 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00009c0f0 con 0x7fc0000a0280 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 5 monclient: authenticate success, global_id 4637 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 10 monclient: _renew_subs 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:21.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc00013c720 con 0x7fc0000a0280 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 10 monclient: _renew_subs 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc00013ea10 con 0x7fc0000a0280 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbff8004b70 con 0x7fc0000a0280 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbff8004da0 con 0x7fc0000a0280 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffe7fc640 4 set_mon_vals no callback set 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 10 monclient: dump: 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.758 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fc0068ae640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe0005090 con 0x7fc0000a0280 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbff8004f40 con 0x7fc0000a0280 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(40..40 src has 1..40) ==== 4634+0+0 (unknown 3230365772 0 0) 0x7fbff8044800 con 0x7fc0000a0280 2024-09-16T05:36:21.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.740+0000 7fbffffff640 10 monclient: get_auth_request con 0x7fbfec036b10 auth_method 0 2024-09-16T05:36:21.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.752+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbff80156c0 con 0x7fc0000a0280 2024-09-16T05:36:21.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.752+0000 7fbffcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:21.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.752+0000 7fbffcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:21.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.920+0000 7fc0068ae640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T05:36:21.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.920+0000 7fc0068ae640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T05:36:21.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.920+0000 7fc0068ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:21.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.920+0000 7fc0068ae640 1 -- 172.21.15.63:0/3876804359 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"} v 0) -- 0x7fbfe0005b90 con 0x7fc0000a0280 2024-09-16T05:36:21.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.924+0000 7fbffcff9640 1 -- 172.21.15.63:0/3876804359 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v40) ==== 290+0+0 (unknown 1561271976 0 0) 0x7fbff8014fc0 con 0x7fc0000a0280 2024-09-16T05:36:21.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.924+0000 7fbffcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T05:36:21.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:21.924+0000 7fbffcff9640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T05:36:21.927 INFO:tasks.workunit.client.0.smithi063.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T05:36:21.932 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:21.933 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool delete foo foo --force 2024-09-16T05:36:21.933 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:21.933 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete foo foo --force 2024-09-16T05:36:22.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient: get_monmap_and_config 2024-09-16T05:36:22.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient: build_initial_monmap 2024-09-16T05:36:22.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient: monmap: 2024-09-16T05:36:22.057 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:22.057 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.057 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:22.057450+0000 2024-09-16T05:36:22.058 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:22.057450+0000 2024-09-16T05:36:22.058 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:22.058 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.058 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:22.058 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:22.058 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:22.058 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 1 Processor -- start 2024-09-16T05:36:22.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 1 -- start start 2024-09-16T05:36:22.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient: init 2024-09-16T05:36:22.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:22.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:22.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient(hunting): picked mon.noname-a con 0x7f7c6c09c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient(hunting): picked mon.noname-c con 0x7f7c6c0a09c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient(hunting): picked mon.noname-b con 0x7f7c6c09d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6c0a0ed0 con 0x7f7c6c09c300 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6c09d950 con 0x7f7c6c0a09c0 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6c09ebd0 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient(hunting): authenticate will time out at 1038.834397s 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7c6c0a09c0 legacy=0x7f7c6c138a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c7092d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7c6c09c300 0x7f7c6c09c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58884/0 (socket says 172.21.15.63:58884) 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c7112e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7c6c09d3a0 0x7f7c6c13c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43398/0 (socket says 172.21.15.63:43398) 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c7092d640 1 -- 172.21.15.63:0/114040574 learned_addr learned my addr 172.21.15.63:0/114040574 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:22.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779763125 0 0) 0x7f7c6c0a0ed0 con 0x7f7c6c09c300 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient(hunting): my global_id is 4641 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c54003530 con 0x7f7c6c09c300 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 926118683 0 0) 0x7f7c6c09ebd0 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient(hunting): my global_id is 4640 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c6c0a0ed0 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2529165924 0 0) 0x7f7c54003530 con 0x7f7c6c09c300 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 >> v1:172.21.15.6:6790/0 conn(0x7f7c6c09d3a0 legacy=0x7f7c6c13c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 >> v1:172.21.15.63:6789/0 conn(0x7f7c6c0a09c0 legacy=0x7f7c6c138a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 monclient: found mon.noname-a 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6c0a2470 con 0x7f7c6c09c300 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c60001620 con 0x7f7c6c09c300 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: dump: 2024-09-16T05:36:22.063 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 5 monclient: authenticate success, global_id 4641 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c60001880 con 0x7f7c6c09c300 2024-09-16T05:36:22.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 1 -- 172.21.15.63:0/114040574 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c60001c50 con 0x7f7c6c09c300 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c6b7fe640 10 monclient: dump: 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.065 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.066 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.066 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.066 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.066 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.066 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:22.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.056+0000 7f7c72b91640 4 set_mon_vals no callback set 2024-09-16T05:36:22.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient: shutdown 2024-09-16T05:36:22.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:22.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- 172.21.15.63:0/114040574 >> v1:172.21.15.6:6789/0 conn(0x7f7c6c09c300 legacy=0x7f7c6c09c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- 172.21.15.63:0/114040574 shutdown_connections 2024-09-16T05:36:22.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- 172.21.15.63:0/114040574 >> 172.21.15.63:0/114040574 conn(0x7f7c6c0a1e90 msgr2=0x7f7c6c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:22.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- 172.21.15.63:0/114040574 shutdown_connections 2024-09-16T05:36:22.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- 172.21.15.63:0/114040574 wait complete. 2024-09-16T05:36:22.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient: build_initial_monmap 2024-09-16T05:36:22.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient: monmap: 2024-09-16T05:36:22.071 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:22.071 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:22.071 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 Processor -- start 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- start start 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient: init 2024-09-16T05:36:22.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient(hunting): picked mon.noname-a con 0x7f7c6c09c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient(hunting): picked mon.noname-c con 0x7f7c6c09d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient(hunting): picked mon.noname-b con 0x7f7c6c09f800 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6c09c0f0 con 0x7f7c6c09c300 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6c06df50 con 0x7f7c6c09f800 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c7112e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7c6c09f800 legacy=0x7f7c6c1d18e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6c121120 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c72b91640 10 monclient(hunting): authenticate will time out at 1038.838777s 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c7092d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7c6c09c300 0x7f7c6c132a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58896/0 (socket says 172.21.15.63:58896) 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c6bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7c6c09d3a0 0x7f7c6c1ce1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43408/0 (socket says 172.21.15.63:43408) 2024-09-16T05:36:22.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c7092d640 1 -- 172.21.15.63:0/1798962817 learned_addr learned my addr 172.21.15.63:0/1798962817 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2846872576 0 0) 0x7f7c6c09c0f0 con 0x7f7c6c09c300 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): my global_id is 4644 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c58003530 con 0x7f7c6c09c300 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404013748 0 0) 0x7f7c6c121120 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): my global_id is 4643 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c6c09c0f0 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2200240976 0 0) 0x7f7c58003530 con 0x7f7c6c09c300 2024-09-16T05:36:22.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c6c121120 con 0x7f7c6c09c300 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c60001520 con 0x7f7c6c09c300 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): dump: 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.075 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.060+0000 7f7c697fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2506615209 0 0) 0x7f7c6c09c0f0 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c58003530 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c5c004110 con 0x7f7c6c09d3a0 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient(hunting): dump: 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.076 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1917835451 0 0) 0x7f7c6c121120 con 0x7f7c6c09c300 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 >> v1:172.21.15.6:6790/0 conn(0x7f7c6c09d3a0 legacy=0x7f7c6c1ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 >> v1:172.21.15.63:6789/0 conn(0x7f7c6c09f800 legacy=0x7f7c6c1d18e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient: _finish_hunting 0 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 monclient: found mon.a 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:22.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6c121400 con 0x7f7c6c09c300 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient: _finish_auth 0 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient: _check_auth_tickets 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 5 monclient: authenticate success, global_id 4644 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 10 monclient: _renew_subs 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 1 -- 172.21.15.63:0/1798962817 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6c11f9f0 con 0x7f7c6c09c300 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 10 monclient: _renew_subs 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 1 -- 172.21.15.63:0/1798962817 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c6c133140 con 0x7f7c6c09c300 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c60001880 con 0x7f7c6c09c300 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:22.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c60004da0 con 0x7f7c6c09c300 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c6affd640 4 set_mon_vals no callback set 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 10 monclient: dump: 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.079 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c72b91640 1 -- 172.21.15.63:0/1798962817 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c005090 con 0x7f7c6c09c300 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7c60004fc0 con 0x7f7c6c09c300 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(40..40 src has 1..40) ==== 4634+0+0 (unknown 3230365772 0 0) 0x7f7c600445f0 con 0x7f7c6c09c300 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.064+0000 7f7c6bfff640 10 monclient: get_auth_request con 0x7f7c58036a00 auth_method 0 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.076+0000 7f7c697fa640 1 -- 172.21.15.63:0/1798962817 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7c600190a0 con 0x7f7c6c09c300 2024-09-16T05:36:22.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.076+0000 7f7c697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.076+0000 7f7c697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:22.257 INFO:tasks.workunit.client.0.smithi063.stderr:--force not valid: --force not one of 'true', 'false' 2024-09-16T05:36:22.257 INFO:tasks.workunit.client.0.smithi063.stderr:Invalid command: unused arguments: ['--force'] 2024-09-16T05:36:22.257 INFO:tasks.workunit.client.0.smithi063.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T05:36:22.257 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T05:36:22.260 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:22.260 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-16T05:36:22.260 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:22.260 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient: get_monmap_and_config 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient: build_initial_monmap 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient: monmap: 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:22.384915+0000 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:22.384915+0000 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:22.385 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:22.386 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:22.386 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 1 Processor -- start 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 1 -- start start 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient: init 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient(hunting): picked mon.noname-c con 0x7ff9700a22a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient(hunting): picked mon.noname-a con 0x7ff9700731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient(hunting): picked mon.noname-b con 0x7ff9701395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:22.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff97009c580 con 0x7ff9700731d0 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9700a1300 con 0x7ff9700a22a0 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff975e7c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff9700a22a0 legacy=0x7ff97009f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff97009d0f0 con 0x7ff9701395a0 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 10 monclient(hunting): authenticate will time out at 1039.162335s 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff97567b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff9700731d0 0x7ff97009fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58910/0 (socket says 172.21.15.63:58910) 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff97567b640 1 -- 172.21.15.63:0/894303323 learned_addr learned my addr 172.21.15.63:0/894303323 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526768900 0 0) 0x7ff97009c580 con 0x7ff9700731d0 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient(hunting): my global_id is 4647 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff95c003530 con 0x7ff9700731d0 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3897006404 0 0) 0x7ff97009d0f0 con 0x7ff9701395a0 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient(hunting): my global_id is 4646 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff97009c580 con 0x7ff9701395a0 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 986904306 0 0) 0x7ff97009c580 con 0x7ff9701395a0 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 >> v1:172.21.15.63:6789/0 conn(0x7ff9700a22a0 legacy=0x7ff97009f790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 >> v1:172.21.15.6:6789/0 conn(0x7ff9700731d0 legacy=0x7ff97009fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:22.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 monclient: found mon.noname-b 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97009d3d0 con 0x7ff9701395a0 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient: _finish_auth 0 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff968001600 con 0x7ff9701395a0 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 5 monclient: authenticate success, global_id 4646 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff974e7a640 10 monclient: dump: 2024-09-16T05:36:22.392 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.384+0000 7ff9780e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff968001860 con 0x7ff9701395a0 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff974e7a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff974e7a640 1 -- 172.21.15.63:0/894303323 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff968001cb0 con 0x7ff9701395a0 2024-09-16T05:36:22.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff974e7a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff974e7a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff974e7a640 10 monclient: dump: 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:22.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 4 set_mon_vals no callback set 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient: shutdown 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- 172.21.15.63:0/894303323 >> v1:172.21.15.6:6790/0 conn(0x7ff9701395a0 legacy=0x7ff97013b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- 172.21.15.63:0/894303323 shutdown_connections 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- 172.21.15.63:0/894303323 >> 172.21.15.63:0/894303323 conn(0x7ff9700a1e90 msgr2=0x7ff970075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- 172.21.15.63:0/894303323 shutdown_connections 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- 172.21.15.63:0/894303323 wait complete. 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient: build_initial_monmap 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient: monmap: 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:22.395 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:22.396 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:22.396 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.396 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:22.396 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:22.396 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:22.396 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 Processor -- start 2024-09-16T05:36:22.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- start start 2024-09-16T05:36:22.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient: init 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient(hunting): picked mon.noname-a con 0x7ff9700731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient(hunting): picked mon.noname-b con 0x7ff9700a22a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient(hunting): picked mon.noname-c con 0x7ff9701395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff970125720 con 0x7ff9700731d0 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff97009c0f0 con 0x7ff9700a22a0 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9700a43d0 con 0x7ff9701395a0 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff97567b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff9700a22a0 legacy=0x7ff97006df70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9780e0640 10 monclient(hunting): authenticate will time out at 1039.166190s 2024-09-16T05:36:22.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff975e7c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff9700731d0 0x7ff97006d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58914/0 (socket says 172.21.15.63:58914) 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff975e7c640 1 -- 172.21.15.63:0/2537088801 learned_addr learned my addr 172.21.15.63:0/2537088801 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561701505 0 0) 0x7ff970125720 con 0x7ff9700731d0 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): my global_id is 4650 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff954003530 con 0x7ff9700731d0 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679598541 0 0) 0x7ff9700a43d0 con 0x7ff9701395a0 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): my global_id is 4649 2024-09-16T05:36:22.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff970125720 con 0x7ff9701395a0 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 748201071 0 0) 0x7ff954003530 con 0x7ff9700731d0 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9700a43d0 con 0x7ff9700731d0 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff96c0044e0 con 0x7ff9700731d0 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): dump: 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.402 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff975e7c640 1 -- 172.21.15.63:0/2537088801 >> v1:172.21.15.6:6789/0 conn(0x7ff9700731d0 legacy=0x7ff97006d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff975e7c640 1 -- 172.21.15.63:0/2537088801 >> v1:172.21.15.6:6789/0 conn(0x7ff9700731d0 legacy=0x7ff97006d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff975e7c640 1 --1- 172.21.15.63:0/2537088801 >> v1:172.21.15.6:6789/0 conn(0x7ff9700731d0 0x7ff97006d860 :-1 s=OPENED pgs=235 cs=1 l=1).handle_message read tag failed 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff975e7c640 1 --1- 172.21.15.63:0/2537088801 >> v1:172.21.15.6:6789/0 conn(0x7ff9700731d0 0x7ff97006d860 :-1 s=OPENED pgs=235 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3950316421 0 0) 0x7ff970125720 con 0x7ff9701395a0 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff954003530 con 0x7ff9701395a0 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff968001bb0 con 0x7ff9701395a0 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:22.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.388+0000 7ff9667fc640 10 monclient(hunting): dump: 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3104737666 0 0) 0x7ff954003530 con 0x7ff9701395a0 2024-09-16T05:36:22.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 >> v1:172.21.15.63:6789/0 conn(0x7ff9700a22a0 legacy=0x7ff97006df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 >> v1:172.21.15.6:6789/0 conn(0x7ff9700731d0 legacy=0x7ff97006d860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 monclient: found mon.c 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9700a3ea0 con 0x7ff9701395a0 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 5 monclient: authenticate success, global_id 4649 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 10 monclient: _renew_subs 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 1 -- 172.21.15.63:0/2537088801 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff970124a70 con 0x7ff9701395a0 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 10 monclient: _renew_subs 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 1 -- 172.21.15.63:0/2537088801 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff970124270 con 0x7ff9701395a0 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff968001820 con 0x7ff9701395a0 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff968003b10 con 0x7ff9701395a0 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff967fff640 4 set_mon_vals no callback set 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 10 monclient: dump: 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.406 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9780e0640 1 -- 172.21.15.63:0/2537088801 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff958005090 con 0x7ff9701395a0 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff96800e8b0 con 0x7ff9701395a0 2024-09-16T05:36:22.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(40..40 src has 1..40) ==== 4634+0+0 (unknown 3230365772 0 0) 0x7ff968043200 con 0x7ff9701395a0 2024-09-16T05:36:22.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.392+0000 7ff975e7c640 10 monclient: get_auth_request con 0x7ff9540369b0 auth_method 0 2024-09-16T05:36:22.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.404+0000 7ff9667fc640 1 -- 172.21.15.63:0/2537088801 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff968014120 con 0x7ff9701395a0 2024-09-16T05:36:22.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.404+0000 7ff9667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.404+0000 7ff9667fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2024-09-16T05:36:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2024-09-16T05:36:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T05:36:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T05:36:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-16T05:36:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-16T05:36:22.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.688+0000 7f27d596c640 10 monclient: get_monmap_and_config 2024-09-16T05:36:22.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.688+0000 7f27d596c640 10 monclient: build_initial_monmap 2024-09-16T05:36:22.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.688+0000 7f27d596c640 10 monclient: monmap: 2024-09-16T05:36:22.691 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:22.691 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.691 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:22.690970+0000 2024-09-16T05:36:22.691 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:22.690970+0000 2024-09-16T05:36:22.691 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:22.691 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.692 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:22.692 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:22.692 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:22.692 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.688+0000 7f27d596c640 1 Processor -- start 2024-09-16T05:36:22.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.688+0000 7f27d596c640 1 -- start start 2024-09-16T05:36:22.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.688+0000 7f27d596c640 10 monclient: init 2024-09-16T05:36:22.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient(hunting): picked mon.noname-a con 0x7f27d009ca20 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient(hunting): picked mon.noname-b con 0x7f27d009ce00 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient(hunting): picked mon.noname-c con 0x7f27d0139250 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27d00a1bd0 con 0x7f27d009ca20 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27d009dc40 con 0x7f27d0139250 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27d009e2a0 con 0x7f27d009ce00 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient(hunting): authenticate will time out at 1039.468688s 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27cf7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f27d0139250 legacy=0x7f27d013b640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27c7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f27d009ce00 0x7f27d009d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43422/0 (socket says 172.21.15.63:43422) 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27c7fff640 1 -- 172.21.15.63:0/857991873 learned_addr learned my addr 172.21.15.63:0/857991873 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:22.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660929703 0 0) 0x7f27d009e2a0 con 0x7f27d009ce00 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient(hunting): my global_id is 4652 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27bc003530 con 0x7f27d009ce00 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742872169 0 0) 0x7f27d00a1bd0 con 0x7f27d009ca20 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient(hunting): my global_id is 4653 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27d009e2a0 con 0x7f27d009ca20 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2506716245 0 0) 0x7f27bc003530 con 0x7f27d009ce00 2024-09-16T05:36:22.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 >> v1:172.21.15.63:6789/0 conn(0x7f27d0139250 legacy=0x7f27d013b640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 >> v1:172.21.15.6:6789/0 conn(0x7f27d009ca20 legacy=0x7f27d00a14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 monclient: found mon.noname-b 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d009f2c0 con 0x7f27d009ce00 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:22.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f27b8001550 con 0x7f27d009ce00 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: dump: 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 5 monclient: authenticate success, global_id 4652 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27b80017b0 con 0x7f27d009ce00 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 1 -- 172.21.15.63:0/857991873 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f27b8001bc0 con 0x7f27d009ce00 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27ce7fc640 10 monclient: dump: 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.699 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.700 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.700 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.700 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.700 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:22.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 4 set_mon_vals no callback set 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient: shutdown 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 -- 172.21.15.63:0/857991873 >> v1:172.21.15.6:6790/0 conn(0x7f27d009ce00 legacy=0x7f27d009d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 -- 172.21.15.63:0/857991873 shutdown_connections 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 -- 172.21.15.63:0/857991873 >> 172.21.15.63:0/857991873 conn(0x7f27d00a2420 msgr2=0x7f27d00a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 -- 172.21.15.63:0/857991873 shutdown_connections 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 -- 172.21.15.63:0/857991873 wait complete. 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient: build_initial_monmap 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 10 monclient: monmap: 2024-09-16T05:36:22.705 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.692+0000 7f27d596c640 1 Processor -- start 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 1 -- start start 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: init 2024-09-16T05:36:22.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient(hunting): picked mon.noname-a con 0x7f27d009ca20 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient(hunting): picked mon.noname-b con 0x7f27d009ce00 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient(hunting): picked mon.noname-c con 0x7f27d0139250 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27d0135040 con 0x7f27d009ca20 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27d0136da0 con 0x7f27d009ce00 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27d0137080 con 0x7f27d0139250 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient(hunting): authenticate will time out at 1039.472561s 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27c7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f27d009ce00 legacy=0x7f27d0134930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27cf7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f27d0139250 0x7f27d01d1c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43436/0 (socket says 172.21.15.63:43436) 2024-09-16T05:36:22.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27cf7fe640 1 -- 172.21.15.63:0/2325363611 learned_addr learned my addr 172.21.15.63:0/2325363611 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2260434322 0 0) 0x7f27d0137080 con 0x7f27d0139250 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): my global_id is 4655 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27b0003560 con 0x7f27d0139250 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3489483513 0 0) 0x7f27d0135040 con 0x7f27d009ca20 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): my global_id is 4656 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27d0137080 con 0x7f27d009ca20 2024-09-16T05:36:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 688006338 0 0) 0x7f27b0003560 con 0x7f27d0139250 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27d0135040 con 0x7f27d0139250 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f27c8004300 con 0x7f27d0139250 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): dump: 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4212633836 0 0) 0x7f27d0137080 con 0x7f27d009ca20 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27b0003560 con 0x7f27d009ca20 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f27c00019a0 con 0x7f27d009ca20 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient(hunting): dump: 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.710 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4047179710 0 0) 0x7f27d0135040 con 0x7f27d0139250 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 >> v1:172.21.15.63:6789/0 conn(0x7f27d009ce00 legacy=0x7f27d0134930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 >> v1:172.21.15.6:6789/0 conn(0x7f27d009ca20 legacy=0x7f27d0134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 monclient: found mon.c 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:22.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d0138180 con 0x7f27d0139250 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 5 monclient: authenticate success, global_id 4655 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: _renew_subs 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 1 -- 172.21.15.63:0/2325363611 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27d0135440 con 0x7f27d0139250 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: _renew_subs 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 1 -- 172.21.15.63:0/2325363611 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f27d009bd30 con 0x7f27d0139250 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27c8004560 con 0x7f27d0139250 2024-09-16T05:36:22.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:22.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f27c8004930 con 0x7f27d0139250 2024-09-16T05:36:22.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27cdffb640 4 set_mon_vals no callback set 2024-09-16T05:36:22.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:22.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 10 monclient: dump: 2024-09-16T05:36:22.713 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.713 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27d596c640 1 -- 172.21.15.63:0/2325363611 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b4005090 con 0x7f27d0139250 2024-09-16T05:36:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f27c8004b90 con 0x7f27d0139250 2024-09-16T05:36:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.696+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(40..40 src has 1..40) ==== 4634+0+0 (unknown 3230365772 0 0) 0x7f27c8044770 con 0x7f27d0139250 2024-09-16T05:36:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.700+0000 7f27ceffd640 10 monclient: get_auth_request con 0x7f27b0036a50 auth_method 0 2024-09-16T05:36:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.708+0000 7f27ccff9640 1 -- 172.21.15.63:0/2325363611 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f27c8015630 con 0x7f27d0139250 2024-09-16T05:36:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.708+0000 7f27ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.708+0000 7f27ccff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:22.869 INFO:tasks.workunit.client.0.smithi063.stderr:foo not valid: foo not one of 'true', 'false' 2024-09-16T05:36:22.869 INFO:tasks.workunit.client.0.smithi063.stderr:Invalid command: unused arguments: ['foo'] 2024-09-16T05:36:22.869 INFO:tasks.workunit.client.0.smithi063.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T05:36:22.869 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: invalid command 2024-09-16T05:36:22.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:22.874 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2024-09-16T05:36:22.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient: get_monmap_and_config 2024-09-16T05:36:22.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient: build_initial_monmap 2024-09-16T05:36:22.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient: monmap: 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:22.984581+0000 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:22.984581+0000 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:22.985 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 Processor -- start 2024-09-16T05:36:22.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- start start 2024-09-16T05:36:22.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient: init 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient(hunting): picked mon.noname-c con 0x7f19040a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient(hunting): picked mon.noname-a con 0x7f19040a4d90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient(hunting): picked mon.noname-b con 0x7f19040a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19040a51e0 con 0x7f19040a4d90 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19040a0d60 con 0x7f19040a3ca0 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19040a1040 con 0x7f19040a4790 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient(hunting): authenticate will time out at 1039.761470s 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190a2d2640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f19040a3ca0 legacy=0x7f19040a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:22.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190aad3640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f19040a4790 0x7f190413f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:43450/0 (socket says 172.21.15.63:43450) 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190aad3640 1 -- 172.21.15.63:0/3126025273 learned_addr learned my addr 172.21.15.63:0/3126025273 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f1909ad1640 1 --1- 172.21.15.63:0/3126025273 >> v1:172.21.15.6:6789/0 conn(0x7f19040a4d90 0x7f190413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58942/0 (socket says 172.21.15.63:58942) 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2260319183 0 0) 0x7f19040a51e0 con 0x7f19040a4d90 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient(hunting): my global_id is 4659 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18f0003530 con 0x7f19040a4d90 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3344200687 0 0) 0x7f19040a1040 con 0x7f19040a4790 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient(hunting): _negotiate 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:22.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient(hunting): my global_id is 4658 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19040a51e0 con 0x7f19040a4790 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 496662322 0 0) 0x7f18f0003530 con 0x7f19040a4d90 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 >> v1:172.21.15.6:6790/0 conn(0x7f19040a4790 legacy=0x7f190413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 >> v1:172.21.15.63:6789/0 conn(0x7f19040a3ca0 legacy=0x7f19040a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: _finish_hunting 0 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 monclient: found mon.noname-a 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19040a0810 con 0x7f19040a4d90 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: _finish_auth 0 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: _check_auth_tickets 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 5 monclient: authenticate success, global_id 4659 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18f4001620 con 0x7f19040a4d90 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: dump: 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18f40018d0 con 0x7f19040a4d90 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 4 set_mon_vals no callback set 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 1 -- 172.21.15.63:0/3126025273 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18f4001ce0 con 0x7f19040a4d90 2024-09-16T05:36:22.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f19092d0640 10 monclient: dump: 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 10 monclient: shutdown 2024-09-16T05:36:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- 172.21.15.63:0/3126025273 >> v1:172.21.15.6:6789/0 conn(0x7f19040a4d90 legacy=0x7f190413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- 172.21.15.63:0/3126025273 shutdown_connections 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- 172.21.15.63:0/3126025273 >> 172.21.15.63:0/3126025273 conn(0x7f1904075470 msgr2=0x7f1904075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- 172.21.15.63:0/3126025273 shutdown_connections 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.984+0000 7f190c536640 1 -- 172.21.15.63:0/3126025273 wait complete. 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: build_initial_monmap 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: monmap: 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:22.996 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:22.996 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:22.996 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:22.996 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:22.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 1 Processor -- start 2024-09-16T05:36:22.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 1 -- start start 2024-09-16T05:36:22.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: init 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient(hunting): picked mon.noname-a con 0x7f19040a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient(hunting): picked mon.noname-b con 0x7f19040a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient(hunting): picked mon.noname-c con 0x7f19040a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1904072580 con 0x7f19040a3ca0 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f190413f9c0 con 0x7f19040a4790 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19040a83b0 con 0x7f19040a4d90 2024-09-16T05:36:22.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient(hunting): authenticate will time out at 1039.765116s 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f1909ad1640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f19040a4790 legacy=0x7f19041cffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190a2d2640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f19040a3ca0 0x7f1904136eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:58956/0 (socket says 172.21.15.63:58956) 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190a2d2640 1 -- 172.21.15.63:0/656581216 learned_addr learned my addr 172.21.15.63:0/656581216 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1846861209 0 0) 0x7f18fc003e20 con 0x7f19040a3ca0 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): my global_id is 4662 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18e8003530 con 0x7f19040a3ca0 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1852362253 0 0) 0x7f19040a83b0 con 0x7f19040a4d90 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:23.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): my global_id is 4661 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18fc003e20 con 0x7f19040a4d90 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1170884045 0 0) 0x7f18e8003530 con 0x7f19040a3ca0 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19040a83b0 con 0x7f19040a3ca0 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18fc0046b0 con 0x7f19040a3ca0 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): dump: 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:23.001 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2641296073 0 0) 0x7f18fc003e20 con 0x7f19040a4d90 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18e8003530 con 0x7f19040a4d90 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1900001880 con 0x7f19040a4d90 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:23.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient(hunting): dump: 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:23.003 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4225270939 0 0) 0x7f19040a83b0 con 0x7f19040a3ca0 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 >> v1:172.21.15.6:6790/0 conn(0x7f19040a4d90 legacy=0x7f19041d36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 >> v1:172.21.15.63:6789/0 conn(0x7f19040a4790 legacy=0x7f19041cffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 monclient: found mon.a 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19040a0500 con 0x7f19040a3ca0 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:23.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 5 monclient: authenticate success, global_id 4662 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: _renew_subs 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f190413f400 con 0x7f19040a3ca0 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: _renew_subs 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f190413dfb0 con 0x7f19040a3ca0 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18fc004850 con 0x7f19040a3ca0 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18fc004c20 con 0x7f19040a3ca0 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f1908acf640 4 set_mon_vals no callback set 2024-09-16T05:36:23.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f18faffd640 10 monclient: dump: 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:23.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:23.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:23.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.988+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18ec005090 con 0x7f19040a3ca0 2024-09-16T05:36:23.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.992+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f18fc004e40 con 0x7f19040a3ca0 2024-09-16T05:36:23.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.992+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(40..40 src has 1..40) ==== 4634+0+0 (unknown 3230365772 0 0) 0x7f18fc044a50 con 0x7f19040a3ca0 2024-09-16T05:36:23.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:22.992+0000 7f1909ad1640 10 monclient: get_auth_request con 0x7f18e80369e0 auth_method 0 2024-09-16T05:36:23.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:23.004+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1904072580 con 0x7f19040a3ca0 2024-09-16T05:36:23.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:23.004+0000 7f18faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:23.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:23.004+0000 7f18faffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:23.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:23.164+0000 7f190c536640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:23.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:23.164+0000 7f190c536640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:23.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:23.164+0000 7f190c536640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:23.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:23.164+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f18ec005380 con 0x7f19040a3ca0 2024-09-16T05:36:24.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.168+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' removed v41) ==== 155+0+0 (unknown 1020377899 0 0) 0x7f18fc0158b0 con 0x7f19040a3ca0 2024-09-16T05:36:24.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.168+0000 7f18faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:24.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.168+0000 7f18faffd640 10 monclient: _finish_command 2 = system:0 pool 'foooo' removed 2024-09-16T05:36:24.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.268+0000 7f190c536640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:24.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.268+0000 7f190c536640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:24.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.268+0000 7f190c536640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:24.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.268+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f18ec001150 con 0x7f19040a3ca0 2024-09-16T05:36:24.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.268+0000 7f18faffd640 1 -- 172.21.15.63:0/656581216 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' does not exist v41) ==== 162+0+0 (unknown 3326524616 0 0) 0x7f18fc0151b0 con 0x7f19040a3ca0 2024-09-16T05:36:24.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.268+0000 7f18faffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:24.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.268+0000 7f18faffd640 10 monclient: _finish_command 3 = system:0 pool 'foooo' does not exist 2024-09-16T05:36:24.271 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'foooo' does not exist 2024-09-16T05:36:24.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.272+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 >> v1:172.21.15.63:6812/3643207211 conn(0x7f18e80369e0 legacy=0x7f18e8038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:24.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.272+0000 7f190c536640 10 monclient: shutdown 2024-09-16T05:36:24.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.272+0000 7f190c536640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:24.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.272+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 >> v1:172.21.15.6:6789/0 conn(0x7f19040a3ca0 legacy=0x7f1904136eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:24.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.272+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 shutdown_connections 2024-09-16T05:36:24.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.272+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 >> 172.21.15.63:0/656581216 conn(0x7f1904075470 msgr2=0x7f19040a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:24.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.272+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 shutdown_connections 2024-09-16T05:36:24.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.272+0000 7f190c536640 1 -- 172.21.15.63:0/656581216 wait complete. 2024-09-16T05:36:24.287 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T05:36:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient: get_monmap_and_config 2024-09-16T05:36:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient: build_initial_monmap 2024-09-16T05:36:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient: monmap: 2024-09-16T05:36:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:24.412621+0000 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:24.412621+0000 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:24.413 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 Processor -- start 2024-09-16T05:36:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 -- start start 2024-09-16T05:36:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient: init 2024-09-16T05:36:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient(hunting): picked mon.noname-b con 0x7f5a740a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient(hunting): picked mon.noname-c con 0x7f5a740a4d90 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient(hunting): picked mon.noname-a con 0x7f5a740a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a740a51e0 con 0x7f5a740a4790 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a740a0d60 con 0x7f5a740a4d90 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a740a1040 con 0x7f5a740a3ca0 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient(hunting): authenticate will time out at 1041.189579s 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a7259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5a740a4d90 legacy=0x7f5a7413bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a72d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5a740a3ca0 0x7f5a740a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53218/0 (socket says 172.21.15.63:53218) 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a72d9d640 1 -- 172.21.15.63:0/1427257317 learned_addr learned my addr 172.21.15.63:0/1427257317 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1377895258 0 0) 0x7f5a740a51e0 con 0x7f5a740a4790 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient(hunting): my global_id is 4665 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a5c003530 con 0x7f5a740a4790 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2463330203 0 0) 0x7f5a740a1040 con 0x7f5a740a3ca0 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient(hunting): my global_id is 4664 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a740a51e0 con 0x7f5a740a3ca0 2024-09-16T05:36:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1172716193 0 0) 0x7f5a5c003530 con 0x7f5a740a4790 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 >> v1:172.21.15.6:6790/0 conn(0x7f5a740a3ca0 legacy=0x7f5a740a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 >> v1:172.21.15.63:6789/0 conn(0x7f5a740a4d90 legacy=0x7f5a7413bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 monclient: found mon.noname-a 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a740a0810 con 0x7f5a740a4790 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: _finish_auth 0 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 5 monclient: authenticate success, global_id 4665 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a6c0019a0 con 0x7f5a740a4790 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: dump: 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:24.420 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a6c001c50 con 0x7f5a740a4790 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 4 set_mon_vals no callback set 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 1 -- 172.21.15.63:0/1427257317 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a6c002020 con 0x7f5a740a4790 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a71d9b640 10 monclient: dump: 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 10 monclient: shutdown 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 -- 172.21.15.63:0/1427257317 >> v1:172.21.15.6:6789/0 conn(0x7f5a740a4790 legacy=0x7f5a7413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 -- 172.21.15.63:0/1427257317 shutdown_connections 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 -- 172.21.15.63:0/1427257317 >> 172.21.15.63:0/1427257317 conn(0x7f5a74075470 msgr2=0x7f5a74075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.412+0000 7f5a79087640 1 -- 172.21.15.63:0/1427257317 shutdown_connections 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 1 -- 172.21.15.63:0/1427257317 wait complete. 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: build_initial_monmap 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: monmap: 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:24.423 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 1 Processor -- start 2024-09-16T05:36:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 1 -- start start 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: init 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient(hunting): picked mon.noname-a con 0x7f5a740a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient(hunting): picked mon.noname-b con 0x7f5a740a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient(hunting): picked mon.noname-c con 0x7f5a740a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a74137ed0 con 0x7f5a740a3ca0 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a74134e50 con 0x7f5a740a4790 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a740a8320 con 0x7f5a740a4d90 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient(hunting): authenticate will time out at 1041.193478s 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a7259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5a740a4790 legacy=0x7f5a74134740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a72d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5a740a3ca0 0x7f5a74134030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43112/0 (socket says 172.21.15.63:43112) 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a72d9d640 1 -- 172.21.15.63:0/1192594593 learned_addr learned my addr 172.21.15.63:0/1192594593 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3900128725 0 0) 0x7f5a74137ed0 con 0x7f5a740a3ca0 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): my global_id is 4668 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a60003530 con 0x7f5a740a3ca0 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1383330681 0 0) 0x7f5a740a8320 con 0x7f5a740a4d90 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:24.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): my global_id is 4667 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a74137ed0 con 0x7f5a740a4d90 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2174883715 0 0) 0x7f5a60003530 con 0x7f5a740a3ca0 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a740a8320 con 0x7f5a740a3ca0 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a64001820 con 0x7f5a740a3ca0 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.430 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 635574380 0 0) 0x7f5a74137ed0 con 0x7f5a740a4d90 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a60003530 con 0x7f5a740a4d90 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a6c001fa0 con 0x7f5a740a4d90 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:24.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:24.432 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 359189616 0 0) 0x7f5a740a8320 con 0x7f5a740a3ca0 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 >> v1:172.21.15.6:6790/0 conn(0x7f5a740a4d90 legacy=0x7f5a741d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 >> v1:172.21.15.63:6789/0 conn(0x7f5a740a4790 legacy=0x7f5a74134740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 monclient: found mon.a 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a74137720 con 0x7f5a740a3ca0 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:24.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 5 monclient: authenticate success, global_id 4668 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: _renew_subs 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a74136570 con 0x7f5a740a3ca0 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: _renew_subs 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a7413fb90 con 0x7f5a740a3ca0 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a640047f0 con 0x7f5a740a3ca0 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a64004b40 con 0x7f5a740a3ca0 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a70d99640 4 set_mon_vals no callback set 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:24.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.416+0000 7f5a5b7fe640 10 monclient: dump: 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.420+0000 7f5a79087640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:24.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.420+0000 7f5a79087640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:24.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.420+0000 7f5a79087640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:24.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.420+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a54005090 con 0x7f5a740a3ca0 2024-09-16T05:36:24.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.420+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5a6400f8e0 con 0x7f5a740a3ca0 2024-09-16T05:36:24.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.420+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(41..41 src has 1..41) ==== 4290+0+0 (unknown 1851259169 0 0) 0x7f5a64044290 con 0x7f5a740a3ca0 2024-09-16T05:36:24.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.420+0000 7f5a7259c640 10 monclient: get_auth_request con 0x7f5a600369d0 auth_method 0 2024-09-16T05:36:24.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.432+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5a640153a0 con 0x7f5a740a3ca0 2024-09-16T05:36:24.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.432+0000 7f5a5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:24.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.432+0000 7f5a5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:24.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.600+0000 7f5a79087640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:24.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.600+0000 7f5a79087640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:24.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.600+0000 7f5a79087640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:24.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:24.600+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5a54005380 con 0x7f5a740a3ca0 2024-09-16T05:36:25.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.172+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' removed v42) ==== 152+0+0 (unknown 1694669109 0 0) 0x7f5a64014ca0 con 0x7f5a740a3ca0 2024-09-16T05:36:25.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.172+0000 7f5a5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:25.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.172+0000 7f5a5b7fe640 10 monclient: _finish_command 2 = system:0 pool 'fooo' removed 2024-09-16T05:36:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.324+0000 7f5a79087640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.324+0000 7f5a79087640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.324+0000 7f5a79087640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.324+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5a54001150 con 0x7f5a740a3ca0 2024-09-16T05:36:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.324+0000 7f5a5b7fe640 1 -- 172.21.15.63:0/1192594593 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v42) ==== 159+0+0 (unknown 784008757 0 0) 0x7f5a6401d8b0 con 0x7f5a740a3ca0 2024-09-16T05:36:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.324+0000 7f5a5b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.324+0000 7f5a5b7fe640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T05:36:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'fooo' does not exist 2024-09-16T05:36:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.328+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5a600369d0 legacy=0x7f5a60038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.328+0000 7f5a79087640 10 monclient: shutdown 2024-09-16T05:36:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.328+0000 7f5a79087640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.328+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 >> v1:172.21.15.6:6789/0 conn(0x7f5a740a3ca0 legacy=0x7f5a74134030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.328+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 shutdown_connections 2024-09-16T05:36:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.328+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 >> 172.21.15.63:0/1192594593 conn(0x7f5a74075470 msgr2=0x7f5a740a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.328+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 shutdown_connections 2024-09-16T05:36:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.328+0000 7f5a79087640 1 -- 172.21.15.63:0/1192594593 wait complete. 2024-09-16T05:36:25.344 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T05:36:25.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient: get_monmap_and_config 2024-09-16T05:36:25.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient: build_initial_monmap 2024-09-16T05:36:25.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient: monmap: 2024-09-16T05:36:25.466 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:25.466 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:25.466 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:25.466410+0000 2024-09-16T05:36:25.466 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:25.466410+0000 2024-09-16T05:36:25.467 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:25.467 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:25.467 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:25.467 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:25.467 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:25.467 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 1 Processor -- start 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 1 -- start start 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient: init 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa19c134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa19c0a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient(hunting): picked mon.noname-a con 0x7fa19c06dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:25.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c135420 con 0x7fa19c06dfa0 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c09e830 con 0x7fa19c0a0810 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c0a0dc0 con 0x7fa19c134930 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a2b6f640 10 monclient(hunting): authenticate will time out at 1042.243261s 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa19bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa19c0a0810 legacy=0x7fa19c139820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a090b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa19c134930 0x7fa19c134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53234/0 (socket says 172.21.15.63:53234) 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.464+0000 7fa1a090b640 1 -- 172.21.15.63:0/4194289406 learned_addr learned my addr 172.21.15.63:0/4194289406 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 581319433 0 0) 0x7fa19c0a0dc0 con 0x7fa19c134930 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient(hunting): my global_id is 4670 2024-09-16T05:36:25.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa184003530 con 0x7fa19c134930 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516600602 0 0) 0x7fa19c135420 con 0x7fa19c06dfa0 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient(hunting): my global_id is 4671 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa19c0a0dc0 con 0x7fa19c06dfa0 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2159093497 0 0) 0x7fa184003530 con 0x7fa19c134930 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 >> v1:172.21.15.63:6789/0 conn(0x7fa19c0a0810 legacy=0x7fa19c139820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 >> v1:172.21.15.6:6789/0 conn(0x7fa19c06dfa0 legacy=0x7fa19c13cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:25.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19c0a2500 con 0x7fa19c134930 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa1940019a0 con 0x7fa19c134930 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: dump: 2024-09-16T05:36:25.472 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:25.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 5 monclient: authenticate success, global_id 4670 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa194001cc0 con 0x7fa19c134930 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 1 -- 172.21.15.63:0/4194289406 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa194002090 con 0x7fa19c134930 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19b7fe640 10 monclient: dump: 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:25.474 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:25.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 4 set_mon_vals no callback set 2024-09-16T05:36:25.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient: shutdown 2024-09-16T05:36:25.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:25.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/4194289406 >> v1:172.21.15.6:6790/0 conn(0x7fa19c134930 legacy=0x7fa19c134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:25.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/4194289406 shutdown_connections 2024-09-16T05:36:25.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/4194289406 >> 172.21.15.63:0/4194289406 conn(0x7fa19c0a1e90 msgr2=0x7fa19c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:25.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/4194289406 shutdown_connections 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/4194289406 wait complete. 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient: build_initial_monmap 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient: monmap: 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:25.480 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 Processor -- start 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- start start 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient: init 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient(hunting): picked mon.noname-b con 0x7fa19c06dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient(hunting): picked mon.noname-c con 0x7fa19c0a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient(hunting): picked mon.noname-a con 0x7fa19c134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c132780 con 0x7fa19c134930 2024-09-16T05:36:25.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c072270 con 0x7fa19c06dfa0 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c09c0f0 con 0x7fa19c0a0810 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a090b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa19c06dfa0 legacy=0x7fa19c1cb9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a2b6f640 10 monclient(hunting): authenticate will time out at 1042.247312s 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa19c0a0810 0x7fa19c1d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53250/0 (socket says 172.21.15.63:53250) 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa1a110c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa19c134930 0x7fa19c1d4100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43138/0 (socket says 172.21.15.63:43138) 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.468+0000 7fa19bfff640 1 -- 172.21.15.63:0/1477868166 learned_addr learned my addr 172.21.15.63:0/1477868166 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969835193 0 0) 0x7fa19c09c0f0 con 0x7fa19c0a0810 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): my global_id is 4673 2024-09-16T05:36:25.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa188003530 con 0x7fa19c0a0810 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092219066 0 0) 0x7fa19c132780 con 0x7fa19c134930 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): my global_id is 4674 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa19c09c0f0 con 0x7fa19c134930 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3482582922 0 0) 0x7fa188003530 con 0x7fa19c0a0810 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa19c132780 con 0x7fa19c0a0810 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa190004160 con 0x7fa19c0a0810 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): dump: 2024-09-16T05:36:25.483 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3677202406 0 0) 0x7fa19c09c0f0 con 0x7fa19c134930 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa188003530 con 0x7fa19c134930 2024-09-16T05:36:25.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa18c0018c0 con 0x7fa19c134930 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient(hunting): dump: 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:25.485 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3594968168 0 0) 0x7fa19c132780 con 0x7fa19c0a0810 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 >> v1:172.21.15.63:6789/0 conn(0x7fa19c06dfa0 legacy=0x7fa19c1cb9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 >> v1:172.21.15.6:6789/0 conn(0x7fa19c134930 legacy=0x7fa19c1d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient: _finish_hunting 0 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 monclient: found mon.c 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19c131ad0 con 0x7fa19c0a0810 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient: _finish_auth 0 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient: _check_auth_tickets 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 5 monclient: authenticate success, global_id 4673 2024-09-16T05:36:25.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 10 monclient: _renew_subs 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa19c1324a0 con 0x7fa19c0a0810 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 10 monclient: _renew_subs 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa19c13dcb0 con 0x7fa19c0a0810 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa1900043c0 con 0x7fa19c0a0810 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa190004710 con 0x7fa19c0a0810 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa19affd640 4 set_mon_vals no callback set 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 10 monclient: dump: 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:25.487 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa17c005090 con 0x7fa19c0a0810 2024-09-16T05:36:25.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa190004930 con 0x7fa19c0a0810 2024-09-16T05:36:25.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(42..42 src has 1..42) ==== 3940+0+0 (unknown 2403909499 0 0) 0x7fa190044160 con 0x7fa19c0a0810 2024-09-16T05:36:25.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.472+0000 7fa1a090b640 10 monclient: get_auth_request con 0x7fa188036a00 auth_method 0 2024-09-16T05:36:25.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.484+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa1900153b0 con 0x7fa19c0a0810 2024-09-16T05:36:25.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.484+0000 7fa1997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:25.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.484+0000 7fa1997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:25.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.648+0000 7fa1a2b6f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:25.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.648+0000 7fa1a2b6f640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:25.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.648+0000 7fa1a2b6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:25.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.648+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa17c005380 con 0x7fa19c0a0810 2024-09-16T05:36:25.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.980+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' removed v43) ==== 149+0+0 (unknown 3480540784 0 0) 0x7fa190014cb0 con 0x7fa19c0a0810 2024-09-16T05:36:25.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.980+0000 7fa1997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:25.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:25.980+0000 7fa1997fa640 10 monclient: _finish_command 2 = system:0 pool 'foo' removed 2024-09-16T05:36:26.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.104+0000 7fa1a2b6f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.104+0000 7fa1a2b6f640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.104+0000 7fa1a2b6f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:26.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.104+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa17c001150 con 0x7fa19c0a0810 2024-09-16T05:36:26.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1997fa640 1 -- 172.21.15.63:0/1477868166 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v43) ==== 156+0+0 (unknown 648946890 0 0) 0x7fa190019000 con 0x7fa19c0a0810 2024-09-16T05:36:26.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1997fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1997fa640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-16T05:36:26.109 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'foo' does not exist 2024-09-16T05:36:26.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa188036a00 legacy=0x7fa188038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1a2b6f640 10 monclient: shutdown 2024-09-16T05:36:26.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1a2b6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:26.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 >> v1:172.21.15.6:6790/0 conn(0x7fa19c0a0810 legacy=0x7fa19c1d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 shutdown_connections 2024-09-16T05:36:26.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 >> 172.21.15.63:0/1477868166 conn(0x7fa19c0a1e90 msgr2=0x7fa19c0a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:26.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 shutdown_connections 2024-09-16T05:36:26.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.108+0000 7fa1a2b6f640 1 -- 172.21.15.63:0/1477868166 wait complete. 2024-09-16T05:36:26.125 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T05:36:26.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient: get_monmap_and_config 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient: build_initial_monmap 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient: monmap: 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:26.249674+0000 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:26.249674+0000 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:26.250 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 1 Processor -- start 2024-09-16T05:36:26.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 1 -- start start 2024-09-16T05:36:26.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient: init 2024-09-16T05:36:26.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:26.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:26.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a00a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:26.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a00a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:26.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a00a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a00a53b0 con 0x7fc1a00a07a0 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a00a0290 con 0x7fc1a00a3b10 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a00a0dc0 con 0x7fc1a00a4600 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a773e640 10 monclient(hunting): authenticate will time out at 1043.026995s 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a54da640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc1a00a3b10 legacy=0x7fc1a00a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a5cdb640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc1a00a4600 0x7fc1a013f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53258/0 (socket says 172.21.15.63:53258) 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a4cd9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc1a00a07a0 0x7fc1a013be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43154/0 (socket says 172.21.15.63:43154) 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc1a5cdb640 1 -- 172.21.15.63:0/2416632898 learned_addr learned my addr 172.21.15.63:0/2416632898 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 118323441 0 0) 0x7fc1a00a53b0 con 0x7fc1a00a07a0 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc197fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc197fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:26.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc197fff640 10 monclient(hunting): my global_id is 4677 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc18c003530 con 0x7fc1a00a07a0 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.248+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4260792053 0 0) 0x7fc1a00a0dc0 con 0x7fc1a00a4600 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient(hunting): my global_id is 4676 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1a00a53b0 con 0x7fc1a00a4600 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2308412544 0 0) 0x7fc18c003530 con 0x7fc1a00a07a0 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 >> v1:172.21.15.6:6790/0 conn(0x7fc1a00a4600 legacy=0x7fc1a013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 >> v1:172.21.15.63:6789/0 conn(0x7fc1a00a3b10 legacy=0x7fc1a00a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:26.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 monclient: found mon.noname-a 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a00a10a0 con 0x7fc1a00a07a0 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: _finish_auth 0 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc190001620 con 0x7fc1a00a07a0 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: dump: 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.259 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 5 monclient: authenticate success, global_id 4677 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1900018d0 con 0x7fc1a00a07a0 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 1 -- 172.21.15.63:0/2416632898 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc190001ca0 con 0x7fc1a00a07a0 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:26.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc197fff640 10 monclient: dump: 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:26.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 4 set_mon_vals no callback set 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient: shutdown 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- 172.21.15.63:0/2416632898 >> v1:172.21.15.6:6789/0 conn(0x7fc1a00a07a0 legacy=0x7fc1a013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- 172.21.15.63:0/2416632898 shutdown_connections 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- 172.21.15.63:0/2416632898 >> 172.21.15.63:0/2416632898 conn(0x7fc1a0075470 msgr2=0x7fc1a0075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- 172.21.15.63:0/2416632898 shutdown_connections 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- 172.21.15.63:0/2416632898 wait complete. 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient: build_initial_monmap 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient: monmap: 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:26.262 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 Processor -- start 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- start start 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient: init 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a00a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a00a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:26.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a00a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a0121310 con 0x7fc1a00a3b10 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a0123600 con 0x7fc1a00a07a0 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a0123320 con 0x7fc1a00a4600 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a54da640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc1a00a07a0 legacy=0x7fc1a0135c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a773e640 10 monclient(hunting): authenticate will time out at 1043.031123s 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a4cd9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc1a00a3b10 0x7fc1a01d0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43160/0 (socket says 172.21.15.63:43160) 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc1a4cd9640 1 -- 172.21.15.63:0/2547617920 learned_addr learned my addr 172.21.15.63:0/2547617920 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.252+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2264302993 0 0) 0x7fc1a0121310 con 0x7fc1a00a3b10 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:26.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): my global_id is 4680 2024-09-16T05:36:26.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc184003530 con 0x7fc1a00a3b10 2024-09-16T05:36:26.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3748144297 0 0) 0x7fc1a0123320 con 0x7fc1a00a4600 2024-09-16T05:36:26.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:26.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:26.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:26.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): my global_id is 4679 2024-09-16T05:36:26.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1a0121310 con 0x7fc1a00a4600 2024-09-16T05:36:26.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1433920995 0 0) 0x7fc184003530 con 0x7fc1a00a3b10 2024-09-16T05:36:26.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1a0123320 con 0x7fc1a00a3b10 2024-09-16T05:36:26.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc190001c20 con 0x7fc1a00a3b10 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): dump: 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.269 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 52135018 0 0) 0x7fc1a0121310 con 0x7fc1a00a4600 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc184003530 con 0x7fc1a00a4600 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc19c0018c0 con 0x7fc1a00a4600 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient(hunting): dump: 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.270 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3635659836 0 0) 0x7fc1a0123320 con 0x7fc1a00a3b10 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 >> v1:172.21.15.6:6790/0 conn(0x7fc1a00a4600 legacy=0x7fc1a01d3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 >> v1:172.21.15.63:6789/0 conn(0x7fc1a00a07a0 legacy=0x7fc1a0135c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 monclient: found mon.a 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a0121bf0 con 0x7fc1a00a3b10 2024-09-16T05:36:26.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 5 monclient: authenticate success, global_id 4680 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 10 monclient: _renew_subs 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a0136330 con 0x7fc1a00a3b10 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 10 monclient: _renew_subs 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1a01cb410 con 0x7fc1a00a3b10 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1900018d0 con 0x7fc1a00a3b10 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc190004af0 con 0x7fc1a00a3b10 2024-09-16T05:36:26.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1977fe640 4 set_mon_vals no callback set 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 10 monclient: dump: 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.273 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc180005090 con 0x7fc1a00a3b10 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc190004d10 con 0x7fc1a00a3b10 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7fc190044110 con 0x7fc1a00a3b10 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.256+0000 7fc1a54da640 10 monclient: get_auth_request con 0x7fc1840369d0 auth_method 0 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.272+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc190015580 con 0x7fc1a00a3b10 2024-09-16T05:36:26.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.272+0000 7fc195ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:26.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.272+0000 7fc195ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:26.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.436+0000 7fc1a773e640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.436+0000 7fc1a773e640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.436+0000 7fc1a773e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.436+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc180005380 con 0x7fc1a00a3b10 2024-09-16T05:36:26.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.436+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v43) ==== 156+0+0 (unknown 648946890 0 0) 0x7fc190014e80 con 0x7fc1a00a3b10 2024-09-16T05:36:26.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.436+0000 7fc195ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.436+0000 7fc195ffb640 10 monclient: _finish_command 2 = system:0 pool 'foo' does not exist 2024-09-16T05:36:26.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.532+0000 7fc1a773e640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.532+0000 7fc1a773e640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.532+0000 7fc1a773e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.532+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc180001150 con 0x7fc1a00a3b10 2024-09-16T05:36:26.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc195ffb640 1 -- 172.21.15.63:0/2547617920 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v43) ==== 156+0+0 (unknown 648946890 0 0) 0x7fc19001d9d0 con 0x7fc1a00a3b10 2024-09-16T05:36:26.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc195ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc195ffb640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-16T05:36:26.537 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'foo' does not exist 2024-09-16T05:36:26.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc1840369d0 legacy=0x7fc184038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc1a773e640 10 monclient: shutdown 2024-09-16T05:36:26.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc1a773e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:26.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 >> v1:172.21.15.6:6789/0 conn(0x7fc1a00a3b10 legacy=0x7fc1a01d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 shutdown_connections 2024-09-16T05:36:26.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 >> 172.21.15.63:0/2547617920 conn(0x7fc1a0075470 msgr2=0x7fc1a01beb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:26.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 shutdown_connections 2024-09-16T05:36:26.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.536+0000 7fc1a773e640 1 -- 172.21.15.63:0/2547617920 wait complete. 2024-09-16T05:36:26.554 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T05:36:26.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient: get_monmap_and_config 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient: build_initial_monmap 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient: monmap: 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:26.681811+0000 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:26.681811+0000 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:26.682 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 1 Processor -- start 2024-09-16T05:36:26.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 1 -- start start 2024-09-16T05:36:26.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient: init 2024-09-16T05:36:26.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:26.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:26.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient(hunting): picked mon.noname-a con 0x7f15140a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient(hunting): picked mon.noname-c con 0x7f15140a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient(hunting): picked mon.noname-b con 0x7f15140a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15140a53b0 con 0x7f15140a3b10 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15140a0290 con 0x7f15140a07a0 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15140a0dc0 con 0x7f15140a4600 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f15189c4640 10 monclient(hunting): authenticate will time out at 1043.459659s 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.680+0000 7f1509d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f15140a07a0 legacy=0x7f151413be20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1512d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f15140a4600 0x7f151413f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53266/0 (socket says 172.21.15.63:53266) 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1512d9d640 1 -- 172.21.15.63:0/3942674132 learned_addr learned my addr 172.21.15.63:0/3942674132 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:26.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f151259c640 1 --1- 172.21.15.63:0/3942674132 >> v1:172.21.15.6:6789/0 conn(0x7f15140a3b10 0x7f15140a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43174/0 (socket says 172.21.15.63:43174) 2024-09-16T05:36:26.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3801487115 0 0) 0x7f15140a53b0 con 0x7f15140a3b10 2024-09-16T05:36:26.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:26.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:26.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient(hunting): my global_id is 4683 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1500003530 con 0x7f15140a3b10 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2732205130 0 0) 0x7f15140a0dc0 con 0x7f15140a4600 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient(hunting): my global_id is 4682 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15140a53b0 con 0x7f15140a4600 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1624489932 0 0) 0x7f1500003530 con 0x7f15140a3b10 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 >> v1:172.21.15.6:6790/0 conn(0x7f15140a4600 legacy=0x7f151413f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 >> v1:172.21.15.63:6789/0 conn(0x7f15140a07a0 legacy=0x7f151413be20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:26.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 monclient: found mon.noname-a 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15140a10a0 con 0x7f15140a3b10 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: _finish_auth 0 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14fc001620 con 0x7f15140a3b10 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: dump: 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.692 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 5 monclient: authenticate success, global_id 4683 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14fc001880 con 0x7f15140a3b10 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 1 -- 172.21.15.63:0/3942674132 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14fc001c10 con 0x7f15140a3b10 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1511d9b640 10 monclient: dump: 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.694 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 4 set_mon_vals no callback set 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient: shutdown 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- 172.21.15.63:0/3942674132 >> v1:172.21.15.6:6789/0 conn(0x7f15140a3b10 legacy=0x7f15140a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- 172.21.15.63:0/3942674132 shutdown_connections 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- 172.21.15.63:0/3942674132 >> 172.21.15.63:0/3942674132 conn(0x7f1514075470 msgr2=0x7f1514075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- 172.21.15.63:0/3942674132 shutdown_connections 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- 172.21.15.63:0/3942674132 wait complete. 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient: build_initial_monmap 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient: monmap: 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:26.695 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 Processor -- start 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- start start 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient: init 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:26.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient(hunting): picked mon.noname-a con 0x7f15140a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient(hunting): picked mon.noname-b con 0x7f15140a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient(hunting): picked mon.noname-c con 0x7f15140a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1514125980 con 0x7f15140a07a0 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15140a8290 con 0x7f15140a3b10 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15140a0500 con 0x7f15140a4600 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f15189c4640 10 monclient(hunting): authenticate will time out at 1043.463695s 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.684+0000 7f1509d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f15140a3b10 legacy=0x7f1514136de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f1512d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f15140a4600 0x7f15141d3260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53268/0 (socket says 172.21.15.63:53268) 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f1512d9d640 1 -- 172.21.15.63:0/1139552982 learned_addr learned my addr 172.21.15.63:0/1139552982 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1779275453 0 0) 0x7f15140a0500 con 0x7f15140a4600 2024-09-16T05:36:26.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): my global_id is 4685 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14f4003560 con 0x7f15140a4600 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1061927294 0 0) 0x7f1514125980 con 0x7f15140a07a0 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): my global_id is 4686 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15140a0500 con 0x7f15140a07a0 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 545550271 0 0) 0x7f15140a0500 con 0x7f15140a07a0 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1514125980 con 0x7f15140a07a0 2024-09-16T05:36:26.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2770041183 0 0) 0x7f14f4003560 con 0x7f15140a4600 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15140a0500 con 0x7f15140a4600 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14fc001520 con 0x7f15140a07a0 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.699 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f150c001c00 con 0x7f15140a4600 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.700 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f1512d9d640 1 -- 172.21.15.63:0/1139552982 >> v1:172.21.15.6:6790/0 conn(0x7f15140a4600 legacy=0x7f15141d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f1512d9d640 1 -- 172.21.15.63:0/1139552982 >> v1:172.21.15.6:6790/0 conn(0x7f15140a4600 legacy=0x7f15141d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f1512d9d640 1 --1- 172.21.15.63:0/1139552982 >> v1:172.21.15.6:6790/0 conn(0x7f15140a4600 0x7f15141d3260 :-1 s=OPENED pgs=243 cs=1 l=1).handle_message read tag failed 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f1512d9d640 1 --1- 172.21.15.63:0/1139552982 >> v1:172.21.15.6:6790/0 conn(0x7f15140a4600 0x7f15141d3260 :-1 s=OPENED pgs=243 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:36:26.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6790/0 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2521418191 0 0) 0x7f1514125980 con 0x7f15140a07a0 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 >> v1:172.21.15.6:6790/0 conn(0x7f15140a4600 legacy=0x7f15141d3260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 >> v1:172.21.15.63:6789/0 conn(0x7f15140a3b10 legacy=0x7f1514136de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 monclient: found mon.a 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1514127210 con 0x7f15140a07a0 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 5 monclient: authenticate success, global_id 4686 2024-09-16T05:36:26.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 10 monclient: _renew_subs 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15141274f0 con 0x7f15140a07a0 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 10 monclient: _renew_subs 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f15141376c0 con 0x7f15140a07a0 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14fc001780 con 0x7f15140a07a0 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f151159a640 4 set_mon_vals no callback set 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f14fc004ce0 con 0x7f15140a07a0 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 10 monclient: dump: 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:26.703 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f8005090 con 0x7f15140a07a0 2024-09-16T05:36:26.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f14fc004f40 con 0x7f15140a07a0 2024-09-16T05:36:26.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7f14fc044180 con 0x7f15140a07a0 2024-09-16T05:36:26.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.688+0000 7f1509d9b640 10 monclient: get_auth_request con 0x7f14f4036a30 auth_method 0 2024-09-16T05:36:26.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.700+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f14fc0153f0 con 0x7f15140a07a0 2024-09-16T05:36:26.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.700+0000 7f150b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:26.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.700+0000 7f150b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:26.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.900+0000 7f15189c4640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.900+0000 7f15189c4640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.904+0000 7f15189c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:26.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.904+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f14f8005380 con 0x7f15140a07a0 2024-09-16T05:36:26.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.904+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) ==== 159+0+0 (unknown 1822715070 0 0) 0x7f14fc019040 con 0x7f15140a07a0 2024-09-16T05:36:26.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.904+0000 7f150b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:26.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:26.904+0000 7f150b7fe640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-16T05:36:27.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:27.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f14f8001180 con 0x7f15140a07a0 2024-09-16T05:36:27.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f150b7fe640 1 -- 172.21.15.63:0/1139552982 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) ==== 159+0+0 (unknown 1822715070 0 0) 0x7f14f8001180 con 0x7f15140a07a0 2024-09-16T05:36:27.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f150b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f150b7fe640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T05:36:27.005 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'fooo' does not exist 2024-09-16T05:36:27.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 >> v1:172.21.15.63:6812/3643207211 conn(0x7f14f4036a30 legacy=0x7f14f4038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 10 monclient: shutdown 2024-09-16T05:36:27.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:27.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 >> v1:172.21.15.6:6789/0 conn(0x7f15140a07a0 legacy=0x7f15141caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 shutdown_connections 2024-09-16T05:36:27.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 >> 172.21.15.63:0/1139552982 conn(0x7f1514075470 msgr2=0x7f151406ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:27.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 shutdown_connections 2024-09-16T05:36:27.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.004+0000 7f15189c4640 1 -- 172.21.15.63:0/1139552982 wait complete. 2024-09-16T05:36:27.022 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T05:36:27.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient: get_monmap_and_config 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient: build_initial_monmap 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient: monmap: 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:27.146880+0000 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:27.146880+0000 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:27.147 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 1 Processor -- start 2024-09-16T05:36:27.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 1 -- start start 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient: init 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient(hunting): picked mon.noname-b con 0x7fb004134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient(hunting): picked mon.noname-c con 0x7fb0040a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient(hunting): picked mon.noname-a con 0x7fb00406dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb004135420 con 0x7fb00406dfa0 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb00409e830 con 0x7fb0040a0810 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0040a0dc0 con 0x7fb004134930 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb0089b9640 10 monclient(hunting): authenticate will time out at 1043.923813s 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.144+0000 7fb001d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fb0040a0810 legacy=0x7fb004139820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb002d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb00406dfa0 0x7fb00413cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43188/0 (socket says 172.21.15.63:43188) 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb002d9d640 1 -- 172.21.15.63:0/2060468900 learned_addr learned my addr 172.21.15.63:0/2060468900 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393900552 0 0) 0x7fb004135420 con 0x7fb00406dfa0 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient(hunting): my global_id is 4689 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafec003530 con 0x7fb00406dfa0 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 929084283 0 0) 0x7fb0040a0dc0 con 0x7fb004134930 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient(hunting): my global_id is 4688 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb004135420 con 0x7fb004134930 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3313756485 0 0) 0x7fb004135420 con 0x7fb004134930 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 >> v1:172.21.15.63:6789/0 conn(0x7fb0040a0810 legacy=0x7fb004139820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 >> v1:172.21.15.6:6789/0 conn(0x7fb00406dfa0 legacy=0x7fb00413cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 monclient: found mon.noname-b 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0040a2500 con 0x7fb004134930 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: _finish_auth 0 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faff4001600 con 0x7fb004134930 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: dump: 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.154 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 5 monclient: authenticate success, global_id 4688 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faff4001860 con 0x7fb004134930 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 1 -- 172.21.15.63:0/2060468900 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faff4001cb0 con 0x7fb004134930 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00159a640 10 monclient: dump: 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 4 set_mon_vals no callback set 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient: shutdown 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- 172.21.15.63:0/2060468900 >> v1:172.21.15.6:6790/0 conn(0x7fb004134930 legacy=0x7fb004134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- 172.21.15.63:0/2060468900 shutdown_connections 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- 172.21.15.63:0/2060468900 >> 172.21.15.63:0/2060468900 conn(0x7fb0040a1e90 msgr2=0x7fb004075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- 172.21.15.63:0/2060468900 shutdown_connections 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- 172.21.15.63:0/2060468900 wait complete. 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient: build_initial_monmap 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient: monmap: 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:27.157 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:27.158 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 Processor -- start 2024-09-16T05:36:27.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- start start 2024-09-16T05:36:27.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient: init 2024-09-16T05:36:27.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:27.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:27.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient(hunting): picked mon.noname-b con 0x7fb00406dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:27.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient(hunting): picked mon.noname-a con 0x7fb0040a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient(hunting): picked mon.noname-c con 0x7fb004134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb004071e00 con 0x7fb0040a0810 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb00413e2e0 con 0x7fb00406dfa0 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb00259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fb00406dfa0 legacy=0x7fb00409c0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb004139eb0 con 0x7fb004134930 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb0089b9640 10 monclient(hunting): authenticate will time out at 1043.927628s 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb001d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb0040a0810 0x7fb00409c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43198/0 (socket says 172.21.15.63:43198) 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb001d9b640 1 -- 172.21.15.63:0/3796983981 learned_addr learned my addr 172.21.15.63:0/3796983981 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.148+0000 7fb002d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb004134930 0x7fb0041d3780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53288/0 (socket says 172.21.15.63:53288) 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352858813 0 0) 0x7fb004139eb0 con 0x7fb004134930 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:27.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): my global_id is 4691 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafe4003530 con 0x7fb004134930 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298417881 0 0) 0x7faff8003a50 con 0x7fb0040a0810 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): my global_id is 4692 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb004139eb0 con 0x7fb0040a0810 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3237409268 0 0) 0x7fafe4003530 con 0x7fb004134930 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faff8003a50 con 0x7fb004134930 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faffc001dc0 con 0x7fb004134930 2024-09-16T05:36:27.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3948387140 0 0) 0x7fb004139eb0 con 0x7fb0040a0810 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafe4003530 con 0x7fb0040a0810 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faff8004260 con 0x7fb0040a0810 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.165 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4264090903 0 0) 0x7faff8003a50 con 0x7fb004134930 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 >> v1:172.21.15.63:6789/0 conn(0x7fb00406dfa0 legacy=0x7fb00409c0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 >> v1:172.21.15.6:6789/0 conn(0x7fb0040a0810 legacy=0x7fb00409c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 monclient: found mon.c 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00413a190 con 0x7fb004134930 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:27.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 5 monclient: authenticate success, global_id 4691 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 10 monclient: _renew_subs 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb00409db20 con 0x7fb004134930 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 10 monclient: _renew_subs 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb00409dfc0 con 0x7fb004134930 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faffc002020 con 0x7fb004134930 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faffc004fc0 con 0x7fb004134930 2024-09-16T05:36:27.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb000d99640 4 set_mon_vals no callback set 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 10 monclient: dump: 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:27.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:27.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe8005090 con 0x7fb004134930 2024-09-16T05:36:27.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7faffc00fd60 con 0x7fb004134930 2024-09-16T05:36:27.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7faffc044430 con 0x7fb004134930 2024-09-16T05:36:27.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.152+0000 7fb00259c640 10 monclient: get_auth_request con 0x7fafe4036ac0 auth_method 0 2024-09-16T05:36:27.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.164+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7faffc0192f0 con 0x7fb004134930 2024-09-16T05:36:27.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.164+0000 7faff2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:27.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.164+0000 7faff2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.328+0000 7fb0089b9640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.328+0000 7fb0089b9640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.328+0000 7fb0089b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.328+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fafe8005380 con 0x7fb004134930 2024-09-16T05:36:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.332+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) ==== 159+0+0 (unknown 1822715070 0 0) 0x7faffc01dbb0 con 0x7fb004134930 2024-09-16T05:36:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.332+0000 7faff2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.332+0000 7faff2ffd640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-16T05:36:27.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.428+0000 7fb0089b9640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.428+0000 7fb0089b9640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.428+0000 7fb0089b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.428+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fafe8001180 con 0x7fb004134930 2024-09-16T05:36:27.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.428+0000 7faff2ffd640 1 -- 172.21.15.63:0/3796983981 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) ==== 159+0+0 (unknown 1822715070 0 0) 0x7fafe8001180 con 0x7fb004134930 2024-09-16T05:36:27.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.428+0000 7faff2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.428+0000 7faff2ffd640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T05:36:27.431 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'fooo' does not exist 2024-09-16T05:36:27.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.432+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 >> v1:172.21.15.63:6812/3643207211 conn(0x7fafe4036ac0 legacy=0x7fafe4038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.432+0000 7fb0089b9640 10 monclient: shutdown 2024-09-16T05:36:27.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.432+0000 7fb0089b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:27.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.432+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 >> v1:172.21.15.6:6790/0 conn(0x7fb004134930 legacy=0x7fb0041d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.432+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 shutdown_connections 2024-09-16T05:36:27.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.432+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 >> 172.21.15.63:0/3796983981 conn(0x7fb0040a1e90 msgr2=0x7fb0040712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:27.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.432+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 shutdown_connections 2024-09-16T05:36:27.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.432+0000 7fb0089b9640 1 -- 172.21.15.63:0/3796983981 wait complete. 2024-09-16T05:36:27.448 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 10 monclient: get_monmap_and_config 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 10 monclient: build_initial_monmap 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 10 monclient: monmap: 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:27.574015+0000 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:27.574015+0000 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:27.574 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:27.575 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:27.575 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 1 Processor -- start 2024-09-16T05:36:27.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 1 -- start start 2024-09-16T05:36:27.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 10 monclient: init 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 10 monclient(hunting): picked mon.noname-a con 0x7fac800a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.572+0000 7fac8792d640 10 monclient(hunting): picked mon.noname-c con 0x7fac800a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient(hunting): picked mon.noname-b con 0x7fac800a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac800a53b0 con 0x7fac800a3b10 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac84ec8640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fac800a07a0 legacy=0x7fac8013be20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac800a0290 con 0x7fac800a07a0 2024-09-16T05:36:27.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac800a0dc0 con 0x7fac800a4600 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient(hunting): authenticate will time out at 1044.352133s 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac856c9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fac800a3b10 0x7fac800a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43208/0 (socket says 172.21.15.63:43208) 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac856c9640 1 -- 172.21.15.63:0/4061528900 learned_addr learned my addr 172.21.15.63:0/4061528900 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4117740866 0 0) 0x7fac800a53b0 con 0x7fac800a3b10 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient(hunting): my global_id is 4695 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac6c003530 con 0x7fac800a3b10 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2691383575 0 0) 0x7fac800a0dc0 con 0x7fac800a4600 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:27.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:27.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:27.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient(hunting): my global_id is 4694 2024-09-16T05:36:27.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac800a53b0 con 0x7fac800a4600 2024-09-16T05:36:27.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1609341707 0 0) 0x7fac800a53b0 con 0x7fac800a4600 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 >> v1:172.21.15.63:6789/0 conn(0x7fac800a07a0 legacy=0x7fac8013be20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 >> v1:172.21.15.6:6789/0 conn(0x7fac800a3b10 legacy=0x7fac800a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 monclient: found mon.noname-b 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac800a10a0 con 0x7fac800a4600 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: _finish_auth 0 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fac78001980 con 0x7fac800a4600 2024-09-16T05:36:27.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 5 monclient: authenticate success, global_id 4694 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: dump: 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.585 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac78001be0 con 0x7fac800a4600 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 1 -- 172.21.15.63:0/4061528900 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fac78001fb0 con 0x7fac800a4600 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 4 set_mon_vals no callback set 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac77fff640 10 monclient: dump: 2024-09-16T05:36:27.586 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient: shutdown 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- 172.21.15.63:0/4061528900 >> v1:172.21.15.6:6790/0 conn(0x7fac800a4600 legacy=0x7fac8013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- 172.21.15.63:0/4061528900 shutdown_connections 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- 172.21.15.63:0/4061528900 >> 172.21.15.63:0/4061528900 conn(0x7fac80075470 msgr2=0x7fac80075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- 172.21.15.63:0/4061528900 shutdown_connections 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- 172.21.15.63:0/4061528900 wait complete. 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient: build_initial_monmap 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient: monmap: 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.588 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 Processor -- start 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 1 -- start start 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient: init 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.576+0000 7fac8792d640 10 monclient(hunting): picked mon.noname-c con 0x7fac800a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient(hunting): picked mon.noname-a con 0x7fac800a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient(hunting): picked mon.noname-b con 0x7fac800a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac800a0500 con 0x7fac800a3b10 2024-09-16T05:36:27.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac801271f0 con 0x7fac800a4600 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac801274d0 con 0x7fac800a07a0 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac85eca640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fac800a4600 legacy=0x7fac801d39c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient(hunting): authenticate will time out at 1044.356108s 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac84ec8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fac800a3b10 0x7fac801d0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43216/0 (socket says 172.21.15.63:43216) 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac84ec8640 1 -- 172.21.15.63:0/3912979182 learned_addr learned my addr 172.21.15.63:0/3912979182 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2214170144 0 0) 0x7fac70003a50 con 0x7fac800a3b10 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): my global_id is 4698 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac64003530 con 0x7fac800a3b10 2024-09-16T05:36:27.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727765923 0 0) 0x7fac801274d0 con 0x7fac800a07a0 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): my global_id is 4697 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac70003a50 con 0x7fac800a07a0 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 939447347 0 0) 0x7fac70003a50 con 0x7fac800a07a0 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac801274d0 con 0x7fac800a07a0 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fac7c001a40 con 0x7fac800a07a0 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): dump: 2024-09-16T05:36:27.591 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 627991675 0 0) 0x7fac64003530 con 0x7fac800a3b10 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac70003a50 con 0x7fac800a3b10 2024-09-16T05:36:27.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fac700042a0 con 0x7fac800a3b10 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient(hunting): dump: 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.593 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3486034253 0 0) 0x7fac801274d0 con 0x7fac800a07a0 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 >> v1:172.21.15.63:6789/0 conn(0x7fac800a4600 legacy=0x7fac801d39c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 >> v1:172.21.15.6:6789/0 conn(0x7fac800a3b10 legacy=0x7fac801d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 monclient: found mon.c 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac800a86c0 con 0x7fac800a07a0 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 5 monclient: authenticate success, global_id 4697 2024-09-16T05:36:27.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient: _renew_subs 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fac80126f10 con 0x7fac800a07a0 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient: _renew_subs 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fac801cbdb0 con 0x7fac800a07a0 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac7c001d30 con 0x7fac800a07a0 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fac7c004c40 con 0x7fac800a07a0 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac777fe640 4 set_mon_vals no callback set 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 10 monclient: dump: 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac60005090 con 0x7fac800a07a0 2024-09-16T05:36:27.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fac7c00f9e0 con 0x7fac800a07a0 2024-09-16T05:36:27.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7fac7c044290 con 0x7fac800a07a0 2024-09-16T05:36:27.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.580+0000 7fac84ec8640 10 monclient: get_auth_request con 0x7fac64036a20 auth_method 0 2024-09-16T05:36:27.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.592+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fac7c015500 con 0x7fac800a07a0 2024-09-16T05:36:27.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.592+0000 7fac75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:27.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.592+0000 7fac75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:27.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.756+0000 7fac8792d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.756+0000 7fac8792d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.756+0000 7fac8792d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.756+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) -- 0x7fac60005380 con 0x7fac800a07a0 2024-09-16T05:36:27.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.760+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v43) ==== 162+0+0 (unknown 2112045314 0 0) 0x7fac7c014e00 con 0x7fac800a07a0 2024-09-16T05:36:27.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.760+0000 7fac75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.760+0000 7fac75ffb640 10 monclient: _finish_command 2 = system:0 pool 'fuggg' does not exist 2024-09-16T05:36:27.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.856+0000 7fac8792d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.856+0000 7fac8792d640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.856+0000 7fac8792d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:27.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.856+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) -- 0x7fac60001180 con 0x7fac800a07a0 2024-09-16T05:36:27.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.856+0000 7fac75ffb640 1 -- 172.21.15.63:0/3912979182 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v43) ==== 162+0+0 (unknown 2112045314 0 0) 0x7fac60001180 con 0x7fac800a07a0 2024-09-16T05:36:27.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.856+0000 7fac75ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:27.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.856+0000 7fac75ffb640 10 monclient: _finish_command 3 = system:0 pool 'fuggg' does not exist 2024-09-16T05:36:27.860 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'fuggg' does not exist 2024-09-16T05:36:27.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.860+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 >> v1:172.21.15.63:6812/3643207211 conn(0x7fac64036a20 legacy=0x7fac64038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.860+0000 7fac8792d640 10 monclient: shutdown 2024-09-16T05:36:27.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.860+0000 7fac8792d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:27.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.860+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 >> v1:172.21.15.6:6790/0 conn(0x7fac800a07a0 legacy=0x7fac801cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:27.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.860+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 shutdown_connections 2024-09-16T05:36:27.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.860+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 >> 172.21.15.63:0/3912979182 conn(0x7fac80075470 msgr2=0x7fac800a7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:27.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.860+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 shutdown_connections 2024-09-16T05:36:27.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:27.860+0000 7fac8792d640 1 -- 172.21.15.63:0/3912979182 wait complete. 2024-09-16T05:36:27.877 INFO:tasks.workunit.client.0.smithi063.stdout:OK 2024-09-16T05:36:27.878 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo OK 2024-09-16T05:36:27.878 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:36:27.878 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:36:27.932 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2024-09-16T05:36:27.934 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2024-09-16T05:36:27.934 DEBUG:teuthology.orchestra.run.smithi063:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-09-16T05:36:27.985 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush dump 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient: get_monmap_and_config 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient: build_initial_monmap 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient: monmap: 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:28.108921+0000 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:28.108921+0000 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:28.109 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:28.110 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 1 Processor -- start 2024-09-16T05:36:28.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 1 -- start start 2024-09-16T05:36:28.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient: init 2024-09-16T05:36:28.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:28.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:28.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient(hunting): picked mon.noname-b con 0x7fa10c133810 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:28.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient(hunting): picked mon.noname-c con 0x7fa10c133bf0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:28.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient(hunting): picked mon.noname-a con 0x7fa10c134020 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:28.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10c09d3c0 con 0x7fa10c134020 2024-09-16T05:36:28.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10c09d910 con 0x7fa10c133bf0 2024-09-16T05:36:28.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10c0a1560 con 0x7fa10c133810 2024-09-16T05:36:28.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:28.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa112ded640 10 monclient(hunting): authenticate will time out at 1044.886715s 2024-09-16T05:36:28.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa10c133bf0 legacy=0x7fa10c138700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:28.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa11138a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa10c134020 0x7fa10c13be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43232/0 (socket says 172.21.15.63:43232) 2024-09-16T05:36:28.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa11138a640 1 -- 172.21.15.63:0/3070187650 learned_addr learned my addr 172.21.15.63:0/3070187650 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:28.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279895628 0 0) 0x7fa10c0a1560 con 0x7fa10c133810 2024-09-16T05:36:28.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient(hunting): my global_id is 4700 2024-09-16T05:36:28.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0fc003530 con 0x7fa10c133810 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641279009 0 0) 0x7fa10c09d3c0 con 0x7fa10c134020 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient(hunting): my global_id is 4701 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa10c0a1560 con 0x7fa10c134020 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 969300748 0 0) 0x7fa0fc003530 con 0x7fa10c133810 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 >> v1:172.21.15.63:6789/0 conn(0x7fa10c133bf0 legacy=0x7fa10c138700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 >> v1:172.21.15.6:6789/0 conn(0x7fa10c134020 legacy=0x7fa10c13be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:36:28.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.108+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10c0a0a60 con 0x7fa10c133810 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa100001620 con 0x7fa10c133810 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 5 monclient: authenticate success, global_id 4700 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: dump: 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.120 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa1000018c0 con 0x7fa10c133810 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 4 set_mon_vals no callback set 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 1 -- 172.21.15.63:0/3070187650 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa100001cd0 con 0x7fa10c133810 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10b7fe640 10 monclient: dump: 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.122 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient: shutdown 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- 172.21.15.63:0/3070187650 >> v1:172.21.15.6:6790/0 conn(0x7fa10c133810 legacy=0x7fa10c09dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- 172.21.15.63:0/3070187650 shutdown_connections 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- 172.21.15.63:0/3070187650 >> 172.21.15.63:0/3070187650 conn(0x7fa10c0a1bf0 msgr2=0x7fa10c0a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- 172.21.15.63:0/3070187650 shutdown_connections 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- 172.21.15.63:0/3070187650 wait complete. 2024-09-16T05:36:28.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient: build_initial_monmap 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient: monmap: 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 Processor -- start 2024-09-16T05:36:28.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- start start 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient: init 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient(hunting): picked mon.noname-a con 0x7fa10c133810 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient(hunting): picked mon.noname-c con 0x7fa10c133bf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient(hunting): picked mon.noname-b con 0x7fa10c134020 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10c11e2f0 con 0x7fa10c133810 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10c072f10 con 0x7fa10c134020 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10c11e010 con 0x7fa10c133bf0 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa11138a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa10c134020 legacy=0x7fa10c1cf3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:28.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa112ded640 10 monclient(hunting): authenticate will time out at 1044.890455s 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa110b89640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa10c133810 0x7fa10c130930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43240/0 (socket says 172.21.15.63:43240) 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa110b89640 1 -- 172.21.15.63:0/2551693016 learned_addr learned my addr 172.21.15.63:0/2551693016 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa10bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa10c133bf0 0x7fa10c1cbc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53322/0 (socket says 172.21.15.63:53322) 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 770756256 0 0) 0x7fa10c11e010 con 0x7fa10c133bf0 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): my global_id is 4703 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f0003530 con 0x7fa10c133bf0 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3318526510 0 0) 0x7fa10c11e2f0 con 0x7fa10c133810 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): my global_id is 4704 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa10c11e010 con 0x7fa10c133810 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1607815529 0 0) 0x7fa0f0003530 con 0x7fa10c133bf0 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa10c11e2f0 con 0x7fa10c133bf0 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa0f80040d0 con 0x7fa10c133bf0 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): dump: 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.127 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2234122642 0 0) 0x7fa10c11e010 con 0x7fa10c133810 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0f0003530 con 0x7fa10c133810 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa100001cd0 con 0x7fa10c133810 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.112+0000 7fa1097fa640 10 monclient(hunting): dump: 2024-09-16T05:36:28.128 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.129 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1128511274 0 0) 0x7fa10c11e2f0 con 0x7fa10c133bf0 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 >> v1:172.21.15.63:6789/0 conn(0x7fa10c134020 legacy=0x7fa10c1cf3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 >> v1:172.21.15.6:6789/0 conn(0x7fa10c133810 legacy=0x7fa10c130930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 10 monclient: _finish_hunting 0 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 monclient: found mon.c 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10c11c8e0 con 0x7fa10c133bf0 2024-09-16T05:36:28.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 10 monclient: _finish_auth 0 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 10 monclient: _check_auth_tickets 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 5 monclient: authenticate success, global_id 4703 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 10 monclient: _renew_subs 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa10c071e80 con 0x7fa10c133bf0 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 10 monclient: _renew_subs 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa10c1c6fc0 con 0x7fa10c133bf0 2024-09-16T05:36:28.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa0f8004330 con 0x7fa10c133bf0 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa0f8004680 con 0x7fa10c133bf0 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa10affd640 4 set_mon_vals no callback set 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 10 monclient: dump: 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.132 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0ec005090 con 0x7fa10c133bf0 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa0f8004820 con 0x7fa10c133bf0 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7fa0f8043f90 con 0x7fa10c133bf0 2024-09-16T05:36:28.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.116+0000 7fa110b89640 10 monclient: get_auth_request con 0x7fa0f0036af0 auth_method 0 2024-09-16T05:36:28.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.128+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa0f80151a0 con 0x7fa10c133bf0 2024-09-16T05:36:28.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.128+0000 7fa1097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.128+0000 7fa1097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:28.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.292+0000 7fa112ded640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-16T05:36:28.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.292+0000 7fa112ded640 10 monclient: _send_command 2 [{"prefix": "osd crush dump"}] 2024-09-16T05:36:28.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.292+0000 7fa112ded640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.292+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush dump"} v 0) -- 0x7fa0ec005380 con 0x7fa10c133bf0 2024-09-16T05:36:28.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.296+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v43) ==== 62+0+7331 (unknown 3486025619 0 1011980395) 0x7fa0f8018e50 con 0x7fa10c133bf0 2024-09-16T05:36:28.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.296+0000 7fa1097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush dump"}] 2024-09-16T05:36:28.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.296+0000 7fa1097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:28.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.392+0000 7fa112ded640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-16T05:36:28.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.392+0000 7fa112ded640 10 monclient: _send_command 3 [{"prefix": "osd crush dump"}] 2024-09-16T05:36:28.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.392+0000 7fa112ded640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.392+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush dump"} v 0) -- 0x7fa0ec0071d0 con 0x7fa10c133bf0 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa1097fa640 1 -- 172.21.15.63:0/2551693016 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v43) ==== 62+0+7331 (unknown 3486025619 0 1011980395) 0x7fa0f801d710 con 0x7fa10c133bf0 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa1097fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush dump"}] 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa1097fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stdout:{ 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stdout: "devices": [ 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 0, 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "osd.0", 2024-09-16T05:36:28.398 INFO:tasks.workunit.client.0.smithi063.stdout: "class": "ssd" 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 1, 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "osd.1", 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: "class": "ssd" 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 2, 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "osd.2", 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: "class": "ssd" 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.399 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 3, 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "osd.3", 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: "class": "ssd" 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 4, 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "osd.4", 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: "class": "ssd" 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 5, 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "osd.5", 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: "class": "ssd" 2024-09-16T05:36:28.400 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: "types": [ 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 0, 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "osd" 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 1, 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "host" 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 2, 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "chassis" 2024-09-16T05:36:28.401 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 3, 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "rack" 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 4, 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "row" 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 5, 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "pdu" 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.402 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 6, 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "pod" 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 7, 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "room" 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 8, 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "datacenter" 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.403 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 9, 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "zone" 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 10, 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "region" 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 11, 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "root" 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: "buckets": [ 2024-09-16T05:36:28.404 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -1, 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "default", 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 11, 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "type_name": "root", 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 34560, 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "alg": "straw2", 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "hash": "rjenkins1", 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "items": [ 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -3, 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 17280, 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 0 2024-09-16T05:36:28.405 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -5, 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 17280, 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 1 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -2, 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "default~ssd", 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 11, 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "type_name": "root", 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 34560, 2024-09-16T05:36:28.406 INFO:tasks.workunit.client.0.smithi063.stdout: "alg": "straw2", 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: "hash": "rjenkins1", 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: "items": [ 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -4, 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 17280, 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 0 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -6, 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 17280, 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 1 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.407 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -3, 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "smithi006", 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 1, 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "type_name": "host", 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 17280, 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "alg": "straw2", 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "hash": "rjenkins1", 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "items": [ 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 2, 2024-09-16T05:36:28.408 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 0 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 1, 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 1 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 0, 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 2 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.409 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.410 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.410 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.410 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -4, 2024-09-16T05:36:28.410 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "smithi006~ssd", 2024-09-16T05:36:28.410 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 1, 2024-09-16T05:36:28.410 INFO:tasks.workunit.client.0.smithi063.stdout: "type_name": "host", 2024-09-16T05:36:28.410 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 17280, 2024-09-16T05:36:28.410 INFO:tasks.workunit.client.0.smithi063.stdout: "alg": "straw2", 2024-09-16T05:36:28.411 INFO:tasks.workunit.client.0.smithi063.stdout: "hash": "rjenkins1", 2024-09-16T05:36:28.411 INFO:tasks.workunit.client.0.smithi063.stdout: "items": [ 2024-09-16T05:36:28.411 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.411 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 2, 2024-09-16T05:36:28.411 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.411 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 0 2024-09-16T05:36:28.411 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 1, 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 1 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 0, 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 2 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.412 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -5, 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "smithi063", 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 1, 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "type_name": "host", 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 17280, 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "alg": "straw2", 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "hash": "rjenkins1", 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "items": [ 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 3, 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 0 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.413 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 4, 2024-09-16T05:36:28.414 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.414 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 1 2024-09-16T05:36:28.414 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.414 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.414 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 5, 2024-09-16T05:36:28.414 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.414 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 2 2024-09-16T05:36:28.414 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: "id": -6, 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: "name": "smithi063~ssd", 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: "type_id": 1, 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: "type_name": "host", 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 17280, 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: "alg": "straw2", 2024-09-16T05:36:28.415 INFO:tasks.workunit.client.0.smithi063.stdout: "hash": "rjenkins1", 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: "items": [ 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 3, 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 0 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 4, 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 1 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.416 INFO:tasks.workunit.client.0.smithi063.stdout: "id": 5, 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: "weight": 5760, 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: "pos": 2 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: "rules": [ 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.417 INFO:tasks.workunit.client.0.smithi063.stdout: "rule_id": 0, 2024-09-16T05:36:28.418 INFO:tasks.workunit.client.0.smithi063.stdout: "rule_name": "replicated_rule", 2024-09-16T05:36:28.418 INFO:tasks.workunit.client.0.smithi063.stdout: "type": 1, 2024-09-16T05:36:28.418 INFO:tasks.workunit.client.0.smithi063.stdout: "steps": [ 2024-09-16T05:36:28.418 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.418 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "take", 2024-09-16T05:36:28.418 INFO:tasks.workunit.client.0.smithi063.stdout: "item": -1, 2024-09-16T05:36:28.418 INFO:tasks.workunit.client.0.smithi063.stdout: "item_name": "default" 2024-09-16T05:36:28.419 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.419 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.419 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "choose_firstn", 2024-09-16T05:36:28.419 INFO:tasks.workunit.client.0.smithi063.stdout: "num": 0, 2024-09-16T05:36:28.419 INFO:tasks.workunit.client.0.smithi063.stdout: "type": "osd" 2024-09-16T05:36:28.419 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.420 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.420 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "emit" 2024-09-16T05:36:28.420 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.420 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.420 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.420 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.420 INFO:tasks.workunit.client.0.smithi063.stdout: "rule_id": 1, 2024-09-16T05:36:28.420 INFO:tasks.workunit.client.0.smithi063.stdout: "rule_name": "erasure-code", 2024-09-16T05:36:28.421 INFO:tasks.workunit.client.0.smithi063.stdout: "type": 3, 2024-09-16T05:36:28.421 INFO:tasks.workunit.client.0.smithi063.stdout: "steps": [ 2024-09-16T05:36:28.421 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.421 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "set_chooseleaf_tries", 2024-09-16T05:36:28.421 INFO:tasks.workunit.client.0.smithi063.stdout: "num": 5 2024-09-16T05:36:28.421 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.422 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.422 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "set_choose_tries", 2024-09-16T05:36:28.422 INFO:tasks.workunit.client.0.smithi063.stdout: "num": 100 2024-09-16T05:36:28.422 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.422 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.423 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "take", 2024-09-16T05:36:28.423 INFO:tasks.workunit.client.0.smithi063.stdout: "item": -1, 2024-09-16T05:36:28.423 INFO:tasks.workunit.client.0.smithi063.stdout: "item_name": "default" 2024-09-16T05:36:28.423 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.423 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.423 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "choose_indep", 2024-09-16T05:36:28.424 INFO:tasks.workunit.client.0.smithi063.stdout: "num": 0, 2024-09-16T05:36:28.424 INFO:tasks.workunit.client.0.smithi063.stdout: "type": "osd" 2024-09-16T05:36:28.424 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.424 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.424 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "emit" 2024-09-16T05:36:28.424 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.424 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.425 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.425 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:36:28.425 INFO:tasks.workunit.client.0.smithi063.stdout: "tunables": { 2024-09-16T05:36:28.425 INFO:tasks.workunit.client.0.smithi063.stdout: "choose_local_tries": 0, 2024-09-16T05:36:28.425 INFO:tasks.workunit.client.0.smithi063.stdout: "choose_local_fallback_tries": 0, 2024-09-16T05:36:28.425 INFO:tasks.workunit.client.0.smithi063.stdout: "choose_total_tries": 50, 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "chooseleaf_descend_once": 1, 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "chooseleaf_vary_r": 1, 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "chooseleaf_stable": 1, 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "msr_descents": 100, 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "msr_collision_tries": 100, 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "straw_calc_version": 1, 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "allowed_bucket_algs": 54, 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "profile": "jewel", 2024-09-16T05:36:28.426 INFO:tasks.workunit.client.0.smithi063.stdout: "optimal_tunables": 1, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "legacy_tunables": 0, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "minimum_required_version": "jewel", 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "require_feature_tunables": 1, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "require_feature_tunables2": 1, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "has_v2_rules": 1, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "require_feature_tunables3": 1, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "has_v3_rules": 0, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "has_v4_buckets": 1, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "require_feature_tunables5": 1, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "has_v5_rules": 0, 2024-09-16T05:36:28.427 INFO:tasks.workunit.client.0.smithi063.stdout: "has_msr_rules": 0 2024-09-16T05:36:28.428 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.428 INFO:tasks.workunit.client.0.smithi063.stdout: "choose_args": {} 2024-09-16T05:36:28.428 INFO:tasks.workunit.client.0.smithi063.stdout:} 2024-09-16T05:36:28.428 INFO:tasks.workunit.client.0.smithi063.stdout: 2024-09-16T05:36:28.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa0f0036af0 legacy=0x7fa0f0038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa112ded640 10 monclient: shutdown 2024-09-16T05:36:28.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa112ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:28.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 >> v1:172.21.15.6:6790/0 conn(0x7fa10c133bf0 legacy=0x7fa10c1cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 shutdown_connections 2024-09-16T05:36:28.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 >> 172.21.15.63:0/2551693016 conn(0x7fa10c0a1bf0 msgr2=0x7fa10c132c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:28.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 shutdown_connections 2024-09-16T05:36:28.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.396+0000 7fa112ded640 1 -- 172.21.15.63:0/2551693016 wait complete. 2024-09-16T05:36:28.430 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule dump 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.532+0000 7f53388b9640 10 monclient: get_monmap_and_config 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.532+0000 7f53388b9640 10 monclient: build_initial_monmap 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.532+0000 7f53388b9640 10 monclient: monmap: 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:28.535869+0000 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:28.535869+0000 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:28.537 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:28.538 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:28.538 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 Processor -- start 2024-09-16T05:36:28.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- start start 2024-09-16T05:36:28.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient: init 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient(hunting): picked mon.noname-a con 0x7f5334075600 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient(hunting): picked mon.noname-b con 0x7f5334072c00 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient(hunting): picked mon.noname-c con 0x7f5334139eb0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5334070720 con 0x7f5334075600 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f533406ce20 con 0x7f5334139eb0 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f533406f680 con 0x7f5334072c00 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient(hunting): authenticate will time out at 1045.313512s 2024-09-16T05:36:28.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f5332d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5334139eb0 legacy=0x7f533413c2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f5331d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5334072c00 0x7f5334138790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53324/0 (socket says 172.21.15.63:53324) 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5334075600 0x7f53340724f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43244/0 (socket says 172.21.15.63:43244) 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f5331d9b640 1 -- 172.21.15.63:0/2223558938 learned_addr learned my addr 172.21.15.63:0/2223558938 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3185446575 0 0) 0x7f533406f680 con 0x7f5334072c00 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient(hunting): my global_id is 4706 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5320005740 con 0x7f5334072c00 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279492330 0 0) 0x7f53240014a0 con 0x7f5334075600 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient(hunting): my global_id is 4707 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f533406f680 con 0x7f5334075600 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 679711152 0 0) 0x7f5320005740 con 0x7f5334072c00 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 >> v1:172.21.15.63:6789/0 conn(0x7f5334139eb0 legacy=0x7f533413c2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 >> v1:172.21.15.6:6789/0 conn(0x7f5334075600 legacy=0x7f53340724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 monclient: found mon.noname-b 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5334070fc0 con 0x7f5334072c00 2024-09-16T05:36:28.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: _finish_auth 0 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f531c001620 con 0x7f5334072c00 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: dump: 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.549 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 5 monclient: authenticate success, global_id 4706 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f531c001880 con 0x7f5334072c00 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 1 -- 172.21.15.63:0/2223558938 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f531c001bd0 con 0x7f5334072c00 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 4 set_mon_vals no callback set 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:28.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f533159a640 10 monclient: dump: 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 10 monclient: shutdown 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:28.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- 172.21.15.63:0/2223558938 >> v1:172.21.15.6:6790/0 conn(0x7f5334072c00 legacy=0x7f5334138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- 172.21.15.63:0/2223558938 shutdown_connections 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- 172.21.15.63:0/2223558938 >> 172.21.15.63:0/2223558938 conn(0x7f53340a2b80 msgr2=0x7f533406bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- 172.21.15.63:0/2223558938 shutdown_connections 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.536+0000 7f53388b9640 1 -- 172.21.15.63:0/2223558938 wait complete. 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: build_initial_monmap 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: monmap: 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.552 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 1 Processor -- start 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 1 -- start start 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: init 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient(hunting): picked mon.noname-b con 0x7f5334072c00 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient(hunting): picked mon.noname-c con 0x7f5334075600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient(hunting): picked mon.noname-a con 0x7f5334139eb0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53340a1560 con 0x7f5334139eb0 2024-09-16T05:36:28.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f533259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5334072c00 legacy=0x7f5334132920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5334139a10 con 0x7f5334072c00 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5334073550 con 0x7f5334075600 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient(hunting): authenticate will time out at 1045.317486s 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f5332d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5334139eb0 0x7f53341d3d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43248/0 (socket says 172.21.15.63:43248) 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f5332d9d640 1 -- 172.21.15.63:0/2697078988 learned_addr learned my addr 172.21.15.63:0/2697078988 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330031801 0 0) 0x7f5334073550 con 0x7f5334075600 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): my global_id is 4709 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5314003530 con 0x7f5334075600 2024-09-16T05:36:28.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831583891 0 0) 0x7f53340a1560 con 0x7f5334139eb0 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): my global_id is 4710 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5334073550 con 0x7f5334139eb0 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2792254510 0 0) 0x7f5314003530 con 0x7f5334075600 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53340a1560 con 0x7f5334075600 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f531c0015e0 con 0x7f5334075600 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): dump: 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.555 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 968767372 0 0) 0x7f5334073550 con 0x7f5334139eb0 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5314003530 con 0x7f5334139eb0 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f532c0040e0 con 0x7f5334139eb0 2024-09-16T05:36:28.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient(hunting): dump: 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 11867928 0 0) 0x7f53340a1560 con 0x7f5334075600 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 >> v1:172.21.15.63:6789/0 conn(0x7f5334072c00 legacy=0x7f5334132920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 >> v1:172.21.15.6:6789/0 conn(0x7f5334139eb0 legacy=0x7f53341d3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 monclient: found mon.c 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5334133030 con 0x7f5334075600 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient: _finish_auth 0 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 5 monclient: authenticate success, global_id 4709 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: _renew_subs 2024-09-16T05:36:28.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5334120de0 con 0x7f5334075600 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: _renew_subs 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5334138a30 con 0x7f5334075600 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f531c001880 con 0x7f5334075600 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f531c004c40 con 0x7f5334075600 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f5330d99640 4 set_mon_vals no callback set 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.540+0000 7f532affd640 10 monclient: dump: 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.559 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.544+0000 7f53388b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.544+0000 7f53388b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.544+0000 7f53388b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.544+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5318005090 con 0x7f5334075600 2024-09-16T05:36:28.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.544+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f531c00f9e0 con 0x7f5334075600 2024-09-16T05:36:28.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.544+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7f531c044110 con 0x7f5334075600 2024-09-16T05:36:28.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.544+0000 7f533259c640 10 monclient: get_auth_request con 0x7f5314036a70 auth_method 0 2024-09-16T05:36:28.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.556+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f531c015380 con 0x7f5334075600 2024-09-16T05:36:28.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.556+0000 7f532affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.556+0000 7f532affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:28.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.724+0000 7f53388b9640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-16T05:36:28.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.724+0000 7f53388b9640 10 monclient: _send_command 2 [{"prefix": "osd crush rule dump"}] 2024-09-16T05:36:28.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.724+0000 7f53388b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.724+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule dump"} v 0) -- 0x7f5318005380 con 0x7f5334075600 2024-09-16T05:36:28.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.724+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v43) ==== 67+0+1058 (unknown 4026406719 0 3068570633) 0x7f531c018fd0 con 0x7f5334075600 2024-09-16T05:36:28.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.724+0000 7f532affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule dump"}] 2024-09-16T05:36:28.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.724+0000 7f532affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:28.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.824+0000 7f53388b9640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-16T05:36:28.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.824+0000 7f53388b9640 10 monclient: _send_command 3 [{"prefix": "osd crush rule dump"}] 2024-09-16T05:36:28.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.824+0000 7f53388b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.824+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule dump"} v 0) -- 0x7f53180008e0 con 0x7f5334075600 2024-09-16T05:36:28.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.824+0000 7f532affd640 1 -- 172.21.15.63:0/2697078988 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v43) ==== 67+0+1058 (unknown 4026406719 0 3068570633) 0x7f531c01d890 con 0x7f5334075600 2024-09-16T05:36:28.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.824+0000 7f532affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule dump"}] 2024-09-16T05:36:28.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.824+0000 7f532affd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout:[ 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: "rule_id": 0, 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: "rule_name": "replicated_rule", 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: "type": 1, 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: "steps": [ 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "take", 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: "item": -1, 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: "item_name": "default" 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.827 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "choose_firstn", 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: "num": 0, 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: "type": "osd" 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "emit" 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: "rule_id": 1, 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: "rule_name": "erasure-code", 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: "type": 3, 2024-09-16T05:36:28.828 INFO:tasks.workunit.client.0.smithi063.stdout: "steps": [ 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "set_chooseleaf_tries", 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: "num": 5 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "set_choose_tries", 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: "num": 100 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "take", 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: "item": -1, 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: "item_name": "default" 2024-09-16T05:36:28.829 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "choose_indep", 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: "num": 0, 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: "type": "osd" 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: "op": "emit" 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout:] 2024-09-16T05:36:28.830 INFO:tasks.workunit.client.0.smithi063.stdout: 2024-09-16T05:36:28.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.828+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5314036a70 legacy=0x7f5314038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.828+0000 7f53388b9640 10 monclient: shutdown 2024-09-16T05:36:28.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.828+0000 7f53388b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:28.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.828+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 >> v1:172.21.15.6:6790/0 conn(0x7f5334075600 legacy=0x7f53341d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.828+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 shutdown_connections 2024-09-16T05:36:28.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.828+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 >> 172.21.15.63:0/2697078988 conn(0x7f53340a2b80 msgr2=0x7f53340a1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:28.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.828+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 shutdown_connections 2024-09-16T05:36:28.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.828+0000 7f53388b9640 1 -- 172.21.15.63:0/2697078988 wait complete. 2024-09-16T05:36:28.843 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls 2024-09-16T05:36:28.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient: get_monmap_and_config 2024-09-16T05:36:28.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient: build_initial_monmap 2024-09-16T05:36:28.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient: monmap: 2024-09-16T05:36:28.970 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:28.970 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.970 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:28.969651+0000 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:28.969651+0000 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 1 Processor -- start 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 1 -- start start 2024-09-16T05:36:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient: init 2024-09-16T05:36:28.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:28.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:28.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient(hunting): picked mon.noname-b con 0x7fcf340a3d80 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:28.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient(hunting): picked mon.noname-a con 0x7fcf340a4870 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:28.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3413d4a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:28.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf340a0880 con 0x7fcf340a4870 2024-09-16T05:36:28.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf340a5340 con 0x7fcf3413d4a0 2024-09-16T05:36:28.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf340a10b0 con 0x7fcf340a3d80 2024-09-16T05:36:28.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:28.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf390c7640 10 monclient(hunting): authenticate will time out at 1045.746536s 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf3359e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcf3413d4a0 legacy=0x7fcf3413f890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf3259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcf340a4870 0x7fcf3413bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43256/0 (socket says 172.21.15.63:43256) 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf32d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcf340a3d80 0x7fcf340a4160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53352/0 (socket says 172.21.15.63:53352) 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf3259c640 1 -- 172.21.15.63:0/2099576001 learned_addr learned my addr 172.21.15.63:0/2099576001 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1642703910 0 0) 0x7fcf340a10b0 con 0x7fcf340a3d80 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 10 monclient(hunting): my global_id is 4712 2024-09-16T05:36:28.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf1c003530 con 0x7fcf340a3d80 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208797530 0 0) 0x7fcf340a0880 con 0x7fcf340a4870 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 10 monclient(hunting): my global_id is 4713 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.968+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf340a10b0 con 0x7fcf340a4870 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 957513829 0 0) 0x7fcf1c003530 con 0x7fcf340a3d80 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 >> v1:172.21.15.63:6789/0 conn(0x7fcf3413d4a0 legacy=0x7fcf3413f890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 >> v1:172.21.15.6:6789/0 conn(0x7fcf340a4870 legacy=0x7fcf3413bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 1 monclient: found mon.noname-b 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:28.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf340a1390 con 0x7fcf340a3d80 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: _finish_auth 0 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf24001620 con 0x7fcf340a3d80 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: dump: 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.980 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 5 monclient: authenticate success, global_id 4712 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf240018d0 con 0x7fcf340a3d80 2024-09-16T05:36:28.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 1 -- 172.21.15.63:0/2099576001 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf24001ca0 con 0x7fcf340a3d80 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf31d9b640 10 monclient: dump: 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.982 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 4 set_mon_vals no callback set 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient: shutdown 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- 172.21.15.63:0/2099576001 >> v1:172.21.15.6:6790/0 conn(0x7fcf340a3d80 legacy=0x7fcf340a4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- 172.21.15.63:0/2099576001 shutdown_connections 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- 172.21.15.63:0/2099576001 >> 172.21.15.63:0/2099576001 conn(0x7fcf34075470 msgr2=0x7fcf34075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- 172.21.15.63:0/2099576001 shutdown_connections 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- 172.21.15.63:0/2099576001 wait complete. 2024-09-16T05:36:28.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient: build_initial_monmap 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient: monmap: 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 Processor -- start 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- start start 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient: init 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient(hunting): picked mon.noname-c con 0x7fcf340a3d80 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient(hunting): picked mon.noname-a con 0x7fcf340a4870 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3413d4a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf3408d380 con 0x7fcf340a4870 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf34131730 con 0x7fcf3413d4a0 2024-09-16T05:36:28.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf3408d660 con 0x7fcf340a3d80 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf3359e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcf3413d4a0 legacy=0x7fcf341d7fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf390c7640 10 monclient(hunting): authenticate will time out at 1045.750753s 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf32d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcf340a3d80 0x7fcf341cf8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53356/0 (socket says 172.21.15.63:53356) 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf32d9d640 1 -- 172.21.15.63:0/948647559 learned_addr learned my addr 172.21.15.63:0/948647559 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537399885 0 0) 0x7fcf3408d660 con 0x7fcf340a3d80 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 10 monclient(hunting): my global_id is 4715 2024-09-16T05:36:28.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf20003530 con 0x7fcf340a3d80 2024-09-16T05:36:28.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2855045306 0 0) 0x7fcf3408d380 con 0x7fcf340a4870 2024-09-16T05:36:28.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:28.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:28.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:28.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 10 monclient(hunting): my global_id is 4716 2024-09-16T05:36:28.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.972+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf3408d660 con 0x7fcf340a4870 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2705279708 0 0) 0x7fcf20003530 con 0x7fcf340a3d80 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf3408d380 con 0x7fcf340a3d80 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2248255345 0 0) 0x7fcf3408d660 con 0x7fcf340a4870 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf20003530 con 0x7fcf340a4870 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf24001c60 con 0x7fcf340a3d80 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.990 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf28001840 con 0x7fcf340a4870 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:28.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4195638101 0 0) 0x7fcf3408d380 con 0x7fcf340a3d80 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 >> v1:172.21.15.63:6789/0 conn(0x7fcf3413d4a0 legacy=0x7fcf341d7fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 >> v1:172.21.15.6:6789/0 conn(0x7fcf340a4870 legacy=0x7fcf341d48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 monclient: found mon.c 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3408d0a0 con 0x7fcf340a3d80 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 5 monclient: authenticate success, global_id 4715 2024-09-16T05:36:28.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 10 monclient: _renew_subs 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf34070ee0 con 0x7fcf340a3d80 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 10 monclient: _renew_subs 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf341cffd0 con 0x7fcf340a3d80 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf240011c0 con 0x7fcf340a3d80 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf24004c90 con 0x7fcf340a3d80 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf3159a640 4 set_mon_vals no callback set 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 10 monclient: dump: 2024-09-16T05:36:28.994 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:28.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf14005090 con 0x7fcf340a3d80 2024-09-16T05:36:28.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcf24004eb0 con 0x7fcf340a3d80 2024-09-16T05:36:28.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7fcf240442a0 con 0x7fcf340a3d80 2024-09-16T05:36:28.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.976+0000 7fcf3259c640 10 monclient: get_auth_request con 0x7fcf200369c0 auth_method 0 2024-09-16T05:36:28.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.988+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fcf240155d0 con 0x7fcf340a3d80 2024-09-16T05:36:28.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.988+0000 7fcf1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:28.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:28.988+0000 7fcf1b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:29.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.156+0000 7fcf390c7640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:29.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.156+0000 7fcf390c7640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:29.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.156+0000 7fcf390c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:29.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.156+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fcf14005380 con 0x7fcf340a3d80 2024-09-16T05:36:29.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.156+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v43) ==== 65+0+29 (unknown 1716862825 0 852635902) 0x7fcf2401da20 con 0x7fcf340a3d80 2024-09-16T05:36:29.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.156+0000 7fcf1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:29.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.156+0000 7fcf1b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:29.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.256+0000 7fcf390c7640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:29.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.256+0000 7fcf390c7640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:29.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.256+0000 7fcf390c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:29.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.256+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fcf14000bd0 con 0x7fcf340a3d80 2024-09-16T05:36:29.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf1b7fe640 1 -- 172.21.15.63:0/948647559 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v43) ==== 65+0+29 (unknown 1716862825 0 852635902) 0x7fcf240132a0 con 0x7fcf340a3d80 2024-09-16T05:36:29.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf1b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:29.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf1b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:29.261 INFO:tasks.workunit.client.0.smithi063.stdout:replicated_rule 2024-09-16T05:36:29.261 INFO:tasks.workunit.client.0.smithi063.stdout:erasure-code 2024-09-16T05:36:29.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 >> v1:172.21.15.63:6812/3643207211 conn(0x7fcf200369c0 legacy=0x7fcf20038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf390c7640 10 monclient: shutdown 2024-09-16T05:36:29.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf390c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:29.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 >> v1:172.21.15.6:6790/0 conn(0x7fcf340a3d80 legacy=0x7fcf341cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 shutdown_connections 2024-09-16T05:36:29.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 >> 172.21.15.63:0/948647559 conn(0x7fcf34075470 msgr2=0x7fcf34133fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:29.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 shutdown_connections 2024-09-16T05:36:29.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.260+0000 7fcf390c7640 1 -- 172.21.15.63:0/948647559 wait complete. 2024-09-16T05:36:29.277 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule list 2024-09-16T05:36:29.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient: get_monmap_and_config 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient: build_initial_monmap 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient: monmap: 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:29.397860+0000 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:29.397860+0000 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:29.399 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 1 Processor -- start 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 1 -- start start 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient: init 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient(hunting): picked mon.noname-a con 0x7f6e940a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient(hunting): picked mon.noname-c con 0x7f6e940a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient(hunting): picked mon.noname-b con 0x7f6e940a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e940a4fd0 con 0x7f6e940a3ca0 2024-09-16T05:36:29.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e940a5b40 con 0x7f6e940a08a0 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e940a1a70 con 0x7f6e940a4790 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9afdf640 10 monclient(hunting): authenticate will time out at 1046.174892s 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e93fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6e940a08a0 legacy=0x7f6e9413beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9957c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6e940a4790 0x7f6e9413f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53360/0 (socket says 172.21.15.63:53360) 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e98d7b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6e940a3ca0 0x7f6e940a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43280/0 (socket says 172.21.15.63:43280) 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e9957c640 1 -- 172.21.15.63:0/3725488202 learned_addr learned my addr 172.21.15.63:0/3725488202 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2366571391 0 0) 0x7f6e940a1a70 con 0x7f6e940a4790 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:29.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 10 monclient(hunting): my global_id is 4718 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e7c003530 con 0x7f6e940a4790 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4145211914 0 0) 0x7f6e940a4fd0 con 0x7f6e940a3ca0 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 10 monclient(hunting): my global_id is 4719 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.396+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e940a1a70 con 0x7f6e940a3ca0 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 392184056 0 0) 0x7f6e7c003530 con 0x7f6e940a4790 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 >> v1:172.21.15.63:6789/0 conn(0x7f6e940a08a0 legacy=0x7f6e9413beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 >> v1:172.21.15.6:6789/0 conn(0x7f6e940a3ca0 legacy=0x7f6e940a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:29.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 1 monclient: found mon.noname-b 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e940a53e0 con 0x7f6e940a4790 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e8c0019a0 con 0x7f6e940a4790 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: dump: 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.406 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 5 monclient: authenticate success, global_id 4718 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e8c001c00 con 0x7f6e940a4790 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 1 -- 172.21.15.63:0/3725488202 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e8c001f50 con 0x7f6e940a4790 2024-09-16T05:36:29.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 4 set_mon_vals no callback set 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e937fe640 10 monclient: dump: 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.408 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient: shutdown 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- 172.21.15.63:0/3725488202 >> v1:172.21.15.6:6790/0 conn(0x7f6e940a4790 legacy=0x7f6e9413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- 172.21.15.63:0/3725488202 shutdown_connections 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- 172.21.15.63:0/3725488202 >> 172.21.15.63:0/3725488202 conn(0x7f6e940a67c0 msgr2=0x7f6e94075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- 172.21.15.63:0/3725488202 shutdown_connections 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- 172.21.15.63:0/3725488202 wait complete. 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient: build_initial_monmap 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient: monmap: 2024-09-16T05:36:29.409 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 Processor -- start 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- start start 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient: init 2024-09-16T05:36:29.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient(hunting): picked mon.noname-c con 0x7f6e940a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient(hunting): picked mon.noname-a con 0x7f6e940a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient(hunting): picked mon.noname-b con 0x7f6e940a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e940a84b0 con 0x7f6e940a3ca0 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e940a0500 con 0x7f6e940a4790 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e94123430 con 0x7f6e940a08a0 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9afdf640 10 monclient(hunting): authenticate will time out at 1046.178412s 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e9957c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6e940a4790 legacy=0x7f6e941d3b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e93fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6e940a3ca0 0x7f6e941d0450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43284/0 (socket says 172.21.15.63:43284) 2024-09-16T05:36:29.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e93fff640 1 -- 172.21.15.63:0/2016893159 learned_addr learned my addr 172.21.15.63:0/2016893159 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:29.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 712124900 0 0) 0x7f6e940a84b0 con 0x7f6e940a3ca0 2024-09-16T05:36:29.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:29.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:29.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:29.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): my global_id is 4722 2024-09-16T05:36:29.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e80003530 con 0x7f6e940a3ca0 2024-09-16T05:36:29.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009971443 0 0) 0x7f6e94123430 con 0x7f6e940a08a0 2024-09-16T05:36:29.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): my global_id is 4721 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e940a84b0 con 0x7f6e940a08a0 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2778520745 0 0) 0x7f6e80003530 con 0x7f6e940a3ca0 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e94123430 con 0x7f6e940a3ca0 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3653045320 0 0) 0x7f6e940a84b0 con 0x7f6e940a08a0 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e80003530 con 0x7f6e940a08a0 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e88004090 con 0x7f6e940a3ca0 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:29.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): dump: 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e84001750 con 0x7f6e940a08a0 2024-09-16T05:36:29.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.400+0000 7f6e917fa640 10 monclient(hunting): dump: 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.417 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3274398205 0 0) 0x7f6e94123430 con 0x7f6e940a3ca0 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 >> v1:172.21.15.6:6790/0 conn(0x7f6e940a08a0 legacy=0x7f6e94135d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 >> v1:172.21.15.63:6789/0 conn(0x7f6e940a4790 legacy=0x7f6e941d3b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 10 monclient: _finish_hunting 0 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 monclient: found mon.a 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e94123710 con 0x7f6e940a3ca0 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 10 monclient: _finish_auth 0 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 10 monclient: _check_auth_tickets 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 5 monclient: authenticate success, global_id 4722 2024-09-16T05:36:29.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 10 monclient: _renew_subs 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e94121d00 con 0x7f6e940a3ca0 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 10 monclient: _renew_subs 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e94136460 con 0x7f6e940a3ca0 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e880042c0 con 0x7f6e940a3ca0 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6e88004690 con 0x7f6e940a3ca0 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e92ffd640 4 set_mon_vals no callback set 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 10 monclient: dump: 2024-09-16T05:36:29.419 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:29.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e74005090 con 0x7f6e940a3ca0 2024-09-16T05:36:29.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6e88004830 con 0x7f6e940a3ca0 2024-09-16T05:36:29.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7f6e88043fd0 con 0x7f6e940a3ca0 2024-09-16T05:36:29.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.404+0000 7f6e98d7b640 10 monclient: get_auth_request con 0x7f6e80036a10 auth_method 0 2024-09-16T05:36:29.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.416+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6e88018e90 con 0x7f6e940a3ca0 2024-09-16T05:36:29.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.416+0000 7f6e917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:29.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.416+0000 7f6e917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:29.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.588+0000 7f6e9afdf640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-16T05:36:29.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.588+0000 7f6e9afdf640 10 monclient: _send_command 2 [{"prefix": "osd crush rule list"}] 2024-09-16T05:36:29.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.588+0000 7f6e9afdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:29.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.588+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule list"} v 0) -- 0x7f6e74005380 con 0x7f6e940a3ca0 2024-09-16T05:36:29.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.588+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v43) ==== 67+0+29 (unknown 1615052641 0 852635902) 0x7f6e8801d750 con 0x7f6e940a3ca0 2024-09-16T05:36:29.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.588+0000 7f6e917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule list"}] 2024-09-16T05:36:29.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.588+0000 7f6e917fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:29.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.684+0000 7f6e9afdf640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-16T05:36:29.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.684+0000 7f6e9afdf640 10 monclient: _send_command 3 [{"prefix": "osd crush rule list"}] 2024-09-16T05:36:29.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.684+0000 7f6e9afdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:29.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.684+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule list"} v 0) -- 0x7f6e74005ac0 con 0x7f6e940a3ca0 2024-09-16T05:36:29.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e917fa640 1 -- 172.21.15.63:0/2016893159 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v43) ==== 67+0+29 (unknown 1615052641 0 852635902) 0x7f6e8803fe90 con 0x7f6e940a3ca0 2024-09-16T05:36:29.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e917fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule list"}] 2024-09-16T05:36:29.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e917fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:29.689 INFO:tasks.workunit.client.0.smithi063.stdout:replicated_rule 2024-09-16T05:36:29.689 INFO:tasks.workunit.client.0.smithi063.stdout:erasure-code 2024-09-16T05:36:29.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6e80036a10 legacy=0x7f6e80038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e9afdf640 10 monclient: shutdown 2024-09-16T05:36:29.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e9afdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:29.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 >> v1:172.21.15.6:6789/0 conn(0x7f6e940a3ca0 legacy=0x7f6e941d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 shutdown_connections 2024-09-16T05:36:29.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 >> 172.21.15.63:0/2016893159 conn(0x7f6e940a67c0 msgr2=0x7f6e94136e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:29.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 shutdown_connections 2024-09-16T05:36:29.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.688+0000 7f6e9afdf640 1 -- 172.21.15.63:0/2016893159 wait complete. 2024-09-16T05:36:29.705 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-16T05:36:29.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.824+0000 7f7f55386640 10 monclient: get_monmap_and_config 2024-09-16T05:36:29.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.824+0000 7f7f55386640 10 monclient: build_initial_monmap 2024-09-16T05:36:29.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.824+0000 7f7f55386640 10 monclient: monmap: 2024-09-16T05:36:29.828 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:29.828 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.828 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:29.828209+0000 2024-09-16T05:36:29.828 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:29.828209+0000 2024-09-16T05:36:29.829 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:29.829 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.829 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:29.829 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:29.829 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:29.829 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 Processor -- start 2024-09-16T05:36:29.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- start start 2024-09-16T05:36:29.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient: init 2024-09-16T05:36:29.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:29.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:29.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient(hunting): picked mon.noname-a con 0x7f7f50134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:29.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient(hunting): picked mon.noname-b con 0x7f7f500a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:29.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient(hunting): picked mon.noname-c con 0x7f7f5006dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:29.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f50135420 con 0x7f7f50134930 2024-09-16T05:36:29.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5009e830 con 0x7f7f5006dfa0 2024-09-16T05:36:29.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f500a0dc0 con 0x7f7f500a0810 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient(hunting): authenticate will time out at 1046.605429s 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7f5006dfa0 legacy=0x7f7f5013cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7f50134930 0x7f7f50134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43292/0 (socket says 172.21.15.63:43292) 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4effd640 1 -- 172.21.15.63:0/119604635 learned_addr learned my addr 172.21.15.63:0/119604635 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652292205 0 0) 0x7f7f50135420 con 0x7f7f50134930 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:29.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient(hunting): my global_id is 4725 2024-09-16T05:36:29.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f38003530 con 0x7f7f50134930 2024-09-16T05:36:29.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2422743412 0 0) 0x7f7f500a0dc0 con 0x7f7f500a0810 2024-09-16T05:36:29.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:29.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:29.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:29.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient(hunting): my global_id is 4724 2024-09-16T05:36:29.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f50135420 con 0x7f7f500a0810 2024-09-16T05:36:29.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2014804230 0 0) 0x7f7f38003530 con 0x7f7f50134930 2024-09-16T05:36:29.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 >> v1:172.21.15.6:6790/0 conn(0x7f7f500a0810 legacy=0x7f7f50139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 >> v1:172.21.15.63:6789/0 conn(0x7f7f5006dfa0 legacy=0x7f7f5013cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:29.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:29.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 monclient: found mon.noname-a 2024-09-16T05:36:29.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:29.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:29.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f500a2500 con 0x7f7f50134930 2024-09-16T05:36:29.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:29.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:29.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:29.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7f40001620 con 0x7f7f50134930 2024-09-16T05:36:29.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:29.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:29.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: dump: 2024-09-16T05:36:29.840 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.840 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.840 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.840 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.841 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.841 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.841 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.841 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.841 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.841 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 5 monclient: authenticate success, global_id 4725 2024-09-16T05:36:29.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:29.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f400018c0 con 0x7f7f50134930 2024-09-16T05:36:29.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:29.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:29.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 4 set_mon_vals no callback set 2024-09-16T05:36:29.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 1 -- 172.21.15.63:0/119604635 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7f40001c90 con 0x7f7f50134930 2024-09-16T05:36:29.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:29.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:29.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f4dffb640 10 monclient: dump: 2024-09-16T05:36:29.843 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.843 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.843 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.844 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.844 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.844 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.844 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.844 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.844 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.845 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 10 monclient: shutdown 2024-09-16T05:36:29.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:29.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- 172.21.15.63:0/119604635 >> v1:172.21.15.6:6789/0 conn(0x7f7f50134930 legacy=0x7f7f50134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- 172.21.15.63:0/119604635 shutdown_connections 2024-09-16T05:36:29.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- 172.21.15.63:0/119604635 >> 172.21.15.63:0/119604635 conn(0x7f7f500a1e90 msgr2=0x7f7f50075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- 172.21.15.63:0/119604635 shutdown_connections 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.828+0000 7f7f55386640 1 -- 172.21.15.63:0/119604635 wait complete. 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: build_initial_monmap 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: monmap: 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:29.846 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 1 Processor -- start 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 1 -- start start 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: init 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient(hunting): picked mon.noname-a con 0x7f7f5006dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient(hunting): picked mon.noname-c con 0x7f7f500a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient(hunting): picked mon.noname-b con 0x7f7f50134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5009cf10 con 0x7f7f5006dfa0 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f500a45c0 con 0x7f7f50134930 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f5013e3c0 con 0x7f7f500a0810 2024-09-16T05:36:29.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:29.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient(hunting): authenticate will time out at 1046.609176s 2024-09-16T05:36:29.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f4f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7f50134930 legacy=0x7f7f501d3780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:29.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f4effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7f5006dfa0 0x7f7f5009c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43304/0 (socket says 172.21.15.63:43304) 2024-09-16T05:36:29.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f4effd640 1 -- 172.21.15.63:0/400977799 learned_addr learned my addr 172.21.15.63:0/400977799 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:29.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685048033 0 0) 0x7f7f5009cf10 con 0x7f7f5006dfa0 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): my global_id is 4728 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f3c003530 con 0x7f7f5006dfa0 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1351676384 0 0) 0x7f7f5013e3c0 con 0x7f7f500a0810 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): my global_id is 4727 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f5009cf10 con 0x7f7f500a0810 2024-09-16T05:36:29.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 13794669 0 0) 0x7f7f5009cf10 con 0x7f7f500a0810 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f5013e3c0 con 0x7f7f500a0810 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7f44001a80 con 0x7f7f500a0810 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): dump: 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.852 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3607353367 0 0) 0x7f7f3c003530 con 0x7f7f5006dfa0 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f5009cf10 con 0x7f7f5006dfa0 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7f40001c10 con 0x7f7f5006dfa0 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:29.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient(hunting): dump: 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.854 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 489137814 0 0) 0x7f7f5013e3c0 con 0x7f7f500a0810 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 >> v1:172.21.15.63:6789/0 conn(0x7f7f50134930 legacy=0x7f7f501d3780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 >> v1:172.21.15.6:6789/0 conn(0x7f7f5006dfa0 legacy=0x7f7f5009c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 monclient: found mon.c 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f501398d0 con 0x7f7f500a0810 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:29.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 5 monclient: authenticate success, global_id 4727 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: _renew_subs 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7f50139bb0 con 0x7f7f500a0810 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: _renew_subs 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7f50133b10 con 0x7f7f500a0810 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f440011e0 con 0x7f7f500a0810 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7f44003a50 con 0x7f7f500a0810 2024-09-16T05:36:29.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f4d7fa640 4 set_mon_vals no callback set 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.832+0000 7f7f377fe640 10 monclient: dump: 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:29.857 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.836+0000 7f7f55386640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.836+0000 7f7f55386640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.836+0000 7f7f55386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.836+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f30005090 con 0x7f7f500a0810 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.836+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7f44003cb0 con 0x7f7f500a0810 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.836+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(43..43 src has 1..43) ==== 3598+0+0 (unknown 3479329539 0 0) 0x7f7f44042fb0 con 0x7f7f500a0810 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.836+0000 7f7f4effd640 10 monclient: get_auth_request con 0x7f7f3c036a70 auth_method 0 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.848+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7f440141c0 con 0x7f7f500a0810 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.848+0000 7f7f377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:29.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:29.848+0000 7f7f377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:30.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.016+0000 7f7f55386640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.016+0000 7f7f55386640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.016+0000 7f7f55386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:30.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.016+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) -- 0x7f7f30005380 con 0x7f7f500a0810 2024-09-16T05:36:30.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.212+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 v44) ==== 126+0+0 (unknown 79162031 0 0) 0x7f7f44017e70 con 0x7f7f500a0810 2024-09-16T05:36:30.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.212+0000 7f7f377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.212+0000 7f7f377fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:30.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.316+0000 7f7f55386640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.316+0000 7f7f55386640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.316+0000 7f7f55386640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:30.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.316+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) -- 0x7f7f30001150 con 0x7f7f500a0810 2024-09-16T05:36:30.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.316+0000 7f7f377fe640 1 -- 172.21.15.63:0/400977799 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v44) ==== 149+0+0 (unknown 2630659675 0 0) 0x7f7f4401c730 con 0x7f7f500a0810 2024-09-16T05:36:30.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f377fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f377fe640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-16T05:36:30.320 INFO:tasks.workunit.client.0.smithi063.stderr:rule foo already exists 2024-09-16T05:36:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7f3c036a70 legacy=0x7f7f3c038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f55386640 10 monclient: shutdown 2024-09-16T05:36:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f55386640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 >> v1:172.21.15.6:6790/0 conn(0x7f7f500a0810 legacy=0x7f7f5009c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 shutdown_connections 2024-09-16T05:36:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 >> 172.21.15.63:0/400977799 conn(0x7f7f500a1e90 msgr2=0x7f7f50071230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 shutdown_connections 2024-09-16T05:36:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.320+0000 7f7f55386640 1 -- 172.21.15.63:0/400977799 wait complete. 2024-09-16T05:36:30.338 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-16T05:36:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.460+0000 7f676749b640 10 monclient: get_monmap_and_config 2024-09-16T05:36:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.460+0000 7f676749b640 10 monclient: build_initial_monmap 2024-09-16T05:36:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.460+0000 7f676749b640 10 monclient: monmap: 2024-09-16T05:36:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:30.463213+0000 2024-09-16T05:36:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:30.463213+0000 2024-09-16T05:36:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:30.464 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 Processor -- start 2024-09-16T05:36:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- start start 2024-09-16T05:36:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient: init 2024-09-16T05:36:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient(hunting): picked mon.noname-c con 0x7f67600a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient(hunting): picked mon.noname-a con 0x7f67600a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient(hunting): picked mon.noname-b con 0x7f67600a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67600a53b0 con 0x7f67600a07a0 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67600a0290 con 0x7f67600a3b10 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67600a0dc0 con 0x7f67600a4600 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient(hunting): authenticate will time out at 1047.240809s 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6765237640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f67600a3b10 legacy=0x7f67600a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6765a38640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f67600a4600 0x7f676013f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53378/0 (socket says 172.21.15.63:53378) 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6765a38640 1 -- 172.21.15.63:0/25748711 learned_addr learned my addr 172.21.15.63:0/25748711 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6764a36640 1 --1- 172.21.15.63:0/25748711 >> v1:172.21.15.6:6789/0 conn(0x7f67600a07a0 0x7f676013be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43316/0 (socket says 172.21.15.63:43316) 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2713691826 0 0) 0x7f67600a0dc0 con 0x7f67600a4600 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient(hunting): my global_id is 4730 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f674c003530 con 0x7f67600a4600 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005522433 0 0) 0x7f67600a53b0 con 0x7f67600a07a0 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient(hunting): my global_id is 4731 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67600a0dc0 con 0x7f67600a07a0 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3988056060 0 0) 0x7f67600a0dc0 con 0x7f67600a07a0 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 >> v1:172.21.15.6:6790/0 conn(0x7f67600a4600 legacy=0x7f676013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 >> v1:172.21.15.63:6789/0 conn(0x7f67600a3b10 legacy=0x7f67600a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 monclient: found mon.noname-a 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67600a10a0 con 0x7f67600a07a0 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: _finish_auth 0 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6750001600 con 0x7f67600a07a0 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: dump: 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 5 monclient: authenticate success, global_id 4731 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f67500018f0 con 0x7f67600a07a0 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 4 set_mon_vals no callback set 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 1 -- 172.21.15.63:0/25748711 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6750001cc0 con 0x7f67600a07a0 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f6757fff640 10 monclient: dump: 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient: shutdown 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- 172.21.15.63:0/25748711 >> v1:172.21.15.6:6789/0 conn(0x7f67600a07a0 legacy=0x7f676013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- 172.21.15.63:0/25748711 shutdown_connections 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- 172.21.15.63:0/25748711 >> 172.21.15.63:0/25748711 conn(0x7f6760075470 msgr2=0x7f6760075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- 172.21.15.63:0/25748711 shutdown_connections 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 1 -- 172.21.15.63:0/25748711 wait complete. 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient: build_initial_monmap 2024-09-16T05:36:30.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.464+0000 7f676749b640 10 monclient: monmap: 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:30.476 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:30.477 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 1 Processor -- start 2024-09-16T05:36:30.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 1 -- start start 2024-09-16T05:36:30.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: init 2024-09-16T05:36:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient(hunting): picked mon.noname-a con 0x7f67600a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient(hunting): picked mon.noname-c con 0x7f67600a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient(hunting): picked mon.noname-b con 0x7f67600a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67601243d0 con 0x7f67600a07a0 2024-09-16T05:36:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67601246b0 con 0x7f67600a4600 2024-09-16T05:36:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67600a81b0 con 0x7f67600a3b10 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient(hunting): authenticate will time out at 1047.244601s 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6765a38640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f67600a4600 legacy=0x7f67601d3320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6765237640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f67600a07a0 0x7f67601cafe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43320/0 (socket says 172.21.15.63:43320) 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6765237640 1 -- 172.21.15.63:0/2348546314 learned_addr learned my addr 172.21.15.63:0/2348546314 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505921936 0 0) 0x7f67600a81b0 con 0x7f67600a3b10 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): my global_id is 4733 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6744003530 con 0x7f67600a3b10 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703946348 0 0) 0x7f67601243d0 con 0x7f67600a07a0 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): my global_id is 4734 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67600a81b0 con 0x7f67600a07a0 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2607146544 0 0) 0x7f6744003530 con 0x7f67600a3b10 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67601243d0 con 0x7f67600a3b10 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1728017872 0 0) 0x7f67600a81b0 con 0x7f67600a07a0 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6744003530 con 0x7f67600a07a0 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f67500015c0 con 0x7f67600a3b10 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): dump: 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f675c0040e0 con 0x7f67600a07a0 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient(hunting): dump: 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4103050643 0 0) 0x7f6744003530 con 0x7f67600a07a0 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 >> v1:172.21.15.6:6790/0 conn(0x7f67600a3b10 legacy=0x7f6760136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 >> v1:172.21.15.63:6789/0 conn(0x7f67600a4600 legacy=0x7f67601d3320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 monclient: found mon.a 2024-09-16T05:36:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67600a8700 con 0x7f67600a07a0 2024-09-16T05:36:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 5 monclient: authenticate success, global_id 4734 2024-09-16T05:36:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: _renew_subs 2024-09-16T05:36:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:30.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67600a0500 con 0x7f67600a07a0 2024-09-16T05:36:30.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: _renew_subs 2024-09-16T05:36:30.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:30.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6760137400 con 0x7f67600a07a0 2024-09-16T05:36:30.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f675c0045f0 con 0x7f67600a07a0 2024-09-16T05:36:30.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:30.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f67577fe640 4 set_mon_vals no callback set 2024-09-16T05:36:30.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f675c004940 con 0x7f67600a07a0 2024-09-16T05:36:30.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:30.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:30.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f6755ffb640 10 monclient: dump: 2024-09-16T05:36:30.488 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.488 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:30.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:30.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:30.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.468+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6740005090 con 0x7f67600a07a0 2024-09-16T05:36:30.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.472+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f675c004ae0 con 0x7f67600a07a0 2024-09-16T05:36:30.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.472+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(44..44 src has 1..44) ==== 3657+0+0 (unknown 121779843 0 0) 0x7f675c044050 con 0x7f67600a07a0 2024-09-16T05:36:30.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.472+0000 7f6764a36640 10 monclient: get_auth_request con 0x7f6744036950 auth_method 0 2024-09-16T05:36:30.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.484+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f675c0152e0 con 0x7f67600a07a0 2024-09-16T05:36:30.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.484+0000 7f6755ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:30.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.484+0000 7f6755ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:30.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.648+0000 7f676749b640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.648+0000 7f676749b640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.648+0000 7f676749b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:30.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.648+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) -- 0x7f6740005380 con 0x7f67600a07a0 2024-09-16T05:36:30.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.652+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v44) ==== 149+0+0 (unknown 2630659675 0 0) 0x7f675c014be0 con 0x7f67600a07a0 2024-09-16T05:36:30.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.652+0000 7f6755ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.652+0000 7f6755ffb640 10 monclient: _finish_command 2 = system:0 rule foo already exists 2024-09-16T05:36:30.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.748+0000 7f676749b640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.748+0000 7f676749b640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.748+0000 7f676749b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:30.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.748+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) -- 0x7f6740001150 con 0x7f67600a07a0 2024-09-16T05:36:30.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.752+0000 7f6755ffb640 1 -- 172.21.15.63:0/2348546314 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v44) ==== 149+0+0 (unknown 2630659675 0 0) 0x7f675c018f30 con 0x7f67600a07a0 2024-09-16T05:36:30.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.752+0000 7f6755ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T05:36:30.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.752+0000 7f6755ffb640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-16T05:36:30.753 INFO:tasks.workunit.client.0.smithi063.stderr:rule foo already exists 2024-09-16T05:36:30.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.756+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6744036950 legacy=0x7f6744038e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.756+0000 7f676749b640 10 monclient: shutdown 2024-09-16T05:36:30.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.756+0000 7f676749b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:30.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.756+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 >> v1:172.21.15.6:6789/0 conn(0x7f67600a07a0 legacy=0x7f67601cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.756+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 shutdown_connections 2024-09-16T05:36:30.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.756+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 >> 172.21.15.63:0/2348546314 conn(0x7f6760075470 msgr2=0x7f67601beb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:30.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.756+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 shutdown_connections 2024-09-16T05:36:30.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.756+0000 7f676749b640 1 -- 172.21.15.63:0/2348546314 wait complete. 2024-09-16T05:36:30.772 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule create-simple bar default host 2024-09-16T05:36:30.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.896+0000 7f3810787640 10 monclient: get_monmap_and_config 2024-09-16T05:36:30.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.896+0000 7f3810787640 10 monclient: build_initial_monmap 2024-09-16T05:36:30.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.896+0000 7f3810787640 10 monclient: monmap: 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:30.898636+0000 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:30.898636+0000 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:30.899 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.896+0000 7f3810787640 1 Processor -- start 2024-09-16T05:36:30.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.896+0000 7f3810787640 1 -- start start 2024-09-16T05:36:30.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.896+0000 7f3810787640 10 monclient: init 2024-09-16T05:36:30.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.896+0000 7f3810787640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:30.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:30.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): picked mon.noname-b con 0x7f380809c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:30.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): picked mon.noname-a con 0x7f38080a09c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): picked mon.noname-c con 0x7f380809d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38080a0ed0 con 0x7f38080a09c0 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380809d950 con 0x7f380809d3a0 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380809ebd0 con 0x7f380809c300 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): authenticate will time out at 1047.676133s 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380ed24640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f380809d3a0 legacy=0x7f380813c150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380dd22640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f38080a09c0 0x7f3808138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43328/0 (socket says 172.21.15.63:43328) 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380dd22640 1 -- 172.21.15.63:0/1687232668 learned_addr learned my addr 172.21.15.63:0/1687232668 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380e523640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f380809c300 0x7f380809c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:53396/0 (socket says 172.21.15.63:53396) 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036697524 0 0) 0x7f380809ebd0 con 0x7f380809c300 2024-09-16T05:36:30.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient(hunting): _negotiate 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient(hunting): my global_id is 4736 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37f4003530 con 0x7f380809c300 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626908058 0 0) 0x7f38080a0ed0 con 0x7f38080a09c0 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient(hunting): _negotiate 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient(hunting): my global_id is 4737 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f380809ebd0 con 0x7f38080a09c0 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1101542727 0 0) 0x7f37f4003530 con 0x7f380809c300 2024-09-16T05:36:30.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 >> v1:172.21.15.63:6789/0 conn(0x7f380809d3a0 legacy=0x7f380813c150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 >> v1:172.21.15.6:6789/0 conn(0x7f38080a09c0 legacy=0x7f3808138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: _finish_hunting 0 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 monclient: found mon.noname-b 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38080a2470 con 0x7f380809c300 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: _finish_auth 0 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: _check_auth_tickets 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 5 monclient: authenticate success, global_id 4736 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:30.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3804001620 con 0x7f380809c300 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: dump: 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.907 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3804001940 con 0x7f380809c300 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 1 -- 172.21.15.63:0/1687232668 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3804001d10 con 0x7f380809c300 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 4 set_mon_vals no callback set 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380d521640 10 monclient: dump: 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.908 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient: shutdown 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- 172.21.15.63:0/1687232668 >> v1:172.21.15.6:6790/0 conn(0x7f380809c300 legacy=0x7f380809c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- 172.21.15.63:0/1687232668 shutdown_connections 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- 172.21.15.63:0/1687232668 >> 172.21.15.63:0/1687232668 conn(0x7f38080a1e90 msgr2=0x7f3808075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:30.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- 172.21.15.63:0/1687232668 shutdown_connections 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- 172.21.15.63:0/1687232668 wait complete. 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient: build_initial_monmap 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient: monmap: 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:30.910 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 Processor -- start 2024-09-16T05:36:30.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- start start 2024-09-16T05:36:30.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient: init 2024-09-16T05:36:30.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:30.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:30.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): picked mon.noname-b con 0x7f380809c300 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:30.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): picked mon.noname-c con 0x7f380809d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): picked mon.noname-a con 0x7f3808134020 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3808122b90 con 0x7f3808134020 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380806e6d0 con 0x7f380809c300 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3808124730 con 0x7f380809d3a0 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f3810787640 10 monclient(hunting): authenticate will time out at 1047.679666s 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380e523640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f380809c300 legacy=0x7f380806d8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380ed24640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3808134020 0x7f38081d3b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43342/0 (socket says 172.21.15.63:43342) 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.900+0000 7f380ed24640 1 -- 172.21.15.63:0/4078621945 learned_addr learned my addr 172.21.15.63:0/4078621945 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354348042 0 0) 0x7f3808124730 con 0x7f380809d3a0 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:30.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): my global_id is 4739 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37ec003530 con 0x7f380809d3a0 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4025872029 0 0) 0x7f3808122b90 con 0x7f3808134020 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): my global_id is 4740 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3808124730 con 0x7f3808134020 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2762166506 0 0) 0x7f37ec003530 con 0x7f380809d3a0 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3808122b90 con 0x7f380809d3a0 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f37f80018c0 con 0x7f380809d3a0 2024-09-16T05:36:30.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): dump: 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1200171194 0 0) 0x7f3808124730 con 0x7f3808134020 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37ec003530 con 0x7f3808134020 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f38000043c0 con 0x7f3808134020 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient(hunting): dump: 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.918 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 70213670 0 0) 0x7f3808122b90 con 0x7f380809d3a0 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 >> v1:172.21.15.63:6789/0 conn(0x7f380809c300 legacy=0x7f380806d8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 >> v1:172.21.15.6:6789/0 conn(0x7f3808134020 legacy=0x7f38081d3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 monclient: found mon.c 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38080a4460 con 0x7f380809d3a0 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:30.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 5 monclient: authenticate success, global_id 4739 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 10 monclient: _renew_subs 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f380809c0f0 con 0x7f380809d3a0 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 10 monclient: _renew_subs 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3808124450 con 0x7f380809d3a0 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37f80047f0 con 0x7f380809d3a0 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f37f8004b40 con 0x7f380809d3a0 2024-09-16T05:36:30.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f380cd20640 4 set_mon_vals no callback set 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 10 monclient: dump: 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:30.921 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f0005090 con 0x7f380809d3a0 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f37f800f8e0 con 0x7f380809d3a0 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(44..44 src has 1..44) ==== 3657+0+0 (unknown 121779843 0 0) 0x7f37f8043f30 con 0x7f380809d3a0 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.904+0000 7f380e523640 10 monclient: get_auth_request con 0x7f37ec036ac0 auth_method 0 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.916+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f37f80151c0 con 0x7f380809d3a0 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.916+0000 7f37feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:30.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:30.916+0000 7f37feffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:31.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:31.080+0000 7f3810787640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T05:36:31.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:31.080+0000 7f3810787640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T05:36:31.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:31.080+0000 7f3810787640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:31.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:31.080+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) -- 0x7f37f0005380 con 0x7f380809d3a0 2024-09-16T05:36:31.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:31.992+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 v46) ==== 126+0+0 (unknown 3455399079 0 0) 0x7f37f8014ac0 con 0x7f380809d3a0 2024-09-16T05:36:31.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:31.992+0000 7f37feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T05:36:31.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:31.992+0000 7f37feffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:32.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.096+0000 7f3810787640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T05:36:32.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.096+0000 7f3810787640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T05:36:32.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.096+0000 7f3810787640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:32.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.096+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) -- 0x7f37f0001150 con 0x7f380809d3a0 2024-09-16T05:36:32.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.096+0000 7f37feffd640 1 -- 172.21.15.63:0/4078621945 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 rule bar already exists v46) ==== 149+0+0 (unknown 697226909 0 0) 0x7f37f803ffd0 con 0x7f380809d3a0 2024-09-16T05:36:32.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.096+0000 7f37feffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T05:36:32.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.096+0000 7f37feffd640 10 monclient: _finish_command 3 = system:0 rule bar already exists 2024-09-16T05:36:32.099 INFO:tasks.workunit.client.0.smithi063.stderr:rule bar already exists 2024-09-16T05:36:32.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.100+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 >> v1:172.21.15.63:6812/3643207211 conn(0x7f37ec036ac0 legacy=0x7f37ec038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:32.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.100+0000 7f3810787640 10 monclient: shutdown 2024-09-16T05:36:32.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.100+0000 7f3810787640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:32.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.100+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 >> v1:172.21.15.6:6790/0 conn(0x7f380809d3a0 legacy=0x7f380806dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:32.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.100+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 shutdown_connections 2024-09-16T05:36:32.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.100+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 >> 172.21.15.63:0/4078621945 conn(0x7f38080a1e90 msgr2=0x7f38081b9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:32.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.100+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 shutdown_connections 2024-09-16T05:36:32.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.100+0000 7f3810787640 1 -- 172.21.15.63:0/4078621945 wait complete. 2024-09-16T05:36:32.116 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm-device-class all 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient: get_monmap_and_config 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient: build_initial_monmap 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient: monmap: 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:32.237904+0000 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:32.237904+0000 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:32.238 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:32.239 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:32.239 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:32.239 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:32.239 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:32.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 1 Processor -- start 2024-09-16T05:36:32.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 1 -- start start 2024-09-16T05:36:32.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient: init 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb806dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb80a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb809f800 con 0x7fdbb809ac90 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb80a0ab0 con 0x7fdbb80a0210 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb80a0d90 con 0x7fdbb806dd30 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbbde3e640 10 monclient(hunting): authenticate will time out at 1049.015426s 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbb7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fdbb80a0210 legacy=0x7fdbb813d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:32.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbb77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdbb809ac90 0x7fdbb809b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43346/0 (socket says 172.21.15.63:43346) 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.236+0000 7fdbb77fe640 1 -- 172.21.15.63:0/150713155 learned_addr learned my addr 172.21.15.63:0/150713155 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317164278 0 0) 0x7fdbb809f800 con 0x7fdbb809ac90 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient(hunting): my global_id is 4743 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba0003530 con 0x7fdbb809ac90 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190460304 0 0) 0x7fdbb80a0d90 con 0x7fdbb806dd30 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient(hunting): my global_id is 4742 2024-09-16T05:36:32.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb809f800 con 0x7fdbb806dd30 2024-09-16T05:36:32.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1936455843 0 0) 0x7fdba0003530 con 0x7fdbb809ac90 2024-09-16T05:36:32.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 >> v1:172.21.15.6:6790/0 conn(0x7fdbb806dd30 legacy=0x7fdbb81399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:32.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 >> v1:172.21.15.63:6789/0 conn(0x7fdbb80a0210 legacy=0x7fdbb813d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:32.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 monclient: found mon.noname-a 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb809e770 con 0x7fdbb809ac90 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdba8001620 con 0x7fdbb809ac90 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:32.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: dump: 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:32.249 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 5 monclient: authenticate success, global_id 4743 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdba8001880 con 0x7fdbb809ac90 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 1 -- 172.21.15.63:0/150713155 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdba8001c50 con 0x7fdbb809ac90 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb67fc640 10 monclient: dump: 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:32.250 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 4 set_mon_vals no callback set 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient: shutdown 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- 172.21.15.63:0/150713155 >> v1:172.21.15.6:6789/0 conn(0x7fdbb809ac90 legacy=0x7fdbb809b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:32.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- 172.21.15.63:0/150713155 shutdown_connections 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- 172.21.15.63:0/150713155 >> 172.21.15.63:0/150713155 conn(0x7fdbb80a1c90 msgr2=0x7fdbb80a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- 172.21.15.63:0/150713155 shutdown_connections 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- 172.21.15.63:0/150713155 wait complete. 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient: build_initial_monmap 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient: monmap: 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:32.252 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 Processor -- start 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- start start 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient: init 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb806dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb809ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb80a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb809c590 con 0x7fdbb80a0210 2024-09-16T05:36:32.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb8072190 con 0x7fdbb806dd30 2024-09-16T05:36:32.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb77fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fdbb806dd30 legacy=0x7fdbb8133ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:32.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb81215e0 con 0x7fdbb809ac90 2024-09-16T05:36:32.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:32.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbbde3e640 10 monclient(hunting): authenticate will time out at 1049.019288s 2024-09-16T05:36:32.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdbb80a0210 0x7fdbb81d3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43350/0 (socket says 172.21.15.63:43350) 2024-09-16T05:36:32.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.240+0000 7fdbb7fff640 1 -- 172.21.15.63:0/3192775108 learned_addr learned my addr 172.21.15.63:0/3192775108 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:32.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4031857308 0 0) 0x7fdbb809c590 con 0x7fdbb80a0210 2024-09-16T05:36:32.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): my global_id is 4746 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba4003530 con 0x7fdbb80a0210 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 649249641 0 0) 0x7fdbb81215e0 con 0x7fdbb809ac90 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): my global_id is 4745 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb809c590 con 0x7fdbb809ac90 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3802586098 0 0) 0x7fdba4003530 con 0x7fdbb80a0210 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbb81215e0 con 0x7fdbb80a0210 2024-09-16T05:36:32.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdbb00043c0 con 0x7fdbb80a0210 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): dump: 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:32.256 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2414810503 0 0) 0x7fdbb809c590 con 0x7fdbb809ac90 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdba4003530 con 0x7fdbb809ac90 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdbac0016b0 con 0x7fdbb809ac90 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient(hunting): dump: 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.257 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2949030365 0 0) 0x7fdbb81215e0 con 0x7fdbb80a0210 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 >> v1:172.21.15.6:6790/0 conn(0x7fdbb809ac90 legacy=0x7fdbb81d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 >> v1:172.21.15.63:6789/0 conn(0x7fdbb806dd30 legacy=0x7fdbb8133ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 monclient: found mon.a 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:32.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb81218c0 con 0x7fdbb80a0210 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient: _finish_auth 0 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 5 monclient: authenticate success, global_id 4746 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 10 monclient: _renew_subs 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb809c0f0 con 0x7fdbb80a0210 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 10 monclient: _renew_subs 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbb81347c0 con 0x7fdbb80a0210 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbb0004620 con 0x7fdbb80a0210 2024-09-16T05:36:32.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdbb0004970 con 0x7fdbb80a0210 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbb57fa640 4 set_mon_vals no callback set 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 10 monclient: dump: 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:32.260 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb98005090 con 0x7fdbb80a0210 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdbb0004b90 con 0x7fdbb80a0210 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(46..46 src has 1..46) ==== 3716+0+0 (unknown 1201266658 0 0) 0x7fdbb00443d0 con 0x7fdbb80a0210 2024-09-16T05:36:32.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.244+0000 7fdbb77fe640 10 monclient: get_auth_request con 0x7fdba4036a20 auth_method 0 2024-09-16T05:36:32.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.256+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fdbb0015560 con 0x7fdbb80a0210 2024-09-16T05:36:32.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.256+0000 7fdb9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:32.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.256+0000 7fdb9ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:32.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.424+0000 7fdbbde3e640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T05:36:32.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.424+0000 7fdbbde3e640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T05:36:32.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.424+0000 7fdbbde3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:32.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:32.424+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) -- 0x7fdb98005380 con 0x7fdbb80a0210 2024-09-16T05:36:33.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:33.220+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 done removing class of osd(s): 0,1,2,3,4,5 v47) ==== 131+0+0 (unknown 3758139335 0 0) 0x7fdbb0019210 con 0x7fdbb80a0210 2024-09-16T05:36:33.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:33.220+0000 7fdb9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T05:36:33.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:33.220+0000 7fdb9ffff640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0,1,2,3,4,5 2024-09-16T05:36:33.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:33.320+0000 7fdbbde3e640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T05:36:33.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:33.320+0000 7fdbbde3e640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T05:36:33.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:33.320+0000 7fdbbde3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:33.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:33.320+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) -- 0x7fdb980010a0 con 0x7fdbb80a0210 2024-09-16T05:36:34.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.224+0000 7fdb9ffff640 1 -- 172.21.15.63:0/3192775108 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): v48) ==== 282+0+0 (unknown 2634339036 0 0) 0x7fdbb001dad0 con 0x7fdbb80a0210 2024-09-16T05:36:34.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.224+0000 7fdb9ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T05:36:34.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.224+0000 7fdb9ffff640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-16T05:36:34.228 INFO:tasks.workunit.client.0.smithi063.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-16T05:36:34.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.228+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 >> v1:172.21.15.63:6812/3643207211 conn(0x7fdba4036a20 legacy=0x7fdba4038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:34.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.228+0000 7fdbbde3e640 10 monclient: shutdown 2024-09-16T05:36:34.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.228+0000 7fdbbde3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:34.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.228+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 >> v1:172.21.15.6:6789/0 conn(0x7fdbb80a0210 legacy=0x7fdbb81d3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:34.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.228+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 shutdown_connections 2024-09-16T05:36:34.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.228+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 >> 172.21.15.63:0/3192775108 conn(0x7fdbb80a1c90 msgr2=0x7fdbb809d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:34.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.228+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 shutdown_connections 2024-09-16T05:36:34.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.228+0000 7fdbbde3e640 1 -- 172.21.15.63:0/3192775108 wait complete. 2024-09-16T05:36:34.251 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush set-device-class ssd osd.0 2024-09-16T05:36:34.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.376+0000 7f5a86a0b640 10 monclient: get_monmap_and_config 2024-09-16T05:36:34.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.376+0000 7f5a86a0b640 10 monclient: build_initial_monmap 2024-09-16T05:36:34.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.376+0000 7f5a86a0b640 10 monclient: monmap: 2024-09-16T05:36:34.378 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:34.378468+0000 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:34.378468+0000 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:34.379 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.376+0000 7f5a86a0b640 1 Processor -- start 2024-09-16T05:36:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.376+0000 7f5a86a0b640 1 -- start start 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.376+0000 7f5a86a0b640 10 monclient: init 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a80136960 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a800a1c40 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a80137450 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a800a09c0 con 0x7f5a800a1c40 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a800a0f80 con 0x7f5a80137450 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a800a4bb0 con 0x7f5a80136960 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient(hunting): authenticate will time out at 1051.156227s 2024-09-16T05:36:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a84fa8640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5a80137450 legacy=0x7f5a8013f4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a77fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5a800a1c40 0x7f5a8013bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55562/0 (socket says 172.21.15.63:55562) 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a77fff640 1 -- 172.21.15.63:0/3806297442 learned_addr learned my addr 172.21.15.63:0/3806297442 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7ffff640 1 --1- 172.21.15.63:0/3806297442 >> v1:172.21.15.6:6790/0 conn(0x7f5a80136960 0x7f5a80136d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35796/0 (socket says 172.21.15.63:35796) 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1303379070 0 0) 0x7f5a800a09c0 con 0x7f5a800a1c40 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient(hunting): my global_id is 4749 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a6c003530 con 0x7f5a800a1c40 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648077116 0 0) 0x7f5a800a4bb0 con 0x7f5a80136960 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient(hunting): my global_id is 4748 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a800a09c0 con 0x7f5a80136960 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3360108922 0 0) 0x7f5a6c003530 con 0x7f5a800a1c40 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 >> v1:172.21.15.6:6790/0 conn(0x7f5a80136960 legacy=0x7f5a80136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 >> v1:172.21.15.63:6789/0 conn(0x7f5a80137450 legacy=0x7f5a8013f4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 monclient: found mon.noname-a 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:34.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a800a4e60 con 0x7f5a800a1c40 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a70001620 con 0x7f5a800a1c40 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: dump: 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.387 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.388 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:34.388 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:34.388 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:34.388 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:34.388 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:34.388 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 5 monclient: authenticate success, global_id 4749 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a700018d0 con 0x7f5a800a1c40 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 1 -- 172.21.15.63:0/3806297442 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a70001ce0 con 0x7f5a800a1c40 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a7f7fe640 10 monclient: dump: 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 4 set_mon_vals no callback set 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient: shutdown 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- 172.21.15.63:0/3806297442 >> v1:172.21.15.6:6789/0 conn(0x7f5a800a1c40 legacy=0x7f5a8013bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- 172.21.15.63:0/3806297442 shutdown_connections 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- 172.21.15.63:0/3806297442 >> 172.21.15.63:0/3806297442 conn(0x7f5a800a60a0 msgr2=0x7f5a800a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- 172.21.15.63:0/3806297442 shutdown_connections 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- 172.21.15.63:0/3806297442 wait complete. 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient: build_initial_monmap 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient: monmap: 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:34.393 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 Processor -- start 2024-09-16T05:36:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 1 -- start start 2024-09-16T05:36:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.380+0000 7f5a86a0b640 10 monclient: init 2024-09-16T05:36:34.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:34.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a800a1c40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a80136960 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a80137450 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5a800a1c40 legacy=0x7f5a80135dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a800a0500 con 0x7f5a80137450 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a800a7500 con 0x7f5a800a1c40 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a801241d0 con 0x7f5a80136960 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient(hunting): authenticate will time out at 1051.160293s 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a77fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5a80136960 0x7f5a801d0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35802/0 (socket says 172.21.15.63:35802) 2024-09-16T05:36:34.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a77fff640 1 -- 172.21.15.63:0/1548082857 learned_addr learned my addr 172.21.15.63:0/1548082857 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3310216313 0 0) 0x7f5a78003d40 con 0x7f5a80137450 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): my global_id is 4752 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a60003530 con 0x7f5a80137450 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2398649418 0 0) 0x7f5a801241d0 con 0x7f5a80136960 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:34.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): my global_id is 4751 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a78003d40 con 0x7f5a80136960 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1468328636 0 0) 0x7f5a60003530 con 0x7f5a80137450 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a801241d0 con 0x7f5a80137450 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a78004510 con 0x7f5a80137450 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): dump: 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:34.402 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3241267126 0 0) 0x7f5a78003d40 con 0x7f5a80136960 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a60003530 con 0x7f5a80136960 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a70001ce0 con 0x7f5a80136960 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:34.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient(hunting): dump: 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:34.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 334827596 0 0) 0x7f5a801241d0 con 0x7f5a80137450 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 >> v1:172.21.15.6:6790/0 conn(0x7f5a80136960 legacy=0x7f5a801d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 >> v1:172.21.15.63:6789/0 conn(0x7f5a800a1c40 legacy=0x7f5a80135dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 monclient: found mon.a 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a801244b0 con 0x7f5a80137450 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 5 monclient: authenticate success, global_id 4752 2024-09-16T05:36:34.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient: _renew_subs 2024-09-16T05:36:34.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:34.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a80122d80 con 0x7f5a80137450 2024-09-16T05:36:34.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient: _renew_subs 2024-09-16T05:36:34.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:34.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a801366a0 con 0x7f5a80137450 2024-09-16T05:36:34.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a78004770 con 0x7f5a80137450 2024-09-16T05:36:34.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:34.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a78004ac0 con 0x7f5a80137450 2024-09-16T05:36:34.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:34.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7effd640 4 set_mon_vals no callback set 2024-09-16T05:36:34.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:34.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 10 monclient: dump: 2024-09-16T05:36:34.407 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:34.407 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:34.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a64005090 con 0x7f5a80137450 2024-09-16T05:36:34.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5a78004ce0 con 0x7f5a80137450 2024-09-16T05:36:34.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(48..48 src has 1..48) ==== 3412+0+0 (unknown 3669235891 0 0) 0x7f5a780443e0 con 0x7f5a80137450 2024-09-16T05:36:34.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.384+0000 7f5a7ffff640 10 monclient: get_auth_request con 0x7f5a60036a20 auth_method 0 2024-09-16T05:36:34.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.396+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5a800a0500 con 0x7f5a80137450 2024-09-16T05:36:34.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.396+0000 7f5a7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:34.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.396+0000 7f5a7d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:34.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.564+0000 7f5a86a0b640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T05:36:34.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.564+0000 7f5a86a0b640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T05:36:34.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.564+0000 7f5a86a0b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:34.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:34.564+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) -- 0x7f5a64005380 con 0x7f5a80137450 2024-09-16T05:36:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:35.228+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'ssd' v49) ==== 135+0+0 (unknown 1316855264 0 0) 0x7f5a78019350 con 0x7f5a80137450 2024-09-16T05:36:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:35.228+0000 7f5a7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T05:36:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:35.228+0000 7f5a7d7fa640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'ssd' 2024-09-16T05:36:35.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:35.328+0000 7f5a86a0b640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T05:36:35.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:35.328+0000 7f5a86a0b640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T05:36:35.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:35.328+0000 7f5a86a0b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:35.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:35.328+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) -- 0x7f5a64005ac0 con 0x7f5a80137450 2024-09-16T05:36:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.240+0000 7f5a7d7fa640 1 -- 172.21.15.63:0/1548082857 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' v50) ==== 237+0+0 (unknown 775395074 0 0) 0x7f5a7801dc10 con 0x7f5a80137450 2024-09-16T05:36:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.240+0000 7f5a7d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T05:36:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.240+0000 7f5a7d7fa640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-16T05:36:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-16T05:36:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.244+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5a60036a20 legacy=0x7f5a60038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.244+0000 7f5a86a0b640 10 monclient: shutdown 2024-09-16T05:36:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.244+0000 7f5a86a0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.244+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 >> v1:172.21.15.6:6789/0 conn(0x7f5a80137450 legacy=0x7f5a801d3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.244+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 shutdown_connections 2024-09-16T05:36:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.244+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 >> 172.21.15.63:0/1548082857 conn(0x7f5a800a60a0 msgr2=0x7f5a800a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.244+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 shutdown_connections 2024-09-16T05:36:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.244+0000 7f5a86a0b640 1 -- 172.21.15.63:0/1548082857 wait complete. 2024-09-16T05:36:36.266 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush set-device-class hdd osd.1 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.388+0000 7fcaa848e640 10 monclient: get_monmap_and_config 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.388+0000 7fcaa848e640 10 monclient: build_initial_monmap 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.388+0000 7fcaa848e640 10 monclient: monmap: 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:36.391855+0000 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:36.391855+0000 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:36.392 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:36.393 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:36.393 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:36.393 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 1 Processor -- start 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 1 -- start start 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient: init 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient(hunting): picked mon.noname-a con 0x7fcaa0075770 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient(hunting): picked mon.noname-c con 0x7fcaa0098e40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient(hunting): picked mon.noname-b con 0x7fcaa009dd40 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa009cac0 con 0x7fcaa0075770 2024-09-16T05:36:36.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa009cda0 con 0x7fcaa0098e40 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa009e190 con 0x7fcaa009dd40 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5a29640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcaa0098e40 legacy=0x7fcaa01398c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient(hunting): authenticate will time out at 1053.169735s 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa622a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcaa0075770 0x7fcaa0098730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55574/0 (socket says 172.21.15.63:55574) 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa6a2b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcaa009dd40 0x7fcaa013cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35814/0 (socket says 172.21.15.63:35814) 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa622a640 1 -- 172.21.15.63:0/2490557702 learned_addr learned my addr 172.21.15.63:0/2490557702 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2691908102 0 0) 0x7fcaa009cac0 con 0x7fcaa0075770 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient(hunting): _negotiate 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient(hunting): my global_id is 4755 2024-09-16T05:36:36.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca8c003530 con 0x7fcaa0075770 2024-09-16T05:36:36.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 306253218 0 0) 0x7fcaa009e190 con 0x7fcaa009dd40 2024-09-16T05:36:36.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient(hunting): _negotiate 2024-09-16T05:36:36.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:36.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:36.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient(hunting): my global_id is 4754 2024-09-16T05:36:36.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaa009cac0 con 0x7fcaa009dd40 2024-09-16T05:36:36.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 266499938 0 0) 0x7fca8c003530 con 0x7fcaa0075770 2024-09-16T05:36:36.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 >> v1:172.21.15.6:6790/0 conn(0x7fcaa009dd40 legacy=0x7fcaa013cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:36.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 >> v1:172.21.15.63:6789/0 conn(0x7fcaa0098e40 legacy=0x7fcaa01398c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:36.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: _finish_hunting 0 2024-09-16T05:36:36.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 monclient: found mon.noname-a 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa009f570 con 0x7fcaa0075770 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: _finish_auth 0 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: _check_auth_tickets 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fca980018d0 con 0x7fcaa0075770 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 5 monclient: authenticate success, global_id 4755 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: dump: 2024-09-16T05:36:36.403 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca98001b00 con 0x7fcaa0075770 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:36.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 1 -- 172.21.15.63:0/2490557702 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fca98001f10 con 0x7fcaa0075770 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa848e640 4 set_mon_vals no callback set 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.392+0000 7fcaa5228640 10 monclient: dump: 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:36.405 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: shutdown 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- 172.21.15.63:0/2490557702 >> v1:172.21.15.6:6789/0 conn(0x7fcaa0075770 legacy=0x7fcaa0098730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- 172.21.15.63:0/2490557702 shutdown_connections 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- 172.21.15.63:0/2490557702 >> 172.21.15.63:0/2490557702 conn(0x7fcaa00a2020 msgr2=0x7fcaa00a2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- 172.21.15.63:0/2490557702 shutdown_connections 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- 172.21.15.63:0/2490557702 wait complete. 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: build_initial_monmap 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: monmap: 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 Processor -- start 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- start start 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: init 2024-09-16T05:36:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient(hunting): picked mon.noname-a con 0x7fcaa0075770 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient(hunting): picked mon.noname-c con 0x7fcaa0098e40 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient(hunting): picked mon.noname-b con 0x7fcaa009dd40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa00999f0 con 0x7fcaa0075770 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa009c710 con 0x7fcaa009dd40 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa009b2b0 con 0x7fcaa0098e40 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient(hunting): authenticate will time out at 1053.173618s 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa6a2b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcaa009dd40 legacy=0x7fcaa0072b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa5a29640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcaa0098e40 0x7fcaa0072480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35816/0 (socket says 172.21.15.63:35816) 2024-09-16T05:36:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa622a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcaa0075770 0x7fcaa0071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55580/0 (socket says 172.21.15.63:55580) 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa5a29640 1 -- 172.21.15.63:0/3383992495 learned_addr learned my addr 172.21.15.63:0/3383992495 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887110536 0 0) 0x7fcaa009b2b0 con 0x7fcaa0098e40 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): my global_id is 4757 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca84003530 con 0x7fcaa0098e40 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2778994691 0 0) 0x7fcaa00999f0 con 0x7fcaa0075770 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): my global_id is 4758 2024-09-16T05:36:36.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaa009b2b0 con 0x7fcaa0075770 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 405482601 0 0) 0x7fcaa009b2b0 con 0x7fcaa0075770 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcaa00999f0 con 0x7fcaa0075770 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1501762146 0 0) 0x7fca84003530 con 0x7fcaa0098e40 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcaa009b2b0 con 0x7fcaa0098e40 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fca98001e50 con 0x7fcaa0075770 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fca9c004010 con 0x7fcaa0098e40 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3657863387 0 0) 0x7fcaa00999f0 con 0x7fcaa0075770 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 >> v1:172.21.15.6:6790/0 conn(0x7fcaa0098e40 legacy=0x7fcaa0072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 >> v1:172.21.15.63:6789/0 conn(0x7fcaa009dd40 legacy=0x7fcaa0072b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 monclient: found mon.a 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa0096560 con 0x7fcaa0075770 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fca92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 5 monclient: authenticate success, global_id 4758 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: _renew_subs 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa0099720 con 0x7fcaa0075770 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: _renew_subs 2024-09-16T05:36:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.396+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcaa0099e00 con 0x7fcaa0075770 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca98004e20 con 0x7fcaa0075770 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fca92ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fca98005050 con 0x7fcaa0075770 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fca92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fca92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fcaa4a27640 4 set_mon_vals no callback set 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fca92ffd640 10 monclient: dump: 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fcaa848e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fcaa848e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fcaa848e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca88005090 con 0x7fcaa0075770 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fca98005270 con 0x7fcaa0075770 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(50..50 src has 1..50) ==== 3636+0+0 (unknown 1917626763 0 0) 0x7fca98044720 con 0x7fcaa0075770 2024-09-16T05:36:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.400+0000 7fcaa5a29640 10 monclient: get_auth_request con 0x7fca84036aa0 auth_method 0 2024-09-16T05:36:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.412+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fca980159c0 con 0x7fcaa0075770 2024-09-16T05:36:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.412+0000 7fca92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.412+0000 7fca92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:36.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.580+0000 7fcaa848e640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T05:36:36.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.580+0000 7fcaa848e640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T05:36:36.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.580+0000 7fcaa848e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:36.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:36.580+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) -- 0x7fca88005380 con 0x7fcaa0075770 2024-09-16T05:36:37.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:37.248+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 set osd(s) 1 to class 'hdd' v51) ==== 135+0+0 (unknown 1791371351 0 0) 0x7fca980152c0 con 0x7fcaa0075770 2024-09-16T05:36:37.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:37.248+0000 7fca92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T05:36:37.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:37.248+0000 7fca92ffd640 10 monclient: _finish_command 2 = system:0 set osd(s) 1 to class 'hdd' 2024-09-16T05:36:37.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:37.352+0000 7fcaa848e640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T05:36:37.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:37.352+0000 7fcaa848e640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T05:36:37.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:37.352+0000 7fcaa848e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:37.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:37.352+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) -- 0x7fca880010a0 con 0x7fcaa0075770 2024-09-16T05:36:38.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.256+0000 7fca92ffd640 1 -- 172.21.15.63:0/3383992495 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' v52) ==== 237+0+0 (unknown 1537954525 0 0) 0x7fca9801de70 con 0x7fcaa0075770 2024-09-16T05:36:38.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.256+0000 7fca92ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T05:36:38.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.256+0000 7fca92ffd640 10 monclient: _finish_command 3 = system:0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-16T05:36:38.258 INFO:tasks.workunit.client.0.smithi063.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-16T05:36:38.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.260+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 >> v1:172.21.15.63:6812/3643207211 conn(0x7fca84036aa0 legacy=0x7fca84038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:38.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.260+0000 7fcaa848e640 10 monclient: shutdown 2024-09-16T05:36:38.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.260+0000 7fcaa848e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:38.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.260+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 >> v1:172.21.15.6:6789/0 conn(0x7fcaa0075770 legacy=0x7fcaa0071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:38.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.260+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 shutdown_connections 2024-09-16T05:36:38.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.260+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 >> 172.21.15.63:0/3383992495 conn(0x7fcaa00a2020 msgr2=0x7fcaa0134890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:38.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.260+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 shutdown_connections 2024-09-16T05:36:38.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.260+0000 7fcaa848e640 1 -- 172.21.15.63:0/3383992495 wait complete. 2024-09-16T05:36:38.279 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2024-09-16T05:36:38.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient: get_monmap_and_config 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient: build_initial_monmap 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient: monmap: 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:38.409828+0000 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:38.409828+0000 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:38.410 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:38.411 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:38.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 1 Processor -- start 2024-09-16T05:36:38.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 1 -- start start 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient: init 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient(hunting): picked mon.noname-a con 0x7f41f809ba70 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient(hunting): picked mon.noname-c con 0x7f41f80a08c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient(hunting): picked mon.noname-b con 0x7f41f80a0400 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f809f8a0 con 0x7f41f809ba70 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f8072f20 con 0x7f41f80a08c0 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f809eb50 con 0x7f41f80a0400 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41fe227640 10 monclient(hunting): authenticate will time out at 1055.186979s 2024-09-16T05:36:38.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f6ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f41f80a08c0 legacy=0x7f41f8139900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f41f809ba70 0x7f41f809fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55594/0 (socket says 172.21.15.63:55594) 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f77fe640 1 -- 172.21.15.63:0/570655526 learned_addr learned my addr 172.21.15.63:0/570655526 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f7fff640 1 --1- 172.21.15.63:0/570655526 >> v1:172.21.15.6:6790/0 conn(0x7f41f80a0400 0x7f41f813d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35826/0 (socket says 172.21.15.63:35826) 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3763966841 0 0) 0x7f41f809f8a0 con 0x7f41f809ba70 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 10 monclient(hunting): my global_id is 4761 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41e4003530 con 0x7f41f809ba70 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 502254291 0 0) 0x7f41f809eb50 con 0x7f41f80a0400 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:38.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 10 monclient(hunting): my global_id is 4760 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.408+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41f809f8a0 con 0x7f41f80a0400 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4227972001 0 0) 0x7f41e4003530 con 0x7f41f809ba70 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 >> v1:172.21.15.6:6790/0 conn(0x7f41f80a0400 legacy=0x7f41f813d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 >> v1:172.21.15.63:6789/0 conn(0x7f41f80a08c0 legacy=0x7f41f8139900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 1 monclient: found mon.noname-a 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f809e6c0 con 0x7f41f809ba70 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f41e8001620 con 0x7f41f809ba70 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: dump: 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 5 monclient: authenticate success, global_id 4761 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41e8001940 con 0x7f41f809ba70 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 1 -- 172.21.15.63:0/570655526 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f41e8001d10 con 0x7f41f809ba70 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:38.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f67fc640 10 monclient: dump: 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:38.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 4 set_mon_vals no callback set 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient: shutdown 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- 172.21.15.63:0/570655526 >> v1:172.21.15.6:6789/0 conn(0x7f41f809ba70 legacy=0x7f41f809fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- 172.21.15.63:0/570655526 shutdown_connections 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- 172.21.15.63:0/570655526 >> 172.21.15.63:0/570655526 conn(0x7f41f8075470 msgr2=0x7f41f8075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- 172.21.15.63:0/570655526 shutdown_connections 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- 172.21.15.63:0/570655526 wait complete. 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient: build_initial_monmap 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient: monmap: 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:38.425 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 Processor -- start 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- start start 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient: init 2024-09-16T05:36:38.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient(hunting): picked mon.noname-c con 0x7f41f809ba70 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient(hunting): picked mon.noname-a con 0x7f41f80a0400 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient(hunting): picked mon.noname-b con 0x7f41f80a08c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f8132920 con 0x7f41f80a0400 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f8134300 con 0x7f41f80a08c0 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f8135b50 con 0x7f41f809ba70 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41fe227640 10 monclient(hunting): authenticate will time out at 1055.190714s 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f41f80a08c0 legacy=0x7f41f81d17b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f6ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f41f80a0400 0x7f41f8132210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55604/0 (socket says 172.21.15.63:55604) 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41f6ffd640 1 -- 172.21.15.63:0/2259097462 learned_addr learned my addr 172.21.15.63:0/2259097462 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:38.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1038862289 0 0) 0x7f41f8132920 con 0x7f41f80a0400 2024-09-16T05:36:38.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:38.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:38.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:38.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): my global_id is 4764 2024-09-16T05:36:38.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41d8003530 con 0x7f41f80a0400 2024-09-16T05:36:38.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4219778348 0 0) 0x7f41f8135b50 con 0x7f41f809ba70 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): my global_id is 4763 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41f8132920 con 0x7f41f809ba70 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2261571167 0 0) 0x7f41d8003530 con 0x7f41f80a0400 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41f8135b50 con 0x7f41f80a0400 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f41e00044d0 con 0x7f41f80a0400 2024-09-16T05:36:38.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.412+0000 7f41effff640 10 monclient(hunting): dump: 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2626390751 0 0) 0x7f41f8132920 con 0x7f41f809ba70 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41d8003530 con 0x7f41f809ba70 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f41e8001440 con 0x7f41f809ba70 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient(hunting): dump: 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2712624227 0 0) 0x7f41f8135b50 con 0x7f41f80a0400 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 >> v1:172.21.15.6:6790/0 conn(0x7f41f809ba70 legacy=0x7f41f8131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 >> v1:172.21.15.63:6789/0 conn(0x7f41f80a08c0 legacy=0x7f41f81d17b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 monclient: found mon.a 2024-09-16T05:36:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f806e610 con 0x7f41f80a0400 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient: _finish_auth 0 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 5 monclient: authenticate success, global_id 4764 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 10 monclient: _renew_subs 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f41f8135870 con 0x7f41f80a0400 2024-09-16T05:36:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 10 monclient: _renew_subs 2024-09-16T05:36:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f41f8134080 con 0x7f41f80a0400 2024-09-16T05:36:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41e0004730 con 0x7f41f80a0400 2024-09-16T05:36:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f41e0004a80 con 0x7f41f80a0400 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41f5ffb640 4 set_mon_vals no callback set 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 10 monclient: dump: 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:38.437 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41dc005090 con 0x7f41f80a0400 2024-09-16T05:36:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f41e0004ca0 con 0x7f41f80a0400 2024-09-16T05:36:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(52..52 src has 1..52) ==== 3868+0+0 (unknown 3136894743 0 0) 0x7f41e0044590 con 0x7f41f80a0400 2024-09-16T05:36:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.416+0000 7f41f77fe640 10 monclient: get_auth_request con 0x7f41d8036ac0 auth_method 0 2024-09-16T05:36:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.428+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f41e00156f0 con 0x7f41f80a0400 2024-09-16T05:36:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.428+0000 7f41effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.428+0000 7f41effff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:38.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.600+0000 7f41fe227640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T05:36:38.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.600+0000 7f41fe227640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T05:36:38.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.600+0000 7f41fe227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:38.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:38.600+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) -- 0x7f41dc005380 con 0x7f41f80a0400 2024-09-16T05:36:39.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.260+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 v53) ==== 150+0+0 (unknown 1284886260 0 0) 0x7f41e0014ff0 con 0x7f41f80a0400 2024-09-16T05:36:39.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.260+0000 7f41effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T05:36:39.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.260+0000 7f41effff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:39.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.364+0000 7f41fe227640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T05:36:39.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.364+0000 7f41fe227640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T05:36:39.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.364+0000 7f41fe227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:39.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.364+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) -- 0x7f41dc005ac0 con 0x7f41f80a0400 2024-09-16T05:36:39.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.364+0000 7f41effff640 1 -- 172.21.15.63:0/2259097462 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 rule foo-ssd already exists v53) ==== 177+0+0 (unknown 1466744201 0 0) 0x7f41e0019340 con 0x7f41f80a0400 2024-09-16T05:36:39.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.364+0000 7f41effff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T05:36:39.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.364+0000 7f41effff640 10 monclient: _finish_command 3 = system:0 rule foo-ssd already exists 2024-09-16T05:36:39.367 INFO:tasks.workunit.client.0.smithi063.stderr:rule foo-ssd already exists 2024-09-16T05:36:39.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.368+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 >> v1:172.21.15.63:6812/3643207211 conn(0x7f41d8036ac0 legacy=0x7f41d8038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:39.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.368+0000 7f41fe227640 10 monclient: shutdown 2024-09-16T05:36:39.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.368+0000 7f41fe227640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:39.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.368+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 >> v1:172.21.15.6:6789/0 conn(0x7f41f80a0400 legacy=0x7f41f8132210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:39.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.368+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 shutdown_connections 2024-09-16T05:36:39.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.368+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 >> 172.21.15.63:0/2259097462 conn(0x7f41f8075470 msgr2=0x7f41f80a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:39.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.368+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 shutdown_connections 2024-09-16T05:36:39.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.368+0000 7f41fe227640 1 -- 172.21.15.63:0/2259097462 wait complete. 2024-09-16T05:36:39.384 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2024-09-16T05:36:39.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient: get_monmap_and_config 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient: build_initial_monmap 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient: monmap: 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:39.512655+0000 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:39.512655+0000 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:39.513 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:39.514 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:39.514 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:39.514 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:39.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 1 Processor -- start 2024-09-16T05:36:39.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 1 -- start start 2024-09-16T05:36:39.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient: init 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient(hunting): picked mon.noname-b con 0x7fea000a1820 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient(hunting): picked mon.noname-c con 0x7fea000a2310 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient(hunting): picked mon.noname-a con 0x7fea0009bc40 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0009b260 con 0x7fea0009bc40 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0009aa60 con 0x7fea000a2310 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0009c090 con 0x7fea000a1820 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient(hunting): authenticate will time out at 1056.289770s 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9ff7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fea000a2310 legacy=0x7fea0013bba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea04e06640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fea0009bc40 0x7fea0013f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55616/0 (socket says 172.21.15.63:55616) 2024-09-16T05:36:39.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea04e06640 1 -- 172.21.15.63:0/1501021231 learned_addr learned my addr 172.21.15.63:0/1501021231 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9fffff640 1 --1- 172.21.15.63:0/1501021231 >> v1:172.21.15.6:6790/0 conn(0x7fea000a1820 0x7fea000a1c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35840/0 (socket says 172.21.15.63:35840) 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 839477180 0 0) 0x7fea0009c090 con 0x7fea000a1820 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient(hunting): my global_id is 4766 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9e8003530 con 0x7fea000a1820 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1389066901 0 0) 0x7fea0009b260 con 0x7fea0009bc40 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:39.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient(hunting): my global_id is 4767 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea0009c090 con 0x7fea0009bc40 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3670847647 0 0) 0x7fe9e8003530 con 0x7fea000a1820 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 >> v1:172.21.15.63:6789/0 conn(0x7fea000a2310 legacy=0x7fea0013bba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 >> v1:172.21.15.6:6789/0 conn(0x7fea0009bc40 legacy=0x7fea0013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 monclient: found mon.noname-b 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea000731f0 con 0x7fea000a1820 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:39.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe9f0001550 con 0x7fea000a1820 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: dump: 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:39.520 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 5 monclient: authenticate success, global_id 4766 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9f00017b0 con 0x7fea000a1820 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fea06869640 4 set_mon_vals no callback set 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 1 -- 172.21.15.63:0/1501021231 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe9f0001b80 con 0x7fea000a1820 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.512+0000 7fe9feffd640 10 monclient: dump: 2024-09-16T05:36:39.521 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: shutdown 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- 172.21.15.63:0/1501021231 >> v1:172.21.15.6:6790/0 conn(0x7fea000a1820 legacy=0x7fea000a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:39.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- 172.21.15.63:0/1501021231 shutdown_connections 2024-09-16T05:36:39.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- 172.21.15.63:0/1501021231 >> 172.21.15.63:0/1501021231 conn(0x7fea00075470 msgr2=0x7fea00075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:39.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- 172.21.15.63:0/1501021231 shutdown_connections 2024-09-16T05:36:39.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- 172.21.15.63:0/1501021231 wait complete. 2024-09-16T05:36:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: build_initial_monmap 2024-09-16T05:36:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: monmap: 2024-09-16T05:36:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 Processor -- start 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- start start 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: init 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient(hunting): picked mon.noname-a con 0x7fea0009bc40 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient(hunting): picked mon.noname-c con 0x7fea000a1820 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient(hunting): picked mon.noname-b con 0x7fea000a2310 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0011d1b0 con 0x7fea0009bc40 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0011ebc0 con 0x7fea000a2310 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea0013fc90 con 0x7fea000a1820 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient(hunting): authenticate will time out at 1056.293899s 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea04e06640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fea000a2310 legacy=0x7fea001d3960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fea0009bc40 0x7fea0013e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55618/0 (socket says 172.21.15.63:55618) 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fffff640 1 -- 172.21.15.63:0/3349391283 learned_addr learned my addr 172.21.15.63:0/3349391283 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835819209 0 0) 0x7fea0011d1b0 con 0x7fea0009bc40 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:39.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): my global_id is 4770 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9ec003530 con 0x7fea0009bc40 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3454596483 0 0) 0x7fea0013fc90 con 0x7fea000a1820 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): my global_id is 4769 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea0011d1b0 con 0x7fea000a1820 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3973152590 0 0) 0x7fe9ec003530 con 0x7fea0009bc40 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea0013fc90 con 0x7fea0009bc40 2024-09-16T05:36:39.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe9f0001510 con 0x7fea0009bc40 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): dump: 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:39.532 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1351152992 0 0) 0x7fea0011d1b0 con 0x7fea000a1820 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9ec003530 con 0x7fea000a1820 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe9f40040d0 con 0x7fea000a1820 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient(hunting): dump: 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3572051451 0 0) 0x7fea0013fc90 con 0x7fea0009bc40 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 >> v1:172.21.15.6:6790/0 conn(0x7fea000a1820 legacy=0x7fea0013edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 >> v1:172.21.15.63:6789/0 conn(0x7fea000a2310 legacy=0x7fea001d3960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 monclient: found mon.a 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0009db30 con 0x7fea0009bc40 2024-09-16T05:36:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 5 monclient: authenticate success, global_id 4770 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: _renew_subs 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fea0006d150 con 0x7fea0009bc40 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: _renew_subs 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fea0011e8e0 con 0x7fea0009bc40 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9f00017b0 con 0x7fea0009bc40 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe9f0004d60 con 0x7fea0009bc40 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fe7fc640 4 set_mon_vals no callback set 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.516+0000 7fe9fcff9640 10 monclient: dump: 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:39.536 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.520+0000 7fea06869640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.520+0000 7fea06869640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.520+0000 7fea06869640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.520+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e0005090 con 0x7fea0009bc40 2024-09-16T05:36:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.520+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe9f0004f80 con 0x7fea0009bc40 2024-09-16T05:36:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.520+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(53..53 src has 1..53) ==== 3931+0+0 (unknown 667884027 0 0) 0x7fe9f00443b0 con 0x7fea0009bc40 2024-09-16T05:36:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.520+0000 7fe9ff7fe640 10 monclient: get_auth_request con 0x7fe9ec036a50 auth_method 0 2024-09-16T05:36:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.532+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe9f0015650 con 0x7fea0009bc40 2024-09-16T05:36:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.532+0000 7fe9fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.532+0000 7fe9fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:39.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.700+0000 7fea06869640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T05:36:39.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.700+0000 7fea06869640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T05:36:39.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.700+0000 7fea06869640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:39.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:39.700+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) -- 0x7fe9e0005380 con 0x7fea0009bc40 2024-09-16T05:36:40.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.272+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 v54) ==== 150+0+0 (unknown 2742814331 0 0) 0x7fe9f0014f50 con 0x7fea0009bc40 2024-09-16T05:36:40.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.272+0000 7fe9fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T05:36:40.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.272+0000 7fe9fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:40.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.376+0000 7fea06869640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T05:36:40.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.376+0000 7fea06869640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T05:36:40.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.376+0000 7fea06869640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:40.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.376+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) -- 0x7fe9e0005ac0 con 0x7fea0009bc40 2024-09-16T05:36:40.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.376+0000 7fe9fcff9640 1 -- 172.21.15.63:0/3349391283 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 rule foo-hdd already exists v54) ==== 177+0+0 (unknown 2890551366 0 0) 0x7fe9f0040340 con 0x7fea0009bc40 2024-09-16T05:36:40.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.376+0000 7fe9fcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T05:36:40.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.376+0000 7fe9fcff9640 10 monclient: _finish_command 3 = system:0 rule foo-hdd already exists 2024-09-16T05:36:40.379 INFO:tasks.workunit.client.0.smithi063.stderr:rule foo-hdd already exists 2024-09-16T05:36:40.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.380+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe9ec036a50 legacy=0x7fe9ec038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.380+0000 7fea06869640 10 monclient: shutdown 2024-09-16T05:36:40.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.380+0000 7fea06869640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:40.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.380+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 >> v1:172.21.15.6:6789/0 conn(0x7fea0009bc40 legacy=0x7fea0013e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.380+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 shutdown_connections 2024-09-16T05:36:40.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.380+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 >> 172.21.15.63:0/3349391283 conn(0x7fea00075470 msgr2=0x7fea0009dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:40.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.380+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 shutdown_connections 2024-09-16T05:36:40.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.380+0000 7fea06869640 1 -- 172.21.15.63:0/3349391283 wait complete. 2024-09-16T05:36:40.397 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-16T05:36:40.397 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep foo-ssd 2024-09-16T05:36:40.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.516+0000 7f97b97a2640 10 monclient: get_monmap_and_config 2024-09-16T05:36:40.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.516+0000 7f97b97a2640 10 monclient: build_initial_monmap 2024-09-16T05:36:40.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.516+0000 7f97b97a2640 10 monclient: monmap: 2024-09-16T05:36:40.520 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:40.520 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.520 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:40.520083+0000 2024-09-16T05:36:40.520 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:40.520083+0000 2024-09-16T05:36:40.520 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:40.521 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.521 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:40.521 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:40.521 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:40.521 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 Processor -- start 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- start start 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient: init 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient(hunting): picked mon.noname-a con 0x7f97b4136960 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient(hunting): picked mon.noname-c con 0x7f97b40a1c40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient(hunting): picked mon.noname-b con 0x7f97b4137450 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b40a09c0 con 0x7f97b4136960 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b40a0f80 con 0x7f97b40a1c40 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b40a4bb0 con 0x7f97b4137450 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97aa7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f97b40a1c40 legacy=0x7f97b413bdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:40.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient(hunting): authenticate will time out at 1057.297157s 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b2ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f97b4136960 0x7f97b4136d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55624/0 (socket says 172.21.15.63:55624) 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b37fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f97b4137450 0x7f97b413f4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35862/0 (socket says 172.21.15.63:35862) 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b2ffd640 1 -- 172.21.15.63:0/3667631277 learned_addr learned my addr 172.21.15.63:0/3667631277 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490028205 0 0) 0x7f97b40a4bb0 con 0x7f97b4137450 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient(hunting): my global_id is 4772 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97a0003530 con 0x7f97b4137450 2024-09-16T05:36:40.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896331037 0 0) 0x7f97b40a09c0 con 0x7f97b4136960 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient(hunting): my global_id is 4773 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97b40a4bb0 con 0x7f97b4136960 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1881972596 0 0) 0x7f97a0003530 con 0x7f97b4137450 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 >> v1:172.21.15.63:6789/0 conn(0x7f97b40a1c40 legacy=0x7f97b413bdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 >> v1:172.21.15.6:6789/0 conn(0x7f97b4136960 legacy=0x7f97b4136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 monclient: found mon.noname-b 2024-09-16T05:36:40.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b40a4e60 con 0x7f97b4137450 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f97ac0018d0 con 0x7f97b4137450 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: dump: 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.527 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 5 monclient: authenticate success, global_id 4772 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97ac001b80 con 0x7f97b4137450 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 1 -- 172.21.15.63:0/3667631277 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f97ac001f50 con 0x7f97b4137450 2024-09-16T05:36:40.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b27fc640 10 monclient: dump: 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.529 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 4 set_mon_vals no callback set 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 10 monclient: shutdown 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- 172.21.15.63:0/3667631277 >> v1:172.21.15.6:6790/0 conn(0x7f97b4137450 legacy=0x7f97b413f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- 172.21.15.63:0/3667631277 shutdown_connections 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- 172.21.15.63:0/3667631277 >> 172.21.15.63:0/3667631277 conn(0x7f97b40a60a0 msgr2=0x7f97b40a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- 172.21.15.63:0/3667631277 shutdown_connections 2024-09-16T05:36:40.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.520+0000 7f97b97a2640 1 -- 172.21.15.63:0/3667631277 wait complete. 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: build_initial_monmap 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: monmap: 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:40.535 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 1 Processor -- start 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 1 -- start start 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: init 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient(hunting): picked mon.noname-b con 0x7f97b40a1c40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient(hunting): picked mon.noname-a con 0x7f97b4136960 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient(hunting): picked mon.noname-c con 0x7f97b4137450 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b40a0500 con 0x7f97b4136960 2024-09-16T05:36:40.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b2ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f97b40a1c40 legacy=0x7f97b41cafb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b4124280 con 0x7f97b40a1c40 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b40a7620 con 0x7f97b4137450 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient(hunting): authenticate will time out at 1057.301104s 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97aa7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f97b4136960 0x7f97b4135e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55632/0 (socket says 172.21.15.63:55632) 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97aa7fc640 1 -- 172.21.15.63:0/2215383807 learned_addr learned my addr 172.21.15.63:0/2215383807 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1467126231 0 0) 0x7f979c003aa0 con 0x7f97b4136960 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): my global_id is 4776 2024-09-16T05:36:40.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9794003530 con 0x7f97b4136960 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250948107 0 0) 0x7f97b40a7620 con 0x7f97b4137450 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): my global_id is 4775 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f979c003aa0 con 0x7f97b4137450 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4280124159 0 0) 0x7f9794003530 con 0x7f97b4136960 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97b40a7620 con 0x7f97b4136960 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f979c004330 con 0x7f97b4136960 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): dump: 2024-09-16T05:36:40.538 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:40.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2400039263 0 0) 0x7f979c003aa0 con 0x7f97b4137450 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9794003530 con 0x7f97b4137450 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f97ac001890 con 0x7f97b4137450 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient(hunting): dump: 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.540 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2083368655 0 0) 0x7f9794003530 con 0x7f97b4137450 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 >> v1:172.21.15.63:6789/0 conn(0x7f97b40a1c40 legacy=0x7f97b41cafb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 >> v1:172.21.15.6:6789/0 conn(0x7f97b4136960 legacy=0x7f97b4135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 monclient: found mon.c 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b4124560 con 0x7f97b4137450 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient: _finish_auth 0 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:40.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 5 monclient: authenticate success, global_id 4775 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: _renew_subs 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97b4122e30 con 0x7f97b4137450 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: _renew_subs 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f97b4136730 con 0x7f97b4137450 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97ac001b80 con 0x7f97b4137450 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f97ac004fc0 con 0x7f97b4137450 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:40.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b1ffb640 4 set_mon_vals no callback set 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97abfff640 10 monclient: dump: 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.543 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.524+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9798005090 con 0x7f97b4137450 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.528+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f97ac0051e0 con 0x7f97b4137450 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.528+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(54..54 src has 1..54) ==== 3994+0+0 (unknown 1411905205 0 0) 0x7f97ac0444e0 con 0x7f97b4137450 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.528+0000 7f97b2ffd640 10 monclient: get_auth_request con 0x7f97940369d0 auth_method 0 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.540+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f97ac015760 con 0x7f97b4137450 2024-09-16T05:36:40.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.540+0000 7f97abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:40.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.540+0000 7f97abfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:40.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.708+0000 7f97b97a2640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:40.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.708+0000 7f97b97a2640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:40.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.708+0000 7f97b97a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.708+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) -- 0x7f9798005b90 con 0x7f97b4137450 2024-09-16T05:36:40.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.712+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v54) ==== 90+0+8 (unknown 2858363626 0 905510949) 0x7f97ac01dc10 con 0x7f97b4137450 2024-09-16T05:36:40.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.712+0000 7f97abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:40.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.712+0000 7f97abfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:40.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97b97a2640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:40.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97b97a2640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:40.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97b97a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) -- 0x7f9798005610 con 0x7f97b4137450 2024-09-16T05:36:40.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97abfff640 1 -- 172.21.15.63:0/2215383807 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v54) ==== 90+0+8 (unknown 2858363626 0 905510949) 0x7f97ac042270 con 0x7f97b4137450 2024-09-16T05:36:40.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97abfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97abfff640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 >> v1:172.21.15.63:6812/3643207211 conn(0x7f97940369d0 legacy=0x7f9794038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97b97a2640 10 monclient: shutdown 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97b97a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.816+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 >> v1:172.21.15.6:6790/0 conn(0x7f97b4137450 legacy=0x7f97b41d3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.820+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 shutdown_connections 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.820+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 >> 172.21.15.63:0/2215383807 conn(0x7f97b40a60a0 msgr2=0x7f97b40a79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.820+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 shutdown_connections 2024-09-16T05:36:40.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.820+0000 7f97b97a2640 1 -- 172.21.15.63:0/2215383807 wait complete. 2024-09-16T05:36:40.942 INFO:tasks.workunit.client.0.smithi063.stdout:foo-ssd 2024-09-16T05:36:40.942 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-16T05:36:40.942 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false grep foo-hdd 2024-09-16T05:36:40.942 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:40.942 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep foo-hdd 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.944+0000 7fe884a4f640 10 monclient: get_monmap_and_config 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.944+0000 7fe884a4f640 10 monclient: build_initial_monmap 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.944+0000 7fe884a4f640 10 monclient: monmap: 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:40.947142+0000 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:40.947142+0000 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.947 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:40.948 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:40.948 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:40.948 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.944+0000 7fe884a4f640 1 Processor -- start 2024-09-16T05:36:40.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.944+0000 7fe884a4f640 1 -- start start 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.944+0000 7fe884a4f640 10 monclient: init 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient(hunting): picked mon.noname-a con 0x7fe88009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient(hunting): picked mon.noname-c con 0x7fe88009b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient(hunting): picked mon.noname-b con 0x7fe880139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8800a14f0 con 0x7fe88009ac90 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe88009c680 con 0x7fe88009b780 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe88009e930 con 0x7fe880139fc0 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87dd9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe88009b780 legacy=0x7fe8801388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:40.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient(hunting): authenticate will time out at 1057.724525s 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87e59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe88009ac90 0x7fe88009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55636/0 (socket says 172.21.15.63:55636) 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87e59c640 1 -- 172.21.15.63:0/2555613164 learned_addr learned my addr 172.21.15.63:0/2555613164 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410998160 0 0) 0x7fe8800a14f0 con 0x7fe88009ac90 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient(hunting): my global_id is 4779 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe868003530 con 0x7fe88009ac90 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4223840505 0 0) 0x7fe88009e930 con 0x7fe880139fc0 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:40.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:40.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient(hunting): my global_id is 4778 2024-09-16T05:36:40.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8800a14f0 con 0x7fe880139fc0 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3391932247 0 0) 0x7fe868003530 con 0x7fe88009ac90 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 >> v1:172.21.15.6:6790/0 conn(0x7fe880139fc0 legacy=0x7fe88013c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 >> v1:172.21.15.63:6789/0 conn(0x7fe88009b780 legacy=0x7fe8801388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 monclient: found mon.noname-a 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88009c400 con 0x7fe88009ac90 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: _finish_auth 0 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:40.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe870001620 con 0x7fe88009ac90 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: dump: 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.958 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 5 monclient: authenticate success, global_id 4779 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe870001880 con 0x7fe88009ac90 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 1 -- 172.21.15.63:0/2555613164 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe870001c10 con 0x7fe88009ac90 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe87d59a640 10 monclient: dump: 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.959 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 4 set_mon_vals no callback set 2024-09-16T05:36:40.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient: shutdown 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- 172.21.15.63:0/2555613164 >> v1:172.21.15.6:6789/0 conn(0x7fe88009ac90 legacy=0x7fe88009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- 172.21.15.63:0/2555613164 shutdown_connections 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- 172.21.15.63:0/2555613164 >> 172.21.15.63:0/2555613164 conn(0x7fe8800a5290 msgr2=0x7fe8800a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- 172.21.15.63:0/2555613164 shutdown_connections 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- 172.21.15.63:0/2555613164 wait complete. 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient: build_initial_monmap 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient: monmap: 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:40.961 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 Processor -- start 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 1 -- start start 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.948+0000 7fe884a4f640 10 monclient: init 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient(hunting): picked mon.noname-c con 0x7fe88009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient(hunting): picked mon.noname-a con 0x7fe88009b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:40.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient(hunting): picked mon.noname-b con 0x7fe880139fc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe88011ca10 con 0x7fe88009b780 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe88011ccf0 con 0x7fe880139fc0 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8800a3c90 con 0x7fe88009ac90 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient(hunting): authenticate will time out at 1057.728444s 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe87ed9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe880139fc0 legacy=0x7fe8801d3cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe87e59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe88009ac90 0x7fe88012e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35892/0 (socket says 172.21.15.63:35892) 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe87e59c640 1 -- 172.21.15.63:0/3509063605 learned_addr learned my addr 172.21.15.63:0/3509063605 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050864142 0 0) 0x7fe8800a3c90 con 0x7fe88009ac90 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:40.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): my global_id is 4781 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe860003560 con 0x7fe88009ac90 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3536412831 0 0) 0x7fe88011ca10 con 0x7fe88009b780 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): my global_id is 4782 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8800a3c90 con 0x7fe88009b780 2024-09-16T05:36:40.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3788841020 0 0) 0x7fe860003560 con 0x7fe88009ac90 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe88011ca10 con 0x7fe88009ac90 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe870001520 con 0x7fe88009ac90 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): dump: 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.965 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 959846675 0 0) 0x7fe8800a3c90 con 0x7fe88009b780 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe860003560 con 0x7fe88009b780 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe874004050 con 0x7fe88009b780 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient(hunting): dump: 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.966 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2100317289 0 0) 0x7fe88011ca10 con 0x7fe88009ac90 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 >> v1:172.21.15.63:6789/0 conn(0x7fe880139fc0 legacy=0x7fe8801d3cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 >> v1:172.21.15.6:6789/0 conn(0x7fe88009b780 legacy=0x7fe8801d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 monclient: found mon.c 2024-09-16T05:36:40.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe880139b20 con 0x7fe88009ac90 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient: _finish_auth 0 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 5 monclient: authenticate success, global_id 4781 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient: _renew_subs 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe88011c730 con 0x7fe88009ac90 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient: _renew_subs 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe88006e890 con 0x7fe88009ac90 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe870001820 con 0x7fe88009ac90 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe870004c40 con 0x7fe88009ac90 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 10 monclient: dump: 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:40.969 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86ffff640 4 set_mon_vals no callback set 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe864005090 con 0x7fe88009ac90 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe87000f9e0 con 0x7fe88009ac90 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(54..54 src has 1..54) ==== 3994+0+0 (unknown 1411905205 0 0) 0x7fe870044170 con 0x7fe88009ac90 2024-09-16T05:36:40.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.952+0000 7fe87dd9b640 10 monclient: get_auth_request con 0x7fe860036af0 auth_method 0 2024-09-16T05:36:40.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.964+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe870015330 con 0x7fe88009ac90 2024-09-16T05:36:40.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.968+0000 7fe86effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:40.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:40.968+0000 7fe86effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:41.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.132+0000 7fe884a4f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:41.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.132+0000 7fe884a4f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:41.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.132+0000 7fe884a4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.132+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) -- 0x7fe864005b90 con 0x7fe88009ac90 2024-09-16T05:36:41.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.132+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v55) ==== 90+0+8 (unknown 832514816 0 905510949) 0x7fe870014c30 con 0x7fe88009ac90 2024-09-16T05:36:41.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.132+0000 7fe86effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:41.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.132+0000 7fe86effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:41.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.228+0000 7fe884a4f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:41.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.228+0000 7fe884a4f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:41.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.228+0000 7fe884a4f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.228+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) -- 0x7fe864005610 con 0x7fe88009ac90 2024-09-16T05:36:41.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.228+0000 7fe86effd640 1 -- 172.21.15.63:0/3509063605 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v55) ==== 90+0+8 (unknown 832514816 0 905510949) 0x7fe87001d7e0 con 0x7fe88009ac90 2024-09-16T05:36:41.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.228+0000 7fe86effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T05:36:41.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.228+0000 7fe86effd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:41.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.232+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe860036af0 legacy=0x7fe860038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.232+0000 7fe884a4f640 10 monclient: shutdown 2024-09-16T05:36:41.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.232+0000 7fe884a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:41.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.232+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 >> v1:172.21.15.6:6790/0 conn(0x7fe88009ac90 legacy=0x7fe88012e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.232+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 shutdown_connections 2024-09-16T05:36:41.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.232+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 >> 172.21.15.63:0/3509063605 conn(0x7fe8800a5290 msgr2=0x7fe88009f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:41.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.232+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 shutdown_connections 2024-09-16T05:36:41.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.232+0000 7fe884a4f640 1 -- 172.21.15.63:0/3509063605 wait complete. 2024-09-16T05:36:41.246 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:41.247 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-16T05:36:41.247 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep foo-hdd 2024-09-16T05:36:41.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.364+0000 7f681f72c640 10 monclient: get_monmap_and_config 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.364+0000 7f681f72c640 10 monclient: build_initial_monmap 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.364+0000 7f681f72c640 10 monclient: monmap: 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:41.367736+0000 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:41.367736+0000 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:41.368 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:41.369 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:41.369 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 Processor -- start 2024-09-16T05:36:41.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- start start 2024-09-16T05:36:41.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient: init 2024-09-16T05:36:41.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:41.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:41.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient(hunting): picked mon.noname-a con 0x7f6818136960 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:41.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient(hunting): picked mon.noname-b con 0x7f68180a1c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient(hunting): picked mon.noname-c con 0x7f6818137450 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68180a09c0 con 0x7f6818136960 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68180a0f80 con 0x7f6818137450 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68180a4bb0 con 0x7f68180a1c40 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient(hunting): authenticate will time out at 1058.145297s 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681dcc9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6818137450 legacy=0x7f681813f4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681ccc7640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f68180a1c40 0x7f681813bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35906/0 (socket says 172.21.15.63:35906) 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681d4c8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6818136960 0x7f6818136d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55642/0 (socket says 172.21.15.63:55642) 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681ccc7640 1 -- 172.21.15.63:0/93308794 learned_addr learned my addr 172.21.15.63:0/93308794 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:41.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806360881 0 0) 0x7f68180a09c0 con 0x7f6818136960 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient(hunting): my global_id is 4785 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6804003530 con 0x7f6818136960 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678338152 0 0) 0x7f68180a4bb0 con 0x7f68180a1c40 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient(hunting): my global_id is 4784 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68180a09c0 con 0x7f68180a1c40 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 221110119 0 0) 0x7f68180a09c0 con 0x7f68180a1c40 2024-09-16T05:36:41.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 >> v1:172.21.15.63:6789/0 conn(0x7f6818137450 legacy=0x7f681813f4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 >> v1:172.21.15.6:6789/0 conn(0x7f6818136960 legacy=0x7f6818136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 monclient: found mon.noname-b 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68180a4e60 con 0x7f68180a1c40 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: _finish_auth 0 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6808001600 con 0x7f68180a1c40 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 5 monclient: authenticate success, global_id 4784 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:41.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: dump: 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6808001860 con 0x7f68180a1c40 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 4 set_mon_vals no callback set 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 1 -- 172.21.15.63:0/93308794 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6808001c30 con 0x7f68180a1c40 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f680ffff640 10 monclient: dump: 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.377 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 10 monclient: shutdown 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- 172.21.15.63:0/93308794 >> v1:172.21.15.6:6790/0 conn(0x7f68180a1c40 legacy=0x7f681813bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- 172.21.15.63:0/93308794 shutdown_connections 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- 172.21.15.63:0/93308794 >> 172.21.15.63:0/93308794 conn(0x7f68180a60a0 msgr2=0x7f68180a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- 172.21.15.63:0/93308794 shutdown_connections 2024-09-16T05:36:41.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.368+0000 7f681f72c640 1 -- 172.21.15.63:0/93308794 wait complete. 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: build_initial_monmap 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: monmap: 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:41.382 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 1 Processor -- start 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 1 -- start start 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: init 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient(hunting): picked mon.noname-c con 0x7f68180a1c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient(hunting): picked mon.noname-b con 0x7f6818136960 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient(hunting): picked mon.noname-a con 0x7f6818137450 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6818124670 con 0x7f6818137450 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68180a7340 con 0x7f6818136960 2024-09-16T05:36:41.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68180a6fa0 con 0x7f68180a1c40 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient(hunting): authenticate will time out at 1058.149348s 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681ccc7640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6818136960 legacy=0x7f68181d00d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681d4c8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f68180a1c40 0x7f6818135f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35918/0 (socket says 172.21.15.63:35918) 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681d4c8640 1 -- 172.21.15.63:0/1761726985 learned_addr learned my addr 172.21.15.63:0/1761726985 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 989959437 0 0) 0x7f68180a6fa0 con 0x7f68180a1c40 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): my global_id is 4787 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67fc003530 con 0x7f68180a1c40 2024-09-16T05:36:41.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2822183523 0 0) 0x7f6818124670 con 0x7f6818137450 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): my global_id is 4788 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68180a6fa0 con 0x7f6818137450 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2485468200 0 0) 0x7f67fc003530 con 0x7f68180a1c40 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6818124670 con 0x7f68180a1c40 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6814001a40 con 0x7f68180a1c40 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): dump: 2024-09-16T05:36:41.385 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 940833073 0 0) 0x7f68180a6fa0 con 0x7f6818137450 2024-09-16T05:36:41.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67fc003530 con 0x7f6818137450 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6810004340 con 0x7f6818137450 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient(hunting): dump: 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.387 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3104850461 0 0) 0x7f6818124670 con 0x7f68180a1c40 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 >> v1:172.21.15.63:6789/0 conn(0x7f6818136960 legacy=0x7f68181d00d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 >> v1:172.21.15.6:6789/0 conn(0x7f6818137450 legacy=0x7f68181d3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 monclient: found mon.c 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6818124390 con 0x7f68180a1c40 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:41.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 5 monclient: authenticate success, global_id 4787 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: _renew_subs 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6818122f40 con 0x7f68180a1c40 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: _renew_subs 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6818136690 con 0x7f68180a1c40 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6814001d40 con 0x7f68180a1c40 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6814004c40 con 0x7f68180a1c40 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:41.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680effd640 4 set_mon_vals no callback set 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f680dffb640 10 monclient: dump: 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:41.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:41.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.372+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f8005090 con 0x7f68180a1c40 2024-09-16T05:36:41.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.376+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f681400f9e0 con 0x7f68180a1c40 2024-09-16T05:36:41.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.376+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(55..55 src has 1..55) ==== 3994+0+0 (unknown 3737852653 0 0) 0x7f6814044390 con 0x7f68180a1c40 2024-09-16T05:36:41.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.376+0000 7f681ccc7640 10 monclient: get_auth_request con 0x7f67fc0369d0 auth_method 0 2024-09-16T05:36:41.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.388+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6814019140 con 0x7f68180a1c40 2024-09-16T05:36:41.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.388+0000 7f680dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:41.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.388+0000 7f680dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:41.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.548+0000 7f681f72c640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.548+0000 7f681f72c640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.548+0000 7f681f72c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.548+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) -- 0x7f67f8005b90 con 0x7f68180a1c40 2024-09-16T05:36:41.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.548+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) ==== 90+0+8 (unknown 3280861249 0 1286160932) 0x7f681401da00 con 0x7f68180a1c40 2024-09-16T05:36:41.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.548+0000 7f680dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.548+0000 7f680dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:41.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) -- 0x7f67f8005610 con 0x7f68180a1c40 2024-09-16T05:36:41.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f680dffb640 1 -- 172.21.15.63:0/1761726985 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) ==== 90+0+8 (unknown 3280861249 0 1286160932) 0x7f68140402e0 con 0x7f68180a1c40 2024-09-16T05:36:41.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f680dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f680dffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:41.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 >> v1:172.21.15.63:6812/3643207211 conn(0x7f67fc0369d0 legacy=0x7f67fc038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 10 monclient: shutdown 2024-09-16T05:36:41.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:41.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 >> v1:172.21.15.6:6790/0 conn(0x7f68180a1c40 legacy=0x7f6818135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 shutdown_connections 2024-09-16T05:36:41.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 >> 172.21.15.63:0/1761726985 conn(0x7f68180a60a0 msgr2=0x7f68180a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:41.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 shutdown_connections 2024-09-16T05:36:41.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.644+0000 7f681f72c640 1 -- 172.21.15.63:0/1761726985 wait complete. 2024-09-16T05:36:41.660 INFO:tasks.workunit.client.0.smithi063.stdout:foo-hdd 2024-09-16T05:36:41.661 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-16T05:36:41.661 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false grep foo-ssd 2024-09-16T05:36:41.661 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:36:41.662 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep foo-ssd 2024-09-16T05:36:41.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient: get_monmap_and_config 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient: build_initial_monmap 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient: monmap: 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:41.776717+0000 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:41.776717+0000 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:41.777 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 Processor -- start 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- start start 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient: init 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient(hunting): picked mon.noname-a con 0x7f19440a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient(hunting): picked mon.noname-c con 0x7f19440a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient(hunting): picked mon.noname-b con 0x7f194413d3c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19440a4dd0 con 0x7f19440a3ca0 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19440a1870 con 0x7f19440a4790 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19440a5210 con 0x7f194413d3c0 2024-09-16T05:36:41.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient(hunting): authenticate will time out at 1058.553529s 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19427fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f19440a4790 legacy=0x7f194413bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1942ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f19440a3ca0 0x7f19440a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55666/0 (socket says 172.21.15.63:55666) 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1942ffd640 1 -- 172.21.15.63:0/2316583040 learned_addr learned my addr 172.21.15.63:0/2316583040 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985206117 0 0) 0x7f19440a4dd0 con 0x7f19440a3ca0 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient(hunting): my global_id is 4791 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f192c003530 con 0x7f19440a3ca0 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599305628 0 0) 0x7f19440a5210 con 0x7f194413d3c0 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:41.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient(hunting): my global_id is 4790 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19440a4dd0 con 0x7f194413d3c0 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4115508059 0 0) 0x7f19440a4dd0 con 0x7f194413d3c0 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 >> v1:172.21.15.63:6789/0 conn(0x7f19440a4790 legacy=0x7f194413bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 >> v1:172.21.15.6:6789/0 conn(0x7f19440a3ca0 legacy=0x7f19440a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 monclient: found mon.noname-b 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19440a0fd0 con 0x7f194413d3c0 2024-09-16T05:36:41.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 5 monclient: authenticate success, global_id 4790 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f193c001600 con 0x7f194413d3c0 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: dump: 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.784 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f193c001860 con 0x7f194413d3c0 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 4 set_mon_vals no callback set 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 1 -- 172.21.15.63:0/2316583040 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f193c001c30 con 0x7f194413d3c0 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:41.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f1941ffb640 10 monclient: dump: 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 10 monclient: shutdown 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:41.786 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- 172.21.15.63:0/2316583040 >> v1:172.21.15.6:6790/0 conn(0x7f194413d3c0 legacy=0x7f194413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- 172.21.15.63:0/2316583040 shutdown_connections 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- 172.21.15.63:0/2316583040 >> 172.21.15.63:0/2316583040 conn(0x7f19440a6650 msgr2=0x7f1944075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- 172.21.15.63:0/2316583040 shutdown_connections 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.776+0000 7f19494b0640 1 -- 172.21.15.63:0/2316583040 wait complete. 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: build_initial_monmap 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: monmap: 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:41.787 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:41.788 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:41.788 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 1 Processor -- start 2024-09-16T05:36:41.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 1 -- start start 2024-09-16T05:36:41.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: init 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient(hunting): picked mon.noname-a con 0x7f19440a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient(hunting): picked mon.noname-c con 0x7f19440a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient(hunting): picked mon.noname-b con 0x7f194413d3c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1944122420 con 0x7f19440a3ca0 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1944123e30 con 0x7f194413d3c0 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19440a8590 con 0x7f19440a4790 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:41.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient(hunting): authenticate will time out at 1058.557235s 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19437fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f194413d3c0 legacy=0x7f19441d3590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19427fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f19440a4790 0x7f1944120810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35948/0 (socket says 172.21.15.63:35948) 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19427fc640 1 -- 172.21.15.63:0/565831199 learned_addr learned my addr 172.21.15.63:0/565831199 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4157235983 0 0) 0x7f19440a8590 con 0x7f19440a4790 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): my global_id is 4793 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1930003560 con 0x7f19440a4790 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583895547 0 0) 0x7f1944122420 con 0x7f19440a3ca0 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:41.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): my global_id is 4794 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19440a8590 con 0x7f19440a3ca0 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2402129906 0 0) 0x7f1930003560 con 0x7f19440a4790 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1944122420 con 0x7f19440a4790 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1938004160 con 0x7f19440a4790 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.794 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3408612250 0 0) 0x7f19440a8590 con 0x7f19440a3ca0 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1930003560 con 0x7f19440a3ca0 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1934001c60 con 0x7f19440a3ca0 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:41.795 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1048391138 0 0) 0x7f1944122420 con 0x7f19440a4790 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 >> v1:172.21.15.63:6789/0 conn(0x7f194413d3c0 legacy=0x7f19441d3590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 >> v1:172.21.15.6:6789/0 conn(0x7f19440a3ca0 legacy=0x7f1944120100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:41.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 monclient: found mon.c 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19440a80f0 con 0x7f19440a4790 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 5 monclient: authenticate success, global_id 4793 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: _renew_subs 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1944123870 con 0x7f19440a4790 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: _renew_subs 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1944120f20 con 0x7f19440a4790 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f19380043c0 con 0x7f19440a4790 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1938004710 con 0x7f19440a4790 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f192b7fe640 10 monclient: dump: 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:41.798 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19417fa640 4 set_mon_vals no callback set 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.780+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1924005090 con 0x7f19440a4790 2024-09-16T05:36:41.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.784+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f19380048b0 con 0x7f19440a4790 2024-09-16T05:36:41.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.784+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(55..55 src has 1..55) ==== 3994+0+0 (unknown 3737852653 0 0) 0x7f1938044200 con 0x7f19440a4790 2024-09-16T05:36:41.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.784+0000 7f1942ffd640 10 monclient: get_auth_request con 0x7f1930036b90 auth_method 0 2024-09-16T05:36:41.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.796+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f19380152e0 con 0x7f19440a4790 2024-09-16T05:36:41.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.796+0000 7f192b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:41.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.796+0000 7f192b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:41.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.960+0000 7f19494b0640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.960+0000 7f19494b0640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.960+0000 7f19494b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:41.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.960+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) -- 0x7f1924005b90 con 0x7f19440a4790 2024-09-16T05:36:41.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.960+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) ==== 90+0+8 (unknown 3280861249 0 1286160932) 0x7f1938031e20 con 0x7f19440a4790 2024-09-16T05:36:41.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.960+0000 7f192b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:41.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:41.960+0000 7f192b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:42.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.056+0000 7f19494b0640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:42.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.056+0000 7f19494b0640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:42.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.056+0000 7f19494b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:42.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.056+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) -- 0x7f1924005610 con 0x7f19440a4790 2024-09-16T05:36:42.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.056+0000 7f192b7fe640 1 -- 172.21.15.63:0/565831199 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v55) ==== 90+0+8 (unknown 3280861249 0 1286160932) 0x7f1938014be0 con 0x7f19440a4790 2024-09-16T05:36:42.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.056+0000 7f192b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T05:36:42.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.056+0000 7f192b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:42.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.060+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1930036b90 legacy=0x7f1930039050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:42.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.060+0000 7f19494b0640 10 monclient: shutdown 2024-09-16T05:36:42.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.060+0000 7f19494b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:42.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.060+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 >> v1:172.21.15.6:6790/0 conn(0x7f19440a4790 legacy=0x7f1944120810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:42.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.060+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 shutdown_connections 2024-09-16T05:36:42.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.060+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 >> 172.21.15.63:0/565831199 conn(0x7f19440a6650 msgr2=0x7f194413c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:42.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.060+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 shutdown_connections 2024-09-16T05:36:42.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.060+0000 7f19494b0640 1 -- 172.21.15.63:0/565831199 wait complete. 2024-09-16T05:36:42.076 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:36:42.076 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2024-09-16T05:36:42.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient: get_monmap_and_config 2024-09-16T05:36:42.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient: build_initial_monmap 2024-09-16T05:36:42.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient: monmap: 2024-09-16T05:36:42.194 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:42.194 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:42.194 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:42.194095+0000 2024-09-16T05:36:42.194 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:42.194095+0000 2024-09-16T05:36:42.194 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:42.195 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:42.195 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:42.195 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:42.195 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:42.195 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:42.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 1 Processor -- start 2024-09-16T05:36:42.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 1 -- start start 2024-09-16T05:36:42.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient: init 2024-09-16T05:36:42.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:42.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:42.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient(hunting): picked mon.noname-b con 0x7fa6540a22a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:42.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient(hunting): picked mon.noname-a con 0x7fa6540731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient(hunting): picked mon.noname-c con 0x7fa6541395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa65409c580 con 0x7fa6540731d0 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6540a1300 con 0x7fa6541395a0 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa65409d0f0 con 0x7fa6540a22a0 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa65a34b640 10 monclient(hunting): authenticate will time out at 1058.971069s 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa6588e8640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa6541395a0 legacy=0x7fa65413b990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa6537fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6540731d0 0x7fa65409fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55692/0 (socket says 172.21.15.63:55692) 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa653fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa6540a22a0 0x7fa65409f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35960/0 (socket says 172.21.15.63:35960) 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.191+0000 7fa6537fe640 1 -- 172.21.15.63:0/4009917592 learned_addr learned my addr 172.21.15.63:0/4009917592 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2506558919 0 0) 0x7fa65409c580 con 0x7fa6540731d0 2024-09-16T05:36:42.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient(hunting): my global_id is 4797 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa640003530 con 0x7fa6540731d0 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278107935 0 0) 0x7fa65409d0f0 con 0x7fa6540a22a0 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient(hunting): my global_id is 4796 2024-09-16T05:36:42.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa65409c580 con 0x7fa6540a22a0 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 111177515 0 0) 0x7fa65409c580 con 0x7fa6540a22a0 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 >> v1:172.21.15.63:6789/0 conn(0x7fa6541395a0 legacy=0x7fa65413b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 >> v1:172.21.15.6:6789/0 conn(0x7fa6540731d0 legacy=0x7fa65409fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 monclient: found mon.noname-b 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65409d3d0 con 0x7fa6540a22a0 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa63c001600 con 0x7fa6540a22a0 2024-09-16T05:36:42.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: dump: 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 5 monclient: authenticate success, global_id 4796 2024-09-16T05:36:42.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa63c001860 con 0x7fa6540a22a0 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 4 set_mon_vals no callback set 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 1 -- 172.21.15.63:0/4009917592 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa63c001c70 con 0x7fa6540a22a0 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa652ffd640 10 monclient: dump: 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.203 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient: shutdown 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- 172.21.15.63:0/4009917592 >> v1:172.21.15.6:6790/0 conn(0x7fa6540a22a0 legacy=0x7fa65409f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- 172.21.15.63:0/4009917592 shutdown_connections 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- 172.21.15.63:0/4009917592 >> 172.21.15.63:0/4009917592 conn(0x7fa6540a1e90 msgr2=0x7fa654075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- 172.21.15.63:0/4009917592 shutdown_connections 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- 172.21.15.63:0/4009917592 wait complete. 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient: build_initial_monmap 2024-09-16T05:36:42.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient: monmap: 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:42.205 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:42.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 Processor -- start 2024-09-16T05:36:42.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- start start 2024-09-16T05:36:42.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient: init 2024-09-16T05:36:42.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:42.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:42.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient(hunting): picked mon.noname-c con 0x7fa6540731d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient(hunting): picked mon.noname-a con 0x7fa6540a22a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient(hunting): picked mon.noname-b con 0x7fa6541395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa654122400 con 0x7fa6540a22a0 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6540a3dc0 con 0x7fa6541395a0 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa65409c0f0 con 0x7fa6540731d0 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa65a34b640 10 monclient(hunting): authenticate will time out at 1058.974782s 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa6588e8640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa6541395a0 legacy=0x7fa6541208c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa653fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa6540731d0 0x7fa65411faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35970/0 (socket says 172.21.15.63:35970) 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa653fff640 1 -- 172.21.15.63:0/24541286 learned_addr learned my addr 172.21.15.63:0/24541286 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1630892905 0 0) 0x7fa65409c0f0 con 0x7fa6540731d0 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 10 monclient(hunting): my global_id is 4799 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa634003530 con 0x7fa6540731d0 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065321838 0 0) 0x7fa654122400 con 0x7fa6540a22a0 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 10 monclient(hunting): my global_id is 4800 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.195+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa65409c0f0 con 0x7fa6540a22a0 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2687404539 0 0) 0x7fa634003530 con 0x7fa6540731d0 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa654122400 con 0x7fa6540731d0 2024-09-16T05:36:42.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa63c001cb0 con 0x7fa6540731d0 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:42.212 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3106537309 0 0) 0x7fa65409c0f0 con 0x7fa6540a22a0 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa634003530 con 0x7fa6540a22a0 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa644001b60 con 0x7fa6540a22a0 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient(hunting): dump: 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:42.213 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1324325655 0 0) 0x7fa654122400 con 0x7fa6540731d0 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 >> v1:172.21.15.63:6789/0 conn(0x7fa6541395a0 legacy=0x7fa6541208c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 >> v1:172.21.15.6:6789/0 conn(0x7fa6540a22a0 legacy=0x7fa6541201b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 monclient: found mon.c 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6540a42f0 con 0x7fa6540731d0 2024-09-16T05:36:42.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 5 monclient: authenticate success, global_id 4799 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 10 monclient: _renew_subs 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa654123300 con 0x7fa6540731d0 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 10 monclient: _renew_subs 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6541d47c0 con 0x7fa6540731d0 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa63c001860 con 0x7fa6540731d0 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa63c004d60 con 0x7fa6540731d0 2024-09-16T05:36:42.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa651ffb640 4 set_mon_vals no callback set 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 10 monclient: dump: 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:42.216 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:42.217 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:42.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:42.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:42.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:42.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa638005090 con 0x7fa6540731d0 2024-09-16T05:36:42.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa63c004f80 con 0x7fa6540731d0 2024-09-16T05:36:42.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(55..55 src has 1..55) ==== 3994+0+0 (unknown 3737852653 0 0) 0x7fa63c0443b0 con 0x7fa6540731d0 2024-09-16T05:36:42.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.199+0000 7fa6537fe640 10 monclient: get_auth_request con 0x7fa634036a70 auth_method 0 2024-09-16T05:36:42.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.211+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa63c0155b0 con 0x7fa6540731d0 2024-09-16T05:36:42.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.211+0000 7fa650ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:42.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.211+0000 7fa650ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:42.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.383+0000 7fa65a34b640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T05:36:42.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.383+0000 7fa65a34b640 10 monclient: _send_command 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T05:36:42.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.383+0000 7fa65a34b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:42.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:42.383+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) -- 0x7fa638005380 con 0x7fa6540731d0 2024-09-16T05:36:43.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.295+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v56) ==== 151+0+0 (unknown 333439644 0 0) 0x7fa63c014eb0 con 0x7fa6540731d0 2024-09-16T05:36:43.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.295+0000 7fa650ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T05:36:43.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.295+0000 7fa650ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:43.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.399+0000 7fa65a34b640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T05:36:43.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.399+0000 7fa65a34b640 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T05:36:43.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.399+0000 7fa65a34b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:43.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.399+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) -- 0x7fa638005ac0 con 0x7fa6540731d0 2024-09-16T05:36:43.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.403+0000 7fa650ff9640 1 -- 172.21.15.63:0/24541286 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v56) ==== 151+0+0 (unknown 333439644 0 0) 0x7fa63c042100 con 0x7fa6540731d0 2024-09-16T05:36:43.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.403+0000 7fa650ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T05:36:43.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.403+0000 7fa650ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:43.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.407+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa634036a70 legacy=0x7fa634038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:43.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.407+0000 7fa65a34b640 10 monclient: shutdown 2024-09-16T05:36:43.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.407+0000 7fa65a34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:43.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.407+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 >> v1:172.21.15.6:6790/0 conn(0x7fa6540731d0 legacy=0x7fa65411faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:43.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.407+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 shutdown_connections 2024-09-16T05:36:43.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.407+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 >> 172.21.15.63:0/24541286 conn(0x7fa6540a1e90 msgr2=0x7fa654138590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:43.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.407+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 shutdown_connections 2024-09-16T05:36:43.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.407+0000 7fa65a34b640 1 -- 172.21.15.63:0/24541286 wait complete. 2024-09-16T05:36:43.423 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2024-09-16T05:36:43.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient: get_monmap_and_config 2024-09-16T05:36:43.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient: build_initial_monmap 2024-09-16T05:36:43.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient: monmap: 2024-09-16T05:36:43.550 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:43.550 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:43.550 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:43.550151+0000 2024-09-16T05:36:43.550 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:43.550151+0000 2024-09-16T05:36:43.551 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:43.551 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:43.551 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:43.551 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:43.551 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:43.551 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:43.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 1 Processor -- start 2024-09-16T05:36:43.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 1 -- start start 2024-09-16T05:36:43.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient: init 2024-09-16T05:36:43.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:43.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:43.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient(hunting): picked mon.noname-b con 0x7f21e40a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:43.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient(hunting): picked mon.noname-a con 0x7f21e40a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient(hunting): picked mon.noname-c con 0x7f21e40a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e40a53b0 con 0x7f21e40a07a0 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e40a0290 con 0x7f21e40a4600 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e40a0dc0 con 0x7f21e40a3b10 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21eb9eb640 10 monclient(hunting): authenticate will time out at 1060.327157s 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21e9f88640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f21e40a4600 legacy=0x7f21e413f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21e8f86640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f21e40a07a0 0x7f21e413be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:55702/0 (socket says 172.21.15.63:55702) 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21e8f86640 1 -- 172.21.15.63:0/188013869 learned_addr learned my addr 172.21.15.63:0/188013869 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877114410 0 0) 0x7f21e40a53b0 con 0x7f21e40a07a0 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21dbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:43.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.547+0000 7f21dbfff640 10 monclient(hunting): my global_id is 4803 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21d0003530 con 0x7f21e40a07a0 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776676795 0 0) 0x7f21e40a0dc0 con 0x7f21e40a3b10 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient(hunting): my global_id is 4802 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21e40a53b0 con 0x7f21e40a3b10 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1388726329 0 0) 0x7f21e40a53b0 con 0x7f21e40a3b10 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 >> v1:172.21.15.63:6789/0 conn(0x7f21e40a4600 legacy=0x7f21e413f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:43.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 >> v1:172.21.15.6:6789/0 conn(0x7f21e40a07a0 legacy=0x7f21e413be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 monclient: found mon.noname-b 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e40a10a0 con 0x7f21e40a3b10 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: _finish_auth 0 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f21e0001600 con 0x7f21e40a3b10 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: dump: 2024-09-16T05:36:43.556 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 5 monclient: authenticate success, global_id 4802 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21e0001870 con 0x7f21e40a3b10 2024-09-16T05:36:43.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 1 -- 172.21.15.63:0/188013869 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f21e0001c40 con 0x7f21e40a3b10 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 4 set_mon_vals no callback set 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21dbfff640 10 monclient: dump: 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:43.558 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:43.559 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:43.559 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:43.559 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:43.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient: shutdown 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- 172.21.15.63:0/188013869 >> v1:172.21.15.6:6790/0 conn(0x7f21e40a3b10 legacy=0x7f21e40a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- 172.21.15.63:0/188013869 shutdown_connections 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- 172.21.15.63:0/188013869 >> 172.21.15.63:0/188013869 conn(0x7f21e4075470 msgr2=0x7f21e4075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- 172.21.15.63:0/188013869 shutdown_connections 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- 172.21.15.63:0/188013869 wait complete. 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient: build_initial_monmap 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient: monmap: 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:43.564 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 Processor -- start 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- start start 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient: init 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:43.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient(hunting): picked mon.noname-c con 0x7f21e40a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient(hunting): picked mon.noname-a con 0x7f21e40a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient(hunting): picked mon.noname-b con 0x7f21e40a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e4125c90 con 0x7f21e40a3b10 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e40a8860 con 0x7f21e40a4600 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e40a8350 con 0x7f21e40a07a0 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21eb9eb640 10 monclient(hunting): authenticate will time out at 1060.330864s 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21e9f88640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f21e40a4600 legacy=0x7f21e41d39d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21e9787640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f21e40a07a0 0x7f21e4121f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:36000/0 (socket says 172.21.15.63:36000) 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21e9787640 1 -- 172.21.15.63:0/2807350509 learned_addr learned my addr 172.21.15.63:0/2807350509 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365716719 0 0) 0x7f21e40a8350 con 0x7f21e40a07a0 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:43.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 10 monclient(hunting): my global_id is 4805 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21c8003560 con 0x7f21e40a07a0 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 548866065 0 0) 0x7f21e4125c90 con 0x7f21e40a3b10 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 10 monclient(hunting): my global_id is 4806 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.551+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21e40a8350 con 0x7f21e40a3b10 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2646971692 0 0) 0x7f21e40a8350 con 0x7f21e40a3b10 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21e4125c90 con 0x7f21e40a3b10 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f21d40018a0 con 0x7f21e40a3b10 2024-09-16T05:36:43.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient(hunting): dump: 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:43.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1158415689 0 0) 0x7f21c8003560 con 0x7f21e40a07a0 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21e40a8350 con 0x7f21e40a07a0 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f21e0001c80 con 0x7f21e40a07a0 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient(hunting): dump: 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3717172820 0 0) 0x7f21e4125c90 con 0x7f21e40a3b10 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 >> v1:172.21.15.6:6790/0 conn(0x7f21e40a07a0 legacy=0x7f21e4121f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 >> v1:172.21.15.63:6789/0 conn(0x7f21e40a4600 legacy=0x7f21e41d39d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 monclient: found mon.a 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e40a0500 con 0x7f21e40a3b10 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 5 monclient: authenticate success, global_id 4806 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 10 monclient: _renew_subs 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f21e41256d0 con 0x7f21e40a3b10 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 10 monclient: _renew_subs 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f21e4124280 con 0x7f21e40a3b10 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21d40047f0 con 0x7f21e40a3b10 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f21d4004b40 con 0x7f21e40a3b10 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:43.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 10 monclient: dump: 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21daffd640 4 set_mon_vals no callback set 2024-09-16T05:36:43.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:43.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:43.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:43.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21c4005090 con 0x7f21e40a3b10 2024-09-16T05:36:43.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f21d400f8e0 con 0x7f21e40a3b10 2024-09-16T05:36:43.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(56..56 src has 1..56) ==== 4291+0+0 (unknown 1941826092 0 0) 0x7f21d4044320 con 0x7f21e40a3b10 2024-09-16T05:36:43.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.555+0000 7f21e9787640 10 monclient: get_auth_request con 0x7f21c8036a00 auth_method 0 2024-09-16T05:36:43.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.567+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f21d4015270 con 0x7f21e40a3b10 2024-09-16T05:36:43.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.567+0000 7f21d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:43.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.567+0000 7f21d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:43.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.735+0000 7f21eb9eb640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T05:36:43.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.735+0000 7f21eb9eb640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T05:36:43.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.735+0000 7f21eb9eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:43.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:43.735+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) -- 0x7f21c4005380 con 0x7f21e40a3b10 2024-09-16T05:36:45.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.303+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' created v58) ==== 177+0+0 (unknown 3937312289 0 0) 0x7f21d4018f20 con 0x7f21e40a3b10 2024-09-16T05:36:45.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.303+0000 7f21d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T05:36:45.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.303+0000 7f21d9ffb640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' created 2024-09-16T05:36:45.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.407+0000 7f21eb9eb640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T05:36:45.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.407+0000 7f21eb9eb640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T05:36:45.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.407+0000 7f21eb9eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:45.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.407+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) -- 0x7f21c4001170 con 0x7f21e40a3b10 2024-09-16T05:36:45.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.407+0000 7f21d9ffb640 1 -- 172.21.15.63:0/2807350509 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v58) ==== 184+0+0 (unknown 3376246152 0 0) 0x7f21d401d7e0 con 0x7f21e40a3b10 2024-09-16T05:36:45.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.407+0000 7f21d9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T05:36:45.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.407+0000 7f21d9ffb640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' already exists 2024-09-16T05:36:45.411 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'ec-foo' already exists 2024-09-16T05:36:45.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.411+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 >> v1:172.21.15.63:6812/3643207211 conn(0x7f21c8036a00 legacy=0x7f21c8038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:45.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.411+0000 7f21eb9eb640 10 monclient: shutdown 2024-09-16T05:36:45.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.411+0000 7f21eb9eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:45.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.411+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 >> v1:172.21.15.6:6789/0 conn(0x7f21e40a3b10 legacy=0x7f21e4122670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:45.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.411+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 shutdown_connections 2024-09-16T05:36:45.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.411+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 >> 172.21.15.63:0/2807350509 conn(0x7f21e4075470 msgr2=0x7f21e41bcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:45.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.411+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 shutdown_connections 2024-09-16T05:36:45.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.411+0000 7f21eb9eb640 1 -- 172.21.15.63:0/2807350509 wait complete. 2024-09-16T05:36:45.428 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2024-09-16T05:36:45.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.547+0000 7f208687d640 10 monclient: get_monmap_and_config 2024-09-16T05:36:45.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.547+0000 7f208687d640 10 monclient: build_initial_monmap 2024-09-16T05:36:45.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.547+0000 7f208687d640 10 monclient: monmap: 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:45.552460+0000 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:45.552460+0000 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:45.553 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:45.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 Processor -- start 2024-09-16T05:36:45.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- start start 2024-09-16T05:36:45.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient: init 2024-09-16T05:36:45.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:45.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:45.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient(hunting): picked mon.noname-a con 0x7f2080134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient(hunting): picked mon.noname-c con 0x7f20800a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient(hunting): picked mon.noname-b con 0x7f208006dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080135420 con 0x7f2080134930 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208009e830 con 0x7f20800a0810 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20800a0dc0 con 0x7f208006dfa0 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient(hunting): authenticate will time out at 1062.329553s 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f20800a0810 legacy=0x7f2080139820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2080134930 0x7f2080134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37176/0 (socket says 172.21.15.63:37176) 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207ffff640 1 -- 172.21.15.63:0/3835521570 learned_addr learned my addr 172.21.15.63:0/3835521570 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468214825 0 0) 0x7f2080135420 con 0x7f2080134930 2024-09-16T05:36:45.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient(hunting): my global_id is 4809 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2068003530 con 0x7f2080134930 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 767473861 0 0) 0x7f20800a0dc0 con 0x7f208006dfa0 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient(hunting): my global_id is 4808 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2080135420 con 0x7f208006dfa0 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 429306157 0 0) 0x7f2068003530 con 0x7f2080134930 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 >> v1:172.21.15.6:6790/0 conn(0x7f208006dfa0 legacy=0x7f208013cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:45.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 >> v1:172.21.15.63:6789/0 conn(0x7f20800a0810 legacy=0x7f2080139820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 monclient: found mon.noname-a 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20800a2500 con 0x7f2080134930 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: _finish_auth 0 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2070001550 con 0x7f2080134930 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 5 monclient: authenticate success, global_id 4809 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:45.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: dump: 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:45.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20700017b0 con 0x7f2080134930 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 4 set_mon_vals no callback set 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 1 -- 172.21.15.63:0/3835521570 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2070001b80 con 0x7f2080134930 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f207effd640 10 monclient: dump: 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.561 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 10 monclient: shutdown 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- 172.21.15.63:0/3835521570 >> v1:172.21.15.6:6789/0 conn(0x7f2080134930 legacy=0x7f2080134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- 172.21.15.63:0/3835521570 shutdown_connections 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- 172.21.15.63:0/3835521570 >> 172.21.15.63:0/3835521570 conn(0x7f20800a1e90 msgr2=0x7f2080075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- 172.21.15.63:0/3835521570 shutdown_connections 2024-09-16T05:36:45.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.551+0000 7f208687d640 1 -- 172.21.15.63:0/3835521570 wait complete. 2024-09-16T05:36:45.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: build_initial_monmap 2024-09-16T05:36:45.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: monmap: 2024-09-16T05:36:45.566 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:45.566 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 1 Processor -- start 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 1 -- start start 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: init 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:45.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:45.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient(hunting): picked mon.noname-a con 0x7f208006dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:45.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient(hunting): picked mon.noname-c con 0x7f20800a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:45.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient(hunting): picked mon.noname-b con 0x7f2080134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:45.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080136780 con 0x7f208006dfa0 2024-09-16T05:36:45.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2080135b40 con 0x7f2080134930 2024-09-16T05:36:45.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208009d6f0 con 0x7f20800a0810 2024-09-16T05:36:45.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:45.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient(hunting): authenticate will time out at 1062.333383s 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f2084e1a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2080134930 legacy=0x7f20801d37e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207f7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f20800a0810 0x7f208009c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55048/0 (socket says 172.21.15.63:55048) 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207f7fe640 1 -- 172.21.15.63:0/49921296 learned_addr learned my addr 172.21.15.63:0/49921296 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2438317277 0 0) 0x7f2080136780 con 0x7f208006dfa0 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): my global_id is 4812 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f206c003530 con 0x7f208006dfa0 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1383015545 0 0) 0x7f208009d6f0 con 0x7f20800a0810 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:45.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): my global_id is 4811 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2080136780 con 0x7f20800a0810 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1524426099 0 0) 0x7f206c003530 con 0x7f208006dfa0 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f208009d6f0 con 0x7f208006dfa0 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2070001510 con 0x7f208006dfa0 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): dump: 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.570 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2250307027 0 0) 0x7f2080136780 con 0x7f20800a0810 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f206c003530 con 0x7f20800a0810 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20740040e0 con 0x7f20800a0810 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:45.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient(hunting): dump: 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2541614490 0 0) 0x7f208009d6f0 con 0x7f208006dfa0 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 >> v1:172.21.15.6:6790/0 conn(0x7f20800a0810 legacy=0x7f208009c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 >> v1:172.21.15.63:6789/0 conn(0x7f2080134930 legacy=0x7f20801d37e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:45.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 monclient: found mon.a 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20800a3fa0 con 0x7f208006dfa0 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient: _finish_auth 0 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 5 monclient: authenticate success, global_id 4812 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: _renew_subs 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20801355f0 con 0x7f208006dfa0 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: _renew_subs 2024-09-16T05:36:45.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f208009e1c0 con 0x7f208006dfa0 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20700017b0 con 0x7f208006dfa0 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2070004c90 con 0x7f208006dfa0 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207e7fc640 4 set_mon_vals no callback set 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.555+0000 7f207cff9640 10 monclient: dump: 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:45.574 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.559+0000 7f208687d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.559+0000 7f208687d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.559+0000 7f208687d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.559+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2060005090 con 0x7f208006dfa0 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.559+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2070004eb0 con 0x7f208006dfa0 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.559+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4732+0+0 (unknown 3196964064 0 0) 0x7f2070044560 con 0x7f208006dfa0 2024-09-16T05:36:45.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.559+0000 7f207f7fe640 10 monclient: get_auth_request con 0x7f206c036a20 auth_method 0 2024-09-16T05:36:45.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.571+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f20700154e0 con 0x7f208006dfa0 2024-09-16T05:36:45.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.571+0000 7f207cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:45.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.571+0000 7f207cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:45.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.735+0000 7f208687d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:45.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.735+0000 7f208687d640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:45.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.735+0000 7f208687d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:45.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.735+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2060005380 con 0x7f208006dfa0 2024-09-16T05:36:45.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.979+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' removed v59) ==== 154+0+0 (unknown 1121732617 0 0) 0x7f2070014de0 con 0x7f208006dfa0 2024-09-16T05:36:45.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.979+0000 7f207cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:45.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:45.979+0000 7f207cff9640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' removed 2024-09-16T05:36:46.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:46.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:46.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:46.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2060001150 con 0x7f208006dfa0 2024-09-16T05:36:46.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f207cff9640 1 -- 172.21.15.63:0/49921296 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' does not exist v59) ==== 161+0+0 (unknown 2063762907 0 0) 0x7f20700403f0 con 0x7f208006dfa0 2024-09-16T05:36:46.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f207cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T05:36:46.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f207cff9640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' does not exist 2024-09-16T05:36:46.082 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'ec-foo' does not exist 2024-09-16T05:36:46.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 >> v1:172.21.15.63:6812/3643207211 conn(0x7f206c036a20 legacy=0x7f206c038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 10 monclient: shutdown 2024-09-16T05:36:46.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:46.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 >> v1:172.21.15.6:6789/0 conn(0x7f208006dfa0 legacy=0x7f208009c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 shutdown_connections 2024-09-16T05:36:46.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.079+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 >> 172.21.15.63:0/49921296 conn(0x7f20800a1e90 msgr2=0x7f208012d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:46.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.083+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 shutdown_connections 2024-09-16T05:36:46.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.083+0000 7f208687d640 1 -- 172.21.15.63:0/49921296 wait complete. 2024-09-16T05:36:46.099 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls 2024-09-16T05:36:46.099 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep foo 2024-09-16T05:36:46.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient: get_monmap_and_config 2024-09-16T05:36:46.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient: build_initial_monmap 2024-09-16T05:36:46.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient: monmap: 2024-09-16T05:36:46.219 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:46.219 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.219 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:46.218987+0000 2024-09-16T05:36:46.219 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:46.218987+0000 2024-09-16T05:36:46.220 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:46.220 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.220 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:46.220 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:46.220 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:46.220 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 1 Processor -- start 2024-09-16T05:36:46.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 1 -- start start 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient: init 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient(hunting): picked mon.noname-a con 0x7f292409ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient(hunting): picked mon.noname-b con 0x7f292406dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient(hunting): picked mon.noname-c con 0x7f29240a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f292409f800 con 0x7f292409ac90 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29240a0ab0 con 0x7f29240a0210 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29240a0d90 con 0x7f292406dd30 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.215+0000 7f2928dc7640 10 monclient(hunting): authenticate will time out at 1062.995932s 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2922d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f29240a0210 legacy=0x7f292413d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:46.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f292259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f292409ac90 0x7f292409b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37194/0 (socket says 172.21.15.63:37194) 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f292259c640 1 -- 172.21.15.63:0/406989476 learned_addr learned my addr 172.21.15.63:0/406989476 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889059493 0 0) 0x7f292409f800 con 0x7f292409ac90 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient(hunting): my global_id is 4815 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2910003530 con 0x7f292409ac90 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922464796 0 0) 0x7f29240a0d90 con 0x7f292406dd30 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:46.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient(hunting): my global_id is 4814 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f292409f800 con 0x7f292406dd30 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1132261684 0 0) 0x7f2910003530 con 0x7f292409ac90 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 >> v1:172.21.15.6:6790/0 conn(0x7f292406dd30 legacy=0x7f29241399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 >> v1:172.21.15.63:6789/0 conn(0x7f29240a0210 legacy=0x7f292413d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 monclient: found mon.noname-a 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292409e770 con 0x7f292409ac90 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: _finish_auth 0 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:46.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2914001620 con 0x7f292409ac90 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: dump: 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.226 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 5 monclient: authenticate success, global_id 4815 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2914001880 con 0x7f292409ac90 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 4 set_mon_vals no callback set 2024-09-16T05:36:46.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 1 -- 172.21.15.63:0/406989476 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2914001c90 con 0x7f292409ac90 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2921d9b640 10 monclient: dump: 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.228 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient: shutdown 2024-09-16T05:36:46.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:46.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- 172.21.15.63:0/406989476 >> v1:172.21.15.6:6789/0 conn(0x7f292409ac90 legacy=0x7f292409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- 172.21.15.63:0/406989476 shutdown_connections 2024-09-16T05:36:46.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- 172.21.15.63:0/406989476 >> 172.21.15.63:0/406989476 conn(0x7f29240a1c90 msgr2=0x7f29240a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:46.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- 172.21.15.63:0/406989476 shutdown_connections 2024-09-16T05:36:46.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- 172.21.15.63:0/406989476 wait complete. 2024-09-16T05:36:46.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient: build_initial_monmap 2024-09-16T05:36:46.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient: monmap: 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 Processor -- start 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- start start 2024-09-16T05:36:46.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient: init 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient(hunting): picked mon.noname-b con 0x7f292406dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient(hunting): picked mon.noname-c con 0x7f292409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient(hunting): picked mon.noname-a con 0x7f29240a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f292259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f292406dd30 legacy=0x7f2924134760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29241c77f0 con 0x7f29240a0210 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f292409c600 con 0x7f292406dd30 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2924122e50 con 0x7f292409ac90 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.219+0000 7f2928dc7640 10 monclient(hunting): authenticate will time out at 1062.999718s 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f292409ac90 0x7f29241cbc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55064/0 (socket says 172.21.15.63:55064) 2024-09-16T05:36:46.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2922d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f29240a0210 0x7f29241cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37200/0 (socket says 172.21.15.63:37200) 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291bfff640 1 -- 172.21.15.63:0/1262168324 learned_addr learned my addr 172.21.15.63:0/1262168324 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090677547 0 0) 0x7f2924122e50 con 0x7f292409ac90 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): my global_id is 4817 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2904003530 con 0x7f292409ac90 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421559598 0 0) 0x7f291c003d40 con 0x7f29240a0210 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): my global_id is 4818 2024-09-16T05:36:46.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2924122e50 con 0x7f29240a0210 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3461509379 0 0) 0x7f2904003530 con 0x7f292409ac90 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f291c003d40 con 0x7f292409ac90 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f290c0018c0 con 0x7f292409ac90 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): dump: 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.236 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3617376036 0 0) 0x7f2924122e50 con 0x7f29240a0210 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2904003530 con 0x7f29240a0210 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f291c004510 con 0x7f29240a0210 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient(hunting): dump: 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.237 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 6623140 0 0) 0x7f291c003d40 con 0x7f292409ac90 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 >> v1:172.21.15.63:6789/0 conn(0x7f292406dd30 legacy=0x7f2924134760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 >> v1:172.21.15.6:6789/0 conn(0x7f29240a0210 legacy=0x7f29241cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 monclient: found mon.c 2024-09-16T05:36:46.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2924123130 con 0x7f292409ac90 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient: _finish_auth 0 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 5 monclient: authenticate success, global_id 4817 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 10 monclient: _renew_subs 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f292409e2b0 con 0x7f292409ac90 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 10 monclient: _renew_subs 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2924122b70 con 0x7f292409ac90 2024-09-16T05:36:46.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f290c001c90 con 0x7f292409ac90 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f290c003a50 con 0x7f292409ac90 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f292159a640 4 set_mon_vals no callback set 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 10 monclient: dump: 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2908005090 con 0x7f292409ac90 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f290c003c70 con 0x7f292409ac90 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(59..59 src has 1..59) ==== 4377+0+0 (unknown 53190516 0 0) 0x7f290c0444d0 con 0x7f292409ac90 2024-09-16T05:36:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.223+0000 7f292259c640 10 monclient: get_auth_request con 0x7f29040369d0 auth_method 0 2024-09-16T05:36:46.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.239+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f290c015490 con 0x7f292409ac90 2024-09-16T05:36:46.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.239+0000 7f291affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:46.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.239+0000 7f291affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:46.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.407+0000 7f2928dc7640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:46.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.407+0000 7f2928dc7640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:46.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.407+0000 7f2928dc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:46.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.407+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f2908005380 con 0x7f292409ac90 2024-09-16T05:36:46.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.407+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) ==== 65+0+60 (unknown 1101553479 0 1596542489) 0x7f290c014d90 con 0x7f292409ac90 2024-09-16T05:36:46.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.407+0000 7f291affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:46.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.407+0000 7f291affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:46.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.507+0000 7f2928dc7640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:46.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.507+0000 7f2928dc7640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:46.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.507+0000 7f2928dc7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:46.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.507+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f2908005ac0 con 0x7f292409ac90 2024-09-16T05:36:46.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.507+0000 7f291affd640 1 -- 172.21.15.63:0/1262168324 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) ==== 65+0+60 (unknown 1101553479 0 1596542489) 0x7f290c0404c0 con 0x7f292409ac90 2024-09-16T05:36:46.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.507+0000 7f291affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:46.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.507+0000 7f291affd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:46.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.511+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 >> v1:172.21.15.63:6812/3643207211 conn(0x7f29040369d0 legacy=0x7f2904038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.511+0000 7f2928dc7640 10 monclient: shutdown 2024-09-16T05:36:46.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.511+0000 7f2928dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:46.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.511+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 >> v1:172.21.15.6:6790/0 conn(0x7f292409ac90 legacy=0x7f29241cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.511+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 shutdown_connections 2024-09-16T05:36:46.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.511+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 >> 172.21.15.63:0/1262168324 conn(0x7f29240a1c90 msgr2=0x7f29240a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:46.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.511+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 shutdown_connections 2024-09-16T05:36:46.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.511+0000 7f2928dc7640 1 -- 172.21.15.63:0/1262168324 wait complete. 2024-09-16T05:36:46.527 INFO:tasks.workunit.client.0.smithi063.stdout:foo 2024-09-16T05:36:46.528 INFO:tasks.workunit.client.0.smithi063.stdout:foo-ssd 2024-09-16T05:36:46.528 INFO:tasks.workunit.client.0.smithi063.stdout:foo-hdd 2024-09-16T05:36:46.528 INFO:tasks.workunit.client.0.smithi063.stdout:ec-foo 2024-09-16T05:36:46.528 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-16T05:36:46.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient: get_monmap_and_config 2024-09-16T05:36:46.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient: build_initial_monmap 2024-09-16T05:36:46.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient: monmap: 2024-09-16T05:36:46.650 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:46.650 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.650 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:46.650035+0000 2024-09-16T05:36:46.650 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:46.650035+0000 2024-09-16T05:36:46.651 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:46.651 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.651 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:46.651 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:46.651 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:46.651 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 1 Processor -- start 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 1 -- start start 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient: init 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient(hunting): picked mon.noname-a con 0x7fa33c134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient(hunting): picked mon.noname-b con 0x7fa33c0a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient(hunting): picked mon.noname-c con 0x7fa33c06dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33c135420 con 0x7fa33c134930 2024-09-16T05:36:46.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33c09e830 con 0x7fa33c06dfa0 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33c0a0dc0 con 0x7fa33c0a0810 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa341371640 10 monclient(hunting): authenticate will time out at 1063.426957s 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa33b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa33c06dfa0 legacy=0x7fa33c13cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa33affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa33c134930 0x7fa33c134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37206/0 (socket says 172.21.15.63:37206) 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa33a7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa33c0a0810 0x7fa33c139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55076/0 (socket says 172.21.15.63:55076) 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa33affd640 1 -- 172.21.15.63:0/302153729 learned_addr learned my addr 172.21.15.63:0/302153729 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1023585 0 0) 0x7fa33c135420 con 0x7fa33c134930 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:46.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 10 monclient(hunting): my global_id is 4821 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa324003530 con 0x7fa33c134930 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458289759 0 0) 0x7fa33c0a0dc0 con 0x7fa33c0a0810 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 10 monclient(hunting): my global_id is 4820 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.647+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa33c135420 con 0x7fa33c0a0810 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4111629756 0 0) 0x7fa324003530 con 0x7fa33c134930 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 >> v1:172.21.15.6:6790/0 conn(0x7fa33c0a0810 legacy=0x7fa33c139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 >> v1:172.21.15.63:6789/0 conn(0x7fa33c06dfa0 legacy=0x7fa33c13cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 1 monclient: found mon.noname-a 2024-09-16T05:36:46.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33c0a2500 con 0x7fa33c134930 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa32c001550 con 0x7fa33c134930 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: dump: 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.658 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 5 monclient: authenticate success, global_id 4821 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa32c001870 con 0x7fa33c134930 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 1 -- 172.21.15.63:0/302153729 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa32c001c40 con 0x7fa33c134930 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:46.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa339ffb640 10 monclient: dump: 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:46.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 4 set_mon_vals no callback set 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient: shutdown 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- 172.21.15.63:0/302153729 >> v1:172.21.15.6:6789/0 conn(0x7fa33c134930 legacy=0x7fa33c134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- 172.21.15.63:0/302153729 shutdown_connections 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- 172.21.15.63:0/302153729 >> 172.21.15.63:0/302153729 conn(0x7fa33c0a1e90 msgr2=0x7fa33c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- 172.21.15.63:0/302153729 shutdown_connections 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- 172.21.15.63:0/302153729 wait complete. 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient: build_initial_monmap 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient: monmap: 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:46.661 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 Processor -- start 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- start start 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient: init 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient(hunting): picked mon.noname-c con 0x7fa33c06dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient(hunting): picked mon.noname-a con 0x7fa33c0a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:46.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient(hunting): picked mon.noname-b con 0x7fa33c134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33c11f180 con 0x7fa33c0a0810 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33c11f400 con 0x7fa33c134930 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33c1208b0 con 0x7fa33c06dfa0 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa341371640 10 monclient(hunting): authenticate will time out at 1063.430397s 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa33b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa33c134930 legacy=0x7fa33c1cfd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa33affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa33c06dfa0 0x7fa33c1c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55088/0 (socket says 172.21.15.63:55088) 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa33a7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa33c0a0810 0x7fa33c1cc610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37222/0 (socket says 172.21.15.63:37222) 2024-09-16T05:36:46.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa33affd640 1 -- 172.21.15.63:0/4242244736 learned_addr learned my addr 172.21.15.63:0/4242244736 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:46.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950557097 0 0) 0x7fa33c11f180 con 0x7fa33c0a0810 2024-09-16T05:36:46.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:46.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:46.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:46.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): my global_id is 4824 2024-09-16T05:36:46.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa328003560 con 0x7fa33c0a0810 2024-09-16T05:36:46.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158512031 0 0) 0x7fa33c1208b0 con 0x7fa33c06dfa0 2024-09-16T05:36:46.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): my global_id is 4823 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa33c11f180 con 0x7fa33c06dfa0 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2693458403 0 0) 0x7fa328003560 con 0x7fa33c0a0810 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa33c1208b0 con 0x7fa33c0a0810 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa3300018c0 con 0x7fa33c0a0810 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): dump: 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.667 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1370241015 0 0) 0x7fa33c11f180 con 0x7fa33c06dfa0 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa328003560 con 0x7fa33c06dfa0 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa32c001510 con 0x7fa33c06dfa0 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:46.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.651+0000 7fa3237fe640 10 monclient(hunting): dump: 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 41919912 0 0) 0x7fa33c1208b0 con 0x7fa33c0a0810 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 >> v1:172.21.15.6:6790/0 conn(0x7fa33c06dfa0 legacy=0x7fa33c1c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 >> v1:172.21.15.63:6789/0 conn(0x7fa33c134930 legacy=0x7fa33c1cfd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 monclient: found mon.a 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33c0a4040 con 0x7fa33c0a0810 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 5 monclient: authenticate success, global_id 4824 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 10 monclient: _renew_subs 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:46.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa33c1202f0 con 0x7fa33c0a0810 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 10 monclient: _renew_subs 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa33c11dc10 con 0x7fa33c0a0810 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3300047f0 con 0x7fa33c0a0810 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa330004b40 con 0x7fa33c0a0810 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3397fa640 4 set_mon_vals no callback set 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 10 monclient: dump: 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:46.671 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31c005090 con 0x7fa33c0a0810 2024-09-16T05:36:46.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa33000f8e0 con 0x7fa33c0a0810 2024-09-16T05:36:46.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(59..59 src has 1..59) ==== 4377+0+0 (unknown 53190516 0 0) 0x7fa330044390 con 0x7fa33c0a0810 2024-09-16T05:36:46.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.655+0000 7fa33affd640 10 monclient: get_auth_request con 0x7fa328036aa0 auth_method 0 2024-09-16T05:36:46.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.667+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa3300152f0 con 0x7fa33c0a0810 2024-09-16T05:36:46.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.667+0000 7fa3237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:46.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.667+0000 7fa3237fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.827+0000 7fa341371640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.827+0000 7fa341371640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.827+0000 7fa341371640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:46.827+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) -- 0x7fa31c005380 con 0x7fa33c0a0810 2024-09-16T05:36:47.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.315+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 v60) ==== 110+0+0 (unknown 3960335042 0 0) 0x7fa330014bf0 con 0x7fa33c0a0810 2024-09-16T05:36:47.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.315+0000 7fa3237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.315+0000 7fa3237fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:47.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.415+0000 7fa341371640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.415+0000 7fa341371640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.415+0000 7fa341371640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:47.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.415+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) -- 0x7fa31c005ac0 con 0x7fa33c0a0810 2024-09-16T05:36:47.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.415+0000 7fa3237fe640 1 -- 172.21.15.63:0/4242244736 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v60) ==== 139+0+0 (unknown 4220189393 0 0) 0x7fa330018f40 con 0x7fa33c0a0810 2024-09-16T05:36:47.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.415+0000 7fa3237fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.415+0000 7fa3237fe640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-16T05:36:47.419 INFO:tasks.workunit.client.0.smithi063.stderr:already renamed to 'foo-asdf' 2024-09-16T05:36:47.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.419+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa328036aa0 legacy=0x7fa328038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:47.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.419+0000 7fa341371640 10 monclient: shutdown 2024-09-16T05:36:47.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.419+0000 7fa341371640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:47.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.419+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 >> v1:172.21.15.6:6789/0 conn(0x7fa33c0a0810 legacy=0x7fa33c1cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:47.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.419+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 shutdown_connections 2024-09-16T05:36:47.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.419+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 >> 172.21.15.63:0/4242244736 conn(0x7fa33c0a1e90 msgr2=0x7fa33c133de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:47.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.419+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 shutdown_connections 2024-09-16T05:36:47.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.419+0000 7fa341371640 1 -- 172.21.15.63:0/4242244736 wait complete. 2024-09-16T05:36:47.435 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-16T05:36:47.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.555+0000 7fd5889a1640 10 monclient: get_monmap_and_config 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.555+0000 7fd5889a1640 10 monclient: build_initial_monmap 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.555+0000 7fd5889a1640 10 monclient: monmap: 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:47.558542+0000 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:47.558542+0000 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:47.559 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:47.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.555+0000 7fd5889a1640 1 Processor -- start 2024-09-16T05:36:47.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.555+0000 7fd5889a1640 1 -- start start 2024-09-16T05:36:47.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.555+0000 7fd5889a1640 10 monclient: init 2024-09-16T05:36:47.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:47.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:47.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient(hunting): picked mon.noname-c con 0x7fd58409c300 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:47.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient(hunting): picked mon.noname-a con 0x7fd5840a09c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:47.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient(hunting): picked mon.noname-b con 0x7fd58409d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:47.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5840a0ed0 con 0x7fd5840a09c0 2024-09-16T05:36:47.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd58409d950 con 0x7fd58409c300 2024-09-16T05:36:47.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd58409ebd0 con 0x7fd58409d3a0 2024-09-16T05:36:47.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:47.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient(hunting): authenticate will time out at 1064.336339s 2024-09-16T05:36:47.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd58259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd58409c300 legacy=0x7fd58409c6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:47.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd582d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd58409d3a0 0x7fd58413c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55094/0 (socket says 172.21.15.63:55094) 2024-09-16T05:36:47.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd582d9d640 1 -- 172.21.15.63:0/1225563974 learned_addr learned my addr 172.21.15.63:0/1225563974 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:47.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd579d9b640 1 --1- 172.21.15.63:0/1225563974 >> v1:172.21.15.6:6789/0 conn(0x7fd5840a09c0 0x7fd584138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37224/0 (socket says 172.21.15.63:37224) 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2227264984 0 0) 0x7fd5840a0ed0 con 0x7fd5840a09c0 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient(hunting): my global_id is 4827 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd570003530 con 0x7fd5840a09c0 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1677018191 0 0) 0x7fd58409ebd0 con 0x7fd58409d3a0 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:47.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient(hunting): my global_id is 4826 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5840a0ed0 con 0x7fd58409d3a0 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2918630956 0 0) 0x7fd570003530 con 0x7fd5840a09c0 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 >> v1:172.21.15.6:6790/0 conn(0x7fd58409d3a0 legacy=0x7fd58413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 >> v1:172.21.15.63:6789/0 conn(0x7fd58409c300 legacy=0x7fd58409c6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 monclient: found mon.noname-a 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5840a2470 con 0x7fd5840a09c0 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: _finish_auth 0 2024-09-16T05:36:47.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:47.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:47.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd56c001620 con 0x7fd5840a09c0 2024-09-16T05:36:47.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:47.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: dump: 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 5 monclient: authenticate success, global_id 4827 2024-09-16T05:36:47.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd56c001880 con 0x7fd5840a09c0 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 1 -- 172.21.15.63:0/1225563974 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd56c001c90 con 0x7fd5840a09c0 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 4 set_mon_vals no callback set 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd581d9b640 10 monclient: dump: 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.570 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient: shutdown 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- 172.21.15.63:0/1225563974 >> v1:172.21.15.6:6789/0 conn(0x7fd5840a09c0 legacy=0x7fd584138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- 172.21.15.63:0/1225563974 shutdown_connections 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- 172.21.15.63:0/1225563974 >> 172.21.15.63:0/1225563974 conn(0x7fd5840a1e90 msgr2=0x7fd584075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- 172.21.15.63:0/1225563974 shutdown_connections 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- 172.21.15.63:0/1225563974 wait complete. 2024-09-16T05:36:47.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient: build_initial_monmap 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient: monmap: 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:47.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 Processor -- start 2024-09-16T05:36:47.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 1 -- start start 2024-09-16T05:36:47.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.559+0000 7fd5889a1640 10 monclient: init 2024-09-16T05:36:47.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:47.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:47.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient(hunting): picked mon.noname-b con 0x7fd58409c300 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:47.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient(hunting): picked mon.noname-c con 0x7fd58409d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:47.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient(hunting): picked mon.noname-a con 0x7fd584134000 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:47.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5841c9a70 con 0x7fd584134000 2024-09-16T05:36:47.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5840a3ee0 con 0x7fd58409c300 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5840a4430 con 0x7fd58409d3a0 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient(hunting): authenticate will time out at 1064.340394s 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd58259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd58409c300 legacy=0x7fd5841c8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd582d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd584134000 0x7fd5841d1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37226/0 (socket says 172.21.15.63:37226) 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd582d9d640 1 -- 172.21.15.63:0/3509686453 learned_addr learned my addr 172.21.15.63:0/3509686453 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1759841896 0 0) 0x7fd5841c9a70 con 0x7fd584134000 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): my global_id is 4830 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd564003530 con 0x7fd584134000 2024-09-16T05:36:47.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1541797134 0 0) 0x7fd5840a4430 con 0x7fd58409d3a0 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): my global_id is 4829 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5841c9a70 con 0x7fd58409d3a0 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3393585824 0 0) 0x7fd564003530 con 0x7fd584134000 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5840a4430 con 0x7fd584134000 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd57c001da0 con 0x7fd584134000 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:47.579 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3165694750 0 0) 0x7fd5841c9a70 con 0x7fd58409d3a0 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd564003530 con 0x7fd58409d3a0 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd56c001bd0 con 0x7fd58409d3a0 2024-09-16T05:36:47.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:47.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4176301105 0 0) 0x7fd5840a4430 con 0x7fd584134000 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 >> v1:172.21.15.6:6790/0 conn(0x7fd58409d3a0 legacy=0x7fd5841c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 >> v1:172.21.15.63:6789/0 conn(0x7fd58409c300 legacy=0x7fd5841c8c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 monclient: found mon.a 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58409c0f0 con 0x7fd584134000 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 5 monclient: authenticate success, global_id 4830 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient: _renew_subs 2024-09-16T05:36:47.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd584131590 con 0x7fd584134000 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient: _renew_subs 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5841d2920 con 0x7fd584134000 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd57c002050 con 0x7fd584134000 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd57c004fc0 con 0x7fd584134000 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd58159a640 4 set_mon_vals no callback set 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 10 monclient: dump: 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:47.583 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd568005090 con 0x7fd584134000 2024-09-16T05:36:47.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd57c00fd60 con 0x7fd584134000 2024-09-16T05:36:47.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(60..60 src has 1..60) ==== 4382+0+0 (unknown 1254701921 0 0) 0x7fd57c0448c0 con 0x7fd584134000 2024-09-16T05:36:47.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.563+0000 7fd58259c640 10 monclient: get_auth_request con 0x7fd5640369b0 auth_method 0 2024-09-16T05:36:47.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.579+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd57c015820 con 0x7fd584134000 2024-09-16T05:36:47.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.579+0000 7fd57b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:47.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.579+0000 7fd57b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:47.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.747+0000 7fd5889a1640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.747+0000 7fd5889a1640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.747+0000 7fd5889a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:47.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.747+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) -- 0x7fd568005380 con 0x7fd584134000 2024-09-16T05:36:47.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.747+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v60) ==== 139+0+0 (unknown 4220189393 0 0) 0x7fd57c015120 con 0x7fd584134000 2024-09-16T05:36:47.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.747+0000 7fd57b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.747+0000 7fd57b7fe640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo-asdf' 2024-09-16T05:36:47.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.851+0000 7fd5889a1640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.851+0000 7fd5889a1640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.851+0000 7fd5889a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:47.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.851+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) -- 0x7fd5680010a0 con 0x7fd584134000 2024-09-16T05:36:47.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.851+0000 7fd57b7fe640 1 -- 172.21.15.63:0/3509686453 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v60) ==== 139+0+0 (unknown 4220189393 0 0) 0x7fd57c019470 con 0x7fd584134000 2024-09-16T05:36:47.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.851+0000 7fd57b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T05:36:47.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.851+0000 7fd57b7fe640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-16T05:36:47.856 INFO:tasks.workunit.client.0.smithi063.stderr:already renamed to 'foo-asdf' 2024-09-16T05:36:47.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.855+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd5640369b0 legacy=0x7fd564038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:47.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.855+0000 7fd5889a1640 10 monclient: shutdown 2024-09-16T05:36:47.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.855+0000 7fd5889a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:47.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.855+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 >> v1:172.21.15.6:6789/0 conn(0x7fd584134000 legacy=0x7fd5841d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:47.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.855+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 shutdown_connections 2024-09-16T05:36:47.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.855+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 >> 172.21.15.63:0/3509686453 conn(0x7fd5840a1e90 msgr2=0x7fd58412c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:47.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.855+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 shutdown_connections 2024-09-16T05:36:47.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.855+0000 7fd5889a1640 1 -- 172.21.15.63:0/3509686453 wait complete. 2024-09-16T05:36:47.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rename bar bar-asdf 2024-09-16T05:36:47.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient: get_monmap_and_config 2024-09-16T05:36:47.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient: build_initial_monmap 2024-09-16T05:36:47.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient: monmap: 2024-09-16T05:36:47.997 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:47.997 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:47.998 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:47.997217+0000 2024-09-16T05:36:47.998 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:47.997217+0000 2024-09-16T05:36:47.998 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:47.998 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:47.998 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:47.998 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:47.998 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:47.998 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:47.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 1 Processor -- start 2024-09-16T05:36:47.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 1 -- start start 2024-09-16T05:36:47.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient: init 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient(hunting): picked mon.noname-a con 0x7f6758098490 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient(hunting): picked mon.noname-b con 0x7f6758099520 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient(hunting): picked mon.noname-c con 0x7f675813a040 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6758099900 con 0x7f6758098490 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675809eec0 con 0x7f675813a040 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6758098f80 con 0x7f6758099520 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675cedd640 10 monclient(hunting): authenticate will time out at 1064.774862s 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f6756d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f675813a040 legacy=0x7f675813c430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:48.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6758098490 0x7f6758098870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37232/0 (socket says 172.21.15.63:37232) 2024-09-16T05:36:48.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675659c640 1 -- 172.21.15.63:0/1521115864 learned_addr learned my addr 172.21.15.63:0/1521115864 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:48.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370297871 0 0) 0x7f6758098f80 con 0x7f6758099520 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 10 monclient(hunting): my global_id is 4832 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6740003530 con 0x7f6758099520 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 179607958 0 0) 0x7f6758099900 con 0x7f6758098490 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 10 monclient(hunting): my global_id is 4833 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.995+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6758098f80 con 0x7f6758098490 2024-09-16T05:36:48.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1435220845 0 0) 0x7f6740003530 con 0x7f6758099520 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 >> v1:172.21.15.63:6789/0 conn(0x7f675813a040 legacy=0x7f675813c430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 >> v1:172.21.15.6:6789/0 conn(0x7f6758098490 legacy=0x7f6758098870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 1 monclient: found mon.noname-b 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675809ebe0 con 0x7f6758099520 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: _finish_auth 0 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f674c001660 con 0x7f6758099520 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 5 monclient: authenticate success, global_id 4832 2024-09-16T05:36:48.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: dump: 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.008 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f674c001930 con 0x7f6758099520 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 4 set_mon_vals no callback set 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 1 -- 172.21.15.63:0/1521115864 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f674c001d00 con 0x7f6758099520 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675559a640 10 monclient: dump: 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.009 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient: shutdown 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- 172.21.15.63:0/1521115864 >> v1:172.21.15.6:6790/0 conn(0x7f6758099520 legacy=0x7f6758138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- 172.21.15.63:0/1521115864 shutdown_connections 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- 172.21.15.63:0/1521115864 >> 172.21.15.63:0/1521115864 conn(0x7f67580a2420 msgr2=0x7f67580a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- 172.21.15.63:0/1521115864 shutdown_connections 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- 172.21.15.63:0/1521115864 wait complete. 2024-09-16T05:36:48.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient: build_initial_monmap 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient: monmap: 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 Processor -- start 2024-09-16T05:36:48.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- start start 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient: init 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient(hunting): picked mon.noname-b con 0x7f6758098490 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient(hunting): picked mon.noname-a con 0x7f6758099520 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient(hunting): picked mon.noname-c con 0x7f675813a040 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67580728b0 con 0x7f6758099520 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6758073960 con 0x7f6758098490 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675808c670 con 0x7f675813a040 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675cedd640 10 monclient(hunting): authenticate will time out at 1064.778524s 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f675659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6758098490 legacy=0x7f6758071a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:48.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6755d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6758099520 0x7f67580721a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37244/0 (socket says 172.21.15.63:37244) 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6755d9b640 1 -- 172.21.15.63:0/5338012 learned_addr learned my addr 172.21.15.63:0/5338012 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096975934 0 0) 0x7f67580728b0 con 0x7f6758099520 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 10 monclient(hunting): my global_id is 4836 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6738003530 con 0x7f6758099520 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3467692255 0 0) 0x7f675808c670 con 0x7f675813a040 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 10 monclient(hunting): my global_id is 4835 2024-09-16T05:36:48.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67580728b0 con 0x7f675813a040 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:47.999+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1404612951 0 0) 0x7f6738003530 con 0x7f6758099520 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f675808c670 con 0x7f6758099520 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f674c001620 con 0x7f6758099520 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.014 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2483934845 0 0) 0x7f67580728b0 con 0x7f675813a040 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6738003530 con 0x7f675813a040 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6750004340 con 0x7f675813a040 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.015 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1888927939 0 0) 0x7f675808c670 con 0x7f6758099520 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 >> v1:172.21.15.6:6790/0 conn(0x7f675813a040 legacy=0x7f67581d3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 >> v1:172.21.15.63:6789/0 conn(0x7f6758098490 legacy=0x7f6758071a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 monclient: found mon.a 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:48.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675808d970 con 0x7f6758099520 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 5 monclient: authenticate success, global_id 4836 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 10 monclient: _renew_subs 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f675808dc50 con 0x7f6758099520 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 10 monclient: _renew_subs 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6758073710 con 0x7f6758099520 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f674c0018d0 con 0x7f6758099520 2024-09-16T05:36:48.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f674c003b90 con 0x7f6758099520 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 10 monclient: dump: 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.018 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.019 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6754d99640 4 set_mon_vals no callback set 2024-09-16T05:36:48.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f673c005090 con 0x7f6758099520 2024-09-16T05:36:48.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f674c003d30 con 0x7f6758099520 2024-09-16T05:36:48.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(60..60 src has 1..60) ==== 4382+0+0 (unknown 1254701921 0 0) 0x7f674c044430 con 0x7f6758099520 2024-09-16T05:36:48.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.003+0000 7f675659c640 10 monclient: get_auth_request con 0x7f6738036a00 auth_method 0 2024-09-16T05:36:48.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.015+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6758073710 con 0x7f6758099520 2024-09-16T05:36:48.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.015+0000 7f6746ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.015+0000 7f6746ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:48.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.183+0000 7f675cedd640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T05:36:48.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.183+0000 7f675cedd640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T05:36:48.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.183+0000 7f675cedd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.183+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) -- 0x7f673c005380 con 0x7f6758099520 2024-09-16T05:36:48.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.315+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 v61) ==== 110+0+0 (unknown 655476828 0 0) 0x7f674c015450 con 0x7f6758099520 2024-09-16T05:36:48.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.315+0000 7f6746ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T05:36:48.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.315+0000 7f6746ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:48.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T05:36:48.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T05:36:48.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) -- 0x7f673c005ac0 con 0x7f6758099520 2024-09-16T05:36:48.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f6746ffd640 1 -- 172.21.15.63:0/5338012 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 already renamed to 'bar-asdf' v61) ==== 139+0+0 (unknown 3268292670 0 0) 0x7f674c01d900 con 0x7f6758099520 2024-09-16T05:36:48.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f6746ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T05:36:48.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f6746ffd640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar-asdf' 2024-09-16T05:36:48.422 INFO:tasks.workunit.client.0.smithi063.stderr:already renamed to 'bar-asdf' 2024-09-16T05:36:48.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6738036a00 legacy=0x7f6738038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 10 monclient: shutdown 2024-09-16T05:36:48.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:48.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 >> v1:172.21.15.6:6789/0 conn(0x7f6758099520 legacy=0x7f67580721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 shutdown_connections 2024-09-16T05:36:48.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 >> 172.21.15.63:0/5338012 conn(0x7f67580a2420 msgr2=0x7f67580a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:48.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 shutdown_connections 2024-09-16T05:36:48.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.419+0000 7f675cedd640 1 -- 172.21.15.63:0/5338012 wait complete. 2024-09-16T05:36:48.439 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls 2024-09-16T05:36:48.439 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep foo-asdf 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient: get_monmap_and_config 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient: build_initial_monmap 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient: monmap: 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:48.553856+0000 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:48.553856+0000 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.554 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:48.555 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:48.555 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:48.555 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 1 Processor -- start 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 1 -- start start 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient: init 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient(hunting): picked mon.noname-a con 0x7f771009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient(hunting): picked mon.noname-b con 0x7f771006dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient(hunting): picked mon.noname-c con 0x7f77100a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f771009f800 con 0x7f771009ac90 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77100a0ab0 con 0x7f77100a0210 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77100a0d90 con 0x7f771006dd30 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7717db5640 10 monclient(hunting): authenticate will time out at 1065.330776s 2024-09-16T05:36:48.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7716352640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f77100a0210 legacy=0x7f771013d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7715b51640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f771009ac90 0x7f771009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37258/0 (socket says 172.21.15.63:37258) 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7715b51640 1 -- 172.21.15.63:0/2194197068 learned_addr learned my addr 172.21.15.63:0/2194197068 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897428650 0 0) 0x7f771009f800 con 0x7f771009ac90 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 10 monclient(hunting): my global_id is 4839 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76fc003530 con 0x7f771009ac90 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1227057875 0 0) 0x7f77100a0d90 con 0x7f771006dd30 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 10 monclient(hunting): my global_id is 4838 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.551+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f771009f800 con 0x7f771006dd30 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2490132546 0 0) 0x7f771009f800 con 0x7f771006dd30 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 >> v1:172.21.15.63:6789/0 conn(0x7f77100a0210 legacy=0x7f771013d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 >> v1:172.21.15.6:6789/0 conn(0x7f771009ac90 legacy=0x7f771009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: _finish_hunting 0 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 1 monclient: found mon.noname-b 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771009e770 con 0x7f771006dd30 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: _finish_auth 0 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: _check_auth_tickets 2024-09-16T05:36:48.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 5 monclient: authenticate success, global_id 4838 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f77000014b0 con 0x7f771006dd30 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: dump: 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.561 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7700001790 con 0x7f771006dd30 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 1 -- 172.21.15.63:0/2194197068 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7700001ba0 con 0x7f771006dd30 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7714b4f640 10 monclient: dump: 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.562 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 4 set_mon_vals no callback set 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient: shutdown 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- 172.21.15.63:0/2194197068 >> v1:172.21.15.6:6790/0 conn(0x7f771006dd30 legacy=0x7f77101399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- 172.21.15.63:0/2194197068 shutdown_connections 2024-09-16T05:36:48.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- 172.21.15.63:0/2194197068 >> 172.21.15.63:0/2194197068 conn(0x7f77100a1c90 msgr2=0x7f77100a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:48.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- 172.21.15.63:0/2194197068 shutdown_connections 2024-09-16T05:36:48.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- 172.21.15.63:0/2194197068 wait complete. 2024-09-16T05:36:48.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient: build_initial_monmap 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient: monmap: 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:48.568 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 Processor -- start 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- start start 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient: init 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient(hunting): picked mon.noname-c con 0x7f771006dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient(hunting): picked mon.noname-b con 0x7f771009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient(hunting): picked mon.noname-a con 0x7f77100a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7710121790 con 0x7f77100a0210 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f771009c0f0 con 0x7f771009ac90 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7715350640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f771009ac90 legacy=0x7f77101d0010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77101bf420 con 0x7f771006dd30 2024-09-16T05:36:48.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7717db5640 10 monclient(hunting): authenticate will time out at 1065.334691s 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7716352640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f77100a0210 0x7f77101d3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37264/0 (socket says 172.21.15.63:37264) 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f7716352640 1 -- 172.21.15.63:0/507450348 learned_addr learned my addr 172.21.15.63:0/507450348 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3591560479 0 0) 0x7f77101bf420 con 0x7f771006dd30 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 10 monclient(hunting): my global_id is 4841 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76f4003530 con 0x7f771006dd30 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 971165822 0 0) 0x7f7710121790 con 0x7f77100a0210 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 10 monclient(hunting): my global_id is 4842 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.555+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77101bf420 con 0x7f77100a0210 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1222073258 0 0) 0x7f77101bf420 con 0x7f77100a0210 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7710121790 con 0x7f77100a0210 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7708004340 con 0x7f77100a0210 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient(hunting): dump: 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.571 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4178932917 0 0) 0x7f76f4003530 con 0x7f771006dd30 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77101bf420 con 0x7f771006dd30 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f770c001220 con 0x7f771006dd30 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient(hunting): dump: 2024-09-16T05:36:48.572 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2610463531 0 0) 0x7f7710121790 con 0x7f77100a0210 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 >> v1:172.21.15.6:6790/0 conn(0x7f771006dd30 legacy=0x7f7710133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 >> v1:172.21.15.63:6789/0 conn(0x7f771009ac90 legacy=0x7f77101d0010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:48.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 monclient: found mon.a 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77101214b0 con 0x7f77100a0210 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 5 monclient: authenticate success, global_id 4842 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 10 monclient: _renew_subs 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f771011fd80 con 0x7f77100a0210 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 10 monclient: _renew_subs 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f77101344e0 con 0x7f77100a0210 2024-09-16T05:36:48.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7708004850 con 0x7f77100a0210 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7708004ba0 con 0x7f77100a0210 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7707fff640 4 set_mon_vals no callback set 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 10 monclient: dump: 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.575 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f8005090 con 0x7f77100a0210 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7708004d40 con 0x7f77100a0210 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7f77080445c0 con 0x7f77100a0210 2024-09-16T05:36:48.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.559+0000 7f7715b51640 10 monclient: get_auth_request con 0x7f76f4036a70 auth_method 0 2024-09-16T05:36:48.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.571+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7708019160 con 0x7f77100a0210 2024-09-16T05:36:48.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.571+0000 7f77067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.571+0000 7f77067fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:48.609 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:36:48.609 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:36:48.609 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:36:48.610 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:36:48.613 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:36:48.613 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:36:48.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:36:48.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.739+0000 7f7717db5640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:48.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.739+0000 7f7717db5640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:48.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.739+0000 7f7717db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.739+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f76f8005380 con 0x7f77100a0210 2024-09-16T05:36:48.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.743+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v61) ==== 65+0+70 (unknown 231978741 0 3909614116) 0x7f770801da20 con 0x7f77100a0210 2024-09-16T05:36:48.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.743+0000 7f77067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:48.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.743+0000 7f77067fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.746+0000 7fdcbd4c2640 10 monclient: get_monmap_and_config 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.746+0000 7fdcbd4c2640 10 monclient: build_initial_monmap 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.746+0000 7fdcbd4c2640 10 monclient: monmap: 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:48.749192+0000 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:48.749192+0000 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:48.749 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:48.750 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:48.750 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:48.750 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:48.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.746+0000 7fdcbd4c2640 1 Processor -- start 2024-09-16T05:36:48.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.746+0000 7fdcbd4c2640 1 -- start start 2024-09-16T05:36:48.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.746+0000 7fdcbd4c2640 10 monclient: init 2024-09-16T05:36:48.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:48.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:48.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): picked mon.noname-b con 0x7fdcb806bc10 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:48.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): picked mon.noname-c con 0x7fdcb806bff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): picked mon.noname-a con 0x7fdcb813cb10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb809fe30 con 0x7fdcb813cb10 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb806ef70 con 0x7fdcb806bff0 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb80a16f0 con 0x7fdcb806bc10 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): authenticate will time out at 1070.556623s 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb6ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdcb806bc10 0x7fdcb8136a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53364/0 (socket says 172.21.15.6:53364) 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb6ffd640 1 -- 172.21.15.6:0/3311140693 learned_addr learned my addr 172.21.15.6:0/3311140693 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1836319479 0 0) 0x7fdcb809fe30 con 0x7fdcb813cb10 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient(hunting): my global_id is 4845 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdca0003530 con 0x7fdcb813cb10 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2409215064 0 0) 0x7fdca0003530 con 0x7fdcb813cb10 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 >> v1:172.21.15.6:6790/0 conn(0x7fdcb806bc10 legacy=0x7fdcb8136a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 >> v1:172.21.15.63:6789/0 conn(0x7fdcb806bff0 legacy=0x7fdcb806c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 monclient: found mon.noname-a 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb8143650 con 0x7fdcb813cb10 2024-09-16T05:36:48.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdcb00019a0 con 0x7fdcb813cb10 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 5 monclient: authenticate success, global_id 4845 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: dump: 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.756 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdcb0001c90 con 0x7fdcb813cb10 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 1 -- 172.21.15.6:0/3311140693 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdcb0002060 con 0x7fdcb813cb10 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:48.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 4 set_mon_vals no callback set 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb5ffb640 10 monclient: dump: 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.758 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: shutdown 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/3311140693 >> v1:172.21.15.6:6789/0 conn(0x7fdcb813cb10 legacy=0x7fdcb813ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/3311140693 shutdown_connections 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/3311140693 >> 172.21.15.6:0/3311140693 conn(0x7fdcb80a2580 msgr2=0x7fdcb80a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/3311140693 shutdown_connections 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/3311140693 wait complete. 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: build_initial_monmap 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: monmap: 2024-09-16T05:36:48.759 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:48.760 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:48.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 Processor -- start 2024-09-16T05:36:48.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- start start 2024-09-16T05:36:48.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: init 2024-09-16T05:36:48.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:48.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:48.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): picked mon.noname-c con 0x7fdcb806bc10 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): picked mon.noname-b con 0x7fdcb806bff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): picked mon.noname-a con 0x7fdcb813cb10 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb80707c0 con 0x7fdcb813cb10 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb8070aa0 con 0x7fdcb806bff0 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb8140370 con 0x7fdcb806bc10 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcbd4c2640 10 monclient(hunting): authenticate will time out at 1070.559880s 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb6ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdcb806bc10 0x7fdcb81cb830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53380/0 (socket says 172.21.15.6:53380) 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.750+0000 7fdcb6ffd640 1 -- 172.21.15.6:0/783730926 learned_addr learned my addr 172.21.15.6:0/783730926 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1933334587 0 0) 0x7fdcb80707c0 con 0x7fdcb813cb10 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient(hunting): my global_id is 4848 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdca4003530 con 0x7fdcb813cb10 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2239822521 0 0) 0x7fdca4003530 con 0x7fdcb813cb10 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdcb80707c0 con 0x7fdcb813cb10 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdcb00017c0 con 0x7fdcb813cb10 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:48.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 457850470 0 0) 0x7fdcb80707c0 con 0x7fdcb813cb10 2024-09-16T05:36:48.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 >> v1:172.21.15.6:6790/0 conn(0x7fdcb806bc10 legacy=0x7fdcb81cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 >> v1:172.21.15.63:6789/0 conn(0x7fdcb806bff0 legacy=0x7fdcb81d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 monclient: found mon.a 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb813ff20 con 0x7fdcb813cb10 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 5 monclient: authenticate success, global_id 4848 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 10 monclient: _renew_subs 2024-09-16T05:36:48.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb80704e0 con 0x7fdcb813cb10 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 10 monclient: _renew_subs 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdcb8139c30 con 0x7fdcb813cb10 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdcb0001c90 con 0x7fdcb813cb10 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdcb0003f10 con 0x7fdcb813cb10 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcb4ff9640 4 set_mon_vals no callback set 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 10 monclient: dump: 2024-09-16T05:36:48.768 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdcb000ecb0 con 0x7fdcb813cb10 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7fdcb0043690 con 0x7fdcb813cb10 2024-09-16T05:36:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc98005090 con 0x7fdcb813cb10 2024-09-16T05:36:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.754+0000 7fdcb6ffd640 10 monclient: get_auth_request con 0x7fdca40369e0 auth_method 0 2024-09-16T05:36:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.758+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fdcb0014640 con 0x7fdcb813cb10 2024-09-16T05:36:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.758+0000 7fdc9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.758+0000 7fdc9f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:48.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.839+0000 7f7717db5640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:48.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.839+0000 7f7717db5640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:48.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.839+0000 7f7717db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.839+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f76f8005ac0 con 0x7f77100a0210 2024-09-16T05:36:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.839+0000 7f77067fc640 1 -- 172.21.15.63:0/507450348 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v61) ==== 65+0+70 (unknown 231978741 0 3909614116) 0x7f77080405a0 con 0x7f77100a0210 2024-09-16T05:36:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.839+0000 7f77067fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.839+0000 7f77067fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:48.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.843+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 >> v1:172.21.15.63:6812/3643207211 conn(0x7f76f4036a70 legacy=0x7f76f4038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.843+0000 7f7717db5640 10 monclient: shutdown 2024-09-16T05:36:48.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.843+0000 7f7717db5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:48.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.843+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 >> v1:172.21.15.6:6789/0 conn(0x7f77100a0210 legacy=0x7f77101d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.843+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 shutdown_connections 2024-09-16T05:36:48.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.843+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 >> 172.21.15.63:0/507450348 conn(0x7f77100a1c90 msgr2=0x7f77100a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:48.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.843+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 shutdown_connections 2024-09-16T05:36:48.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.843+0000 7f7717db5640 1 -- 172.21.15.63:0/507450348 wait complete. 2024-09-16T05:36:48.858 INFO:tasks.workunit.client.0.smithi063.stdout:foo-asdf 2024-09-16T05:36:48.859 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls 2024-09-16T05:36:48.859 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep bar-asdf 2024-09-16T05:36:48.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient: get_monmap_and_config 2024-09-16T05:36:48.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient: build_initial_monmap 2024-09-16T05:36:48.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient: monmap: 2024-09-16T05:36:48.966 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:48.966 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.967 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:48.966150+0000 2024-09-16T05:36:48.967 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:48.966150+0000 2024-09-16T05:36:48.967 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:48.967 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.967 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:48.967 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:48.967 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:48.967 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 1 Processor -- start 2024-09-16T05:36:48.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 1 -- start start 2024-09-16T05:36:48.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient: init 2024-09-16T05:36:48.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:48.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:48.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b80a12b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:48.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient(hunting): picked mon.noname-a con 0x7fb9b8072fa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b81395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b80a05b0 con 0x7fb9b8072fa0 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b809c510 con 0x7fb9b81395a0 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b809ce10 con 0x7fb9b80a12b0 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9bcc7b640 10 monclient(hunting): authenticate will time out at 1065.742887s 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9b6d9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb9b81395a0 0x7fb9b813b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47250/0 (socket says 172.21.15.63:47250) 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9b6d9d640 1 -- 172.21.15.63:0/2048416772 learned_addr learned my addr 172.21.15.63:0/2048416772 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723925008 0 0) 0x7fb9b809ce10 con 0x7fb9b80a12b0 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9b559a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9b559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9b559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9b559a640 10 monclient(hunting): my global_id is 4844 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.963+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9a4003450 con 0x7fb9b80a12b0 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1739994710 0 0) 0x7fb9a4003450 con 0x7fb9b80a12b0 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 >> v1:172.21.15.63:6789/0 conn(0x7fb9b81395a0 legacy=0x7fb9b813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 >> v1:172.21.15.6:6789/0 conn(0x7fb9b8072fa0 legacy=0x7fb9b809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 1 monclient: found mon.noname-b 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b809d0f0 con 0x7fb9b80a12b0 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: _finish_auth 0 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:48.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb9a80018d0 con 0x7fb9b80a12b0 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: dump: 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.972 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 5 monclient: authenticate success, global_id 4844 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9a8001b30 con 0x7fb9b80a12b0 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 1 -- 172.21.15.63:0/2048416772 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb9a8001f00 con 0x7fb9b80a12b0 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b559a640 10 monclient: dump: 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.973 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.974 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.974 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.974 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.974 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.974 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:48.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 4 set_mon_vals no callback set 2024-09-16T05:36:48.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient: shutdown 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/2048416772 >> v1:172.21.15.6:6790/0 conn(0x7fb9b80a12b0 legacy=0x7fb9b809f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/2048416772 shutdown_connections 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/2048416772 >> 172.21.15.63:0/2048416772 conn(0x7fb9b80a1c90 msgr2=0x7fb9b80a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/2048416772 shutdown_connections 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/2048416772 wait complete. 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient: build_initial_monmap 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient: monmap: 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 Processor -- start 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- start start 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient: init 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient(hunting): picked mon.noname-a con 0x7fb9b8072fa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient(hunting): picked mon.noname-c con 0x7fb9b80a12b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient(hunting): picked mon.noname-b con 0x7fb9b81395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b80a3e20 con 0x7fb9b8072fa0 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b811c480 con 0x7fb9b81395a0 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b811e700 con 0x7fb9b80a12b0 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9bcc7b640 10 monclient(hunting): authenticate will time out at 1065.746872s 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b6d9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb9b81395a0 0x7fb9b81cf7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47260/0 (socket says 172.21.15.63:47260) 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9b6d9d640 1 -- 172.21.15.63:0/3715047832 learned_addr learned my addr 172.21.15.63:0/3715047832 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706177644 0 0) 0x7fb9b80a3e20 con 0x7fb9b8072fa0 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9aeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:48.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9aeffd640 10 monclient(hunting): my global_id is 4851 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.967+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb998003530 con 0x7fb9b8072fa0 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3243969425 0 0) 0x7fb998003530 con 0x7fb9b8072fa0 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9b80a3e20 con 0x7fb9b8072fa0 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb9a8001810 con 0x7fb9b8072fa0 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 10 monclient(hunting): dump: 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.981 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2947342649 0 0) 0x7fb9b80a3e20 con 0x7fb9b8072fa0 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 >> v1:172.21.15.6:6790/0 conn(0x7fb9b80a12b0 legacy=0x7fb9b81cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 >> v1:172.21.15.63:6789/0 conn(0x7fb9b81395a0 legacy=0x7fb9b81cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 1 monclient: found mon.a 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:48.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b811de60 con 0x7fb9b8072fa0 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 5 monclient: authenticate success, global_id 4851 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 10 monclient: _renew_subs 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b809c0f0 con 0x7fb9b8072fa0 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 10 monclient: _renew_subs 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb9b8134480 con 0x7fb9b8072fa0 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.971+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b811e260 con 0x7fb9b8072fa0 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9a8001a70 con 0x7fb9b8072fa0 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb9a8003dd0 con 0x7fb9b8072fa0 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9aeffd640 10 monclient: dump: 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:48.984 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9b4d99640 4 set_mon_vals no callback set 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb9a8003f70 con 0x7fb9b8072fa0 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7fb9a8043560 con 0x7fb9b8072fa0 2024-09-16T05:36:48.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.975+0000 7fb9b5d9b640 10 monclient: get_auth_request con 0x7fb998036b00 auth_method 0 2024-09-16T05:36:48.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.986+0000 7fdcbd4c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:36:48.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.986+0000 7fdcbd4c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:36:48.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.986+0000 7fdcbd4c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:48.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.986+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdc98005380 con 0x7fdcb813cb10 2024-09-16T05:36:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.987+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fb9a80144b0 con 0x7fb9b8072fa0 2024-09-16T05:36:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.987+0000 7fb9aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:48.987+0000 7fb9aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:48.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdc9f7fe640 1 -- 172.21.15.6:0/783730926 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2630803666) 0x7fdcb0013f40 con 0x7fdcb813cb10 2024-09-16T05:36:48.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdc9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:36:48.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdc9f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:48.992 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:36:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 >> v1:172.21.15.63:6812/3643207211 conn(0x7fdca40369e0 legacy=0x7fdca4038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdcbd4c2640 10 monclient: shutdown 2024-09-16T05:36:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdcbd4c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 >> v1:172.21.15.6:6789/0 conn(0x7fdcb813cb10 legacy=0x7fdcb81d3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 shutdown_connections 2024-09-16T05:36:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 >> 172.21.15.6:0/783730926 conn(0x7fdcb80a2580 msgr2=0x7fdcb81357f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 shutdown_connections 2024-09-16T05:36:48.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:48.990+0000 7fdcbd4c2640 1 -- 172.21.15.6:0/783730926 wait complete. 2024-09-16T05:36:49.010 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:36:49.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:36:49.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:36:49.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient: get_monmap_and_config 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient: build_initial_monmap 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient: monmap: 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:49.135684+0000 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:49.135684+0000 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:49.136 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:49.137 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:49.137 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 1 Processor -- start 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 1 -- start start 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient: init 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient(hunting): picked mon.noname-b con 0x7f86dc09bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient(hunting): picked mon.noname-c con 0x7f86dc09bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient(hunting): picked mon.noname-a con 0x7f86dc1397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86dc0a1350 con 0x7f86dc1397f0 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86dc0a10d0 con 0x7f86dc09bf20 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86dc09d940 con 0x7f86dc09bb40 2024-09-16T05:36:49.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient(hunting): authenticate will time out at 1070.942817s 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86db7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f86dc09bb40 0x7f86dc09e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53388/0 (socket says 172.21.15.6:53388) 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86db7fe640 1 -- 172.21.15.6:0/1018539520 learned_addr learned my addr 172.21.15.6:0/1018539520 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86dbfff640 1 --1- 172.21.15.6:0/1018539520 >> v1:172.21.15.6:6789/0 conn(0x7f86dc1397f0 0x7f86dc13bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52386/0 (socket says 172.21.15.6:52386) 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410827140 0 0) 0x7f86dc09d940 con 0x7f86dc09bb40 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient(hunting): my global_id is 4847 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86c4003530 con 0x7f86dc09bb40 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054103997 0 0) 0x7f86dc0a1350 con 0x7f86dc1397f0 2024-09-16T05:36:49.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient(hunting): my global_id is 4854 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86dc09d940 con 0x7f86dc1397f0 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4239589182 0 0) 0x7f86dc09d940 con 0x7f86dc1397f0 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 >> v1:172.21.15.6:6790/0 conn(0x7f86dc09bb40 legacy=0x7f86dc09e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 >> v1:172.21.15.63:6789/0 conn(0x7f86dc09bf20 legacy=0x7f86dc09c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 monclient: found mon.noname-a 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:49.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86dc09dc20 con 0x7f86dc1397f0 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86d4001600 con 0x7f86dc1397f0 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 5 monclient: authenticate success, global_id 4854 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: dump: 2024-09-16T05:36:49.143 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86d4001870 con 0x7f86dc1397f0 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:49.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 1 -- 172.21.15.6:0/1018539520 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86d4001c80 con 0x7f86dc1397f0 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86e203c640 4 set_mon_vals no callback set 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.134+0000 7f86da7fc640 10 monclient: dump: 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: shutdown 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- 172.21.15.6:0/1018539520 >> v1:172.21.15.6:6789/0 conn(0x7f86dc1397f0 legacy=0x7f86dc13bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- 172.21.15.6:0/1018539520 shutdown_connections 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- 172.21.15.6:0/1018539520 >> 172.21.15.6:0/1018539520 conn(0x7f86dc0a1650 msgr2=0x7f86dc0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- 172.21.15.6:0/1018539520 shutdown_connections 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- 172.21.15.6:0/1018539520 wait complete. 2024-09-16T05:36:49.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: build_initial_monmap 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: monmap: 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:49.147 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 Processor -- start 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- start start 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: init 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient(hunting): picked mon.noname-b con 0x7f86dc09bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient(hunting): picked mon.noname-a con 0x7f86dc09bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient(hunting): picked mon.noname-c con 0x7f86dc1397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86dc11e960 con 0x7f86dc09bf20 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86dc11bfc0 con 0x7f86dc09bb40 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86dc11c670 con 0x7f86dc1397f0 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient(hunting): authenticate will time out at 1070.946081s 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86daffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86dc09bf20 0x7f86dc11b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52396/0 (socket says 172.21.15.6:52396) 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86daffd640 1 -- 172.21.15.6:0/4007882976 learned_addr learned my addr 172.21.15.6:0/4007882976 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2562841022 0 0) 0x7f86dc11e960 con 0x7f86dc09bf20 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4857 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86c8003530 con 0x7f86dc09bf20 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221143195 0 0) 0x7f86dc11c670 con 0x7f86dc1397f0 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4850 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86dc11e960 con 0x7f86dc1397f0 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806386994 0 0) 0x7f86dc11bfc0 con 0x7f86dc09bb40 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): my global_id is 94102 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86dc11c670 con 0x7f86dc09bb40 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1297073907 0 0) 0x7f86dc11e960 con 0x7f86dc1397f0 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86dc11bfc0 con 0x7f86dc1397f0 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86d4001bc0 con 0x7f86dc1397f0 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): dump: 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1705569032 0 0) 0x7f86c8003530 con 0x7f86dc09bf20 2024-09-16T05:36:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86dc11e960 con 0x7f86dc09bf20 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86d00018a0 con 0x7f86dc09bf20 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient(hunting): dump: 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 301922467 0 0) 0x7f86dc11bfc0 con 0x7f86dc1397f0 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 >> v1:172.21.15.63:6789/0 conn(0x7f86dc09bb40 legacy=0x7f86dc11b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 >> v1:172.21.15.6:6789/0 conn(0x7f86dc09bf20 legacy=0x7f86dc11b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 monclient: found mon.c 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86dc11e680 con 0x7f86dc1397f0 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient: _finish_auth 0 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 5 monclient: authenticate success, global_id 4850 2024-09-16T05:36:49.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: _renew_subs 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86dc072810 con 0x7f86dc1397f0 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: _renew_subs 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f86dc1d4ea0 con 0x7f86dc1397f0 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86d4001870 con 0x7f86dc1397f0 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86d4004ce0 con 0x7f86dc1397f0 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86d9ffb640 4 set_mon_vals no callback set 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86c3fff640 10 monclient: dump: 2024-09-16T05:36:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.138+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86bc0065d0 con 0x7f86bc002870 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f86d4004e80 con 0x7f86dc1397f0 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7f86d40444d0 con 0x7f86dc1397f0 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628778950 0 0) 0x7f86bc0065d0 con 0x7f86bc002870 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4860 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86dc11bfc0 con 0x7f86bc002870 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86daffd640 10 monclient: get_auth_request con 0x7f86c8036b40 auth_method 0 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3886716320 0 0) 0x7f86dc11bfc0 con 0x7f86bc002870 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86bc0065d0 con 0x7f86bc002870 2024-09-16T05:36:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86cc001430 con 0x7f86bc002870 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient: dump: 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.162 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3364243104 0 0) 0x7f86bc0065d0 con 0x7f86bc002870 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86bc00a160 con 0x7f86bc002870 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f86cc004b70 con 0x7f86bc002870 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 >> v1:172.21.15.6:6789/0 conn(0x7f86bc002870 legacy=0x7f86bc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.142+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 >> v1:172.21.15.6:6789/0 conn(0x7f86bc002870 legacy=0x7f86bc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.146+0000 7f86e203c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.146+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86bc009e50 con 0x7f86bc009110 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.146+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417892322 0 0) 0x7f86bc009e50 con 0x7f86bc009110 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.146+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.146+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.146+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.146+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4863 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.146+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86bc0065d0 con 0x7f86bc009110 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3839744771 0 0) 0x7f86bc0065d0 con 0x7f86bc009110 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86bc009e50 con 0x7f86bc009110 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86cc004d20 con 0x7f86bc009110 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 10 monclient: dump: 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3267916395 0 0) 0x7f86bc009e50 con 0x7f86bc009110 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86bc00c3e0 con 0x7f86bc009110 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 1421473068) 0x7f86cc0060e0 con 0x7f86bc009110 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 >> v1:172.21.15.6:6789/0 conn(0x7f86bc009110 legacy=0x7f86bc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86c3fff640 1 -- 172.21.15.6:0/4007882976 >> v1:172.21.15.6:6789/0 conn(0x7f86bc009110 legacy=0x7f86bc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:49.166 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:36:49.166 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:36:49.166 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:36:49.166 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:36:49.166 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 14, 2024-09-16T05:36:49.166 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 138206, 2024-09-16T05:36:49.166 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.167 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.168 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:36:49.169 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:36:49.170 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:36:49.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.173 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.174 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.175 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.176 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.177 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:36:49.178 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:36:49.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.155+0000 7fb9bcc7b640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:49.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.155+0000 7fb9bcc7b640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:49.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.155+0000 7fb9bcc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:49.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.155+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fb9b8138f30 con 0x7fb9b8072fa0 2024-09-16T05:36:49.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.155+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v61) ==== 65+0+70 (unknown 231978741 0 3909614116) 0x7fb9a801c9c0 con 0x7fb9b8072fa0 2024-09-16T05:36:49.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.155+0000 7fb9aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:49.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.155+0000 7fb9aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:49.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 >> v1:172.21.15.63:6812/3643207211 conn(0x7f86c8036b40 legacy=0x7f86c8039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86e203c640 10 monclient: shutdown 2024-09-16T05:36:49.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86e203c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 >> v1:172.21.15.6:6790/0 conn(0x7f86dc1397f0 legacy=0x7f86dc1d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86dbfff640 1 -- 172.21.15.6:0/4007882976 reap_dead start 2024-09-16T05:36:49.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 shutdown_connections 2024-09-16T05:36:49.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 >> 172.21.15.6:0/4007882976 conn(0x7f86dc0a1650 msgr2=0x7f86dc13c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 shutdown_connections 2024-09-16T05:36:49.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.150+0000 7f86e203c640 1 -- 172.21.15.6:0/4007882976 wait complete. 2024-09-16T05:36:49.181 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:36:49.181 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:36:49.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:36:49.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:49.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:49.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:49.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fb9b80008e0 con 0x7fb9b8072fa0 2024-09-16T05:36:49.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9aeffd640 1 -- 172.21.15.63:0/3715047832 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v61) ==== 65+0+70 (unknown 231978741 0 3909614116) 0x7fb9a803f780 con 0x7fb9b8072fa0 2024-09-16T05:36:49.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9aeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:36:49.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9aeffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:36:49.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb998036b00 legacy=0x7fb998038fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 10 monclient: shutdown 2024-09-16T05:36:49.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 >> v1:172.21.15.6:6789/0 conn(0x7fb9b8072fa0 legacy=0x7fb9b8133d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 shutdown_connections 2024-09-16T05:36:49.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 >> 172.21.15.63:0/3715047832 conn(0x7fb9b80a1c90 msgr2=0x7fb9b8134f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 shutdown_connections 2024-09-16T05:36:49.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:49.251+0000 7fb9bcc7b640 1 -- 172.21.15.63:0/3715047832 wait complete. 2024-09-16T05:36:49.268 INFO:tasks.workunit.client.0.smithi063.stdout:bar-asdf 2024-09-16T05:36:49.269 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rm foo 2024-09-16T05:36:49.269 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'does not exist' 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient: get_monmap_and_config 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient: build_initial_monmap 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient: monmap: 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:49.296022+0000 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:49.296022+0000 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:49.296 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.297 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:49.297 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:49.297 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:49.297 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 1 Processor -- start 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 1 -- start start 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient: init 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient(hunting): picked mon.noname-a con 0x7f42ac09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient(hunting): picked mon.noname-c con 0x7f42ac09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient(hunting): picked mon.noname-b con 0x7f42ac0a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:49.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac0a1370 con 0x7f42ac09aa30 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac09ead0 con 0x7f42ac09e230 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac09edb0 con 0x7f42ac0a0da0 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient(hunting): authenticate will time out at 1071.102661s 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b141c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f42ac09aa30 0x7f42ac09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59274/0 (socket says 172.21.15.6:59274) 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b141c640 1 -- 172.21.15.6:0/4011807568 learned_addr learned my addr 172.21.15.6:0/4011807568 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405511991 0 0) 0x7f42ac09edb0 con 0x7f42ac0a0da0 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): my global_id is 4853 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4298003530 con 0x7f42ac0a0da0 2024-09-16T05:36:49.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1074096459 0 0) 0x7f42ac0a1370 con 0x7f42ac09aa30 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): my global_id is 4866 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ac09edb0 con 0x7f42ac09aa30 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330757268 0 0) 0x7f42ac09ead0 con 0x7f42ac09e230 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient(hunting): my global_id is 94105 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ac0a1370 con 0x7f42ac09e230 2024-09-16T05:36:49.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4040414688 0 0) 0x7f4298003530 con 0x7f42ac0a0da0 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 >> v1:172.21.15.63:6789/0 conn(0x7f42ac09e230 legacy=0x7f42ac1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 >> v1:172.21.15.6:6789/0 conn(0x7f42ac09aa30 legacy=0x7f42ac09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 monclient: found mon.noname-b 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac09dd50 con 0x7f42ac0a0da0 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: _finish_auth 0 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a4001620 con 0x7f42ac0a0da0 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: dump: 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 5 monclient: authenticate success, global_id 4853 2024-09-16T05:36:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42a4001880 con 0x7f42ac0a0da0 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 1 -- 172.21.15.6:0/4011807568 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a4001c10 con 0x7f42ac0a0da0 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f429ffff640 10 monclient: dump: 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.309 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.294+0000 7f42b3680640 4 set_mon_vals no callback set 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: shutdown 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- 172.21.15.6:0/4011807568 >> v1:172.21.15.6:6790/0 conn(0x7f42ac0a0da0 legacy=0x7f42ac13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- 172.21.15.6:0/4011807568 shutdown_connections 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- 172.21.15.6:0/4011807568 >> 172.21.15.6:0/4011807568 conn(0x7f42ac0a1650 msgr2=0x7f42ac0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- 172.21.15.6:0/4011807568 shutdown_connections 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- 172.21.15.6:0/4011807568 wait complete. 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: build_initial_monmap 2024-09-16T05:36:49.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: monmap: 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 Processor -- start 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- start start 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: init 2024-09-16T05:36:49.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient(hunting): picked mon.noname-c con 0x7f42ac09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient(hunting): picked mon.noname-b con 0x7f42ac09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient(hunting): picked mon.noname-a con 0x7f42ac0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac06e640 con 0x7f42ac0a0da0 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac13b530 con 0x7f42ac09e230 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac13b810 con 0x7f42ac09aa30 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b141c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f42ac09aa30 0x7f42ac09bdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50844/0 (socket says 172.21.15.6:50844) 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient(hunting): authenticate will time out at 1071.106009s 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b141c640 1 -- 172.21.15.6:0/3239086992 learned_addr learned my addr 172.21.15.6:0/3239086992 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859122416 0 0) 0x7f42ac13b810 con 0x7f42ac09aa30 2024-09-16T05:36:49.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): my global_id is 4856 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4290003530 con 0x7f42ac09aa30 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1531673319 0 0) 0x7f42ac09dd50 con 0x7f42ac0a0da0 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): my global_id is 4869 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ac13b810 con 0x7f42ac0a0da0 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658198837 0 0) 0x7f42ac13b530 con 0x7f42ac09e230 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): my global_id is 94108 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ac09dd50 con 0x7f42ac09e230 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 668595458 0 0) 0x7f42ac13b810 con 0x7f42ac0a0da0 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42ac13b530 con 0x7f42ac0a0da0 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a4001520 con 0x7f42ac0a0da0 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): dump: 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.314 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 866726469 0 0) 0x7f4290003530 con 0x7f42ac09aa30 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42ac13b810 con 0x7f42ac09aa30 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a0001a20 con 0x7f42ac09aa30 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient(hunting): dump: 2024-09-16T05:36:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2713091258 0 0) 0x7f42ac13b530 con 0x7f42ac0a0da0 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 >> v1:172.21.15.6:6790/0 conn(0x7f42ac09aa30 legacy=0x7f42ac09bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 >> v1:172.21.15.63:6789/0 conn(0x7f42ac09e230 legacy=0x7f42ac09c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:49.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 monclient: found mon.a 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac13a5a0 con 0x7f42ac0a0da0 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 5 monclient: authenticate success, global_id 4869 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: _renew_subs 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42ac09cbf0 con 0x7f42ac0a0da0 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: _renew_subs 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:49.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42ac1d4e40 con 0x7f42ac0a0da0 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42a40011d0 con 0x7f42ac0a0da0 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a4003b10 con 0x7f42ac0a0da0 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 10 monclient: dump: 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.318 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429f7fe640 4 set_mon_vals no callback set 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f428c0065d0 con 0x7f428c002870 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.298+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f42a400e710 con 0x7f42ac0a0da0 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7f42a40432c0 con 0x7f42ac0a0da0 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653194878 0 0) 0x7f428c0065d0 con 0x7f428c002870 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient(hunting): my global_id is 4859 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ac13b530 con 0x7f428c002870 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f42b0c1b640 10 monclient: get_auth_request con 0x7f42900368f0 auth_method 0 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 170544770 0 0) 0x7f42ac13b530 con 0x7f428c002870 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f428c0065d0 con 0x7f428c002870 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a0004990 con 0x7f428c002870 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient: dump: 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.320 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3165212570 0 0) 0x7f428c0065d0 con 0x7f428c002870 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f428c00a160 con 0x7f428c002870 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f42a0005eb0 con 0x7f428c002870 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 >> v1:172.21.15.6:6790/0 conn(0x7f428c002870 legacy=0x7f428c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.302+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 >> v1:172.21.15.6:6790/0 conn(0x7f428c002870 legacy=0x7f428c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f42b3680640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f428c009e30 con 0x7f428c0090f0 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310156238 0 0) 0x7f428c009e30 con 0x7f428c0090f0 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 10 monclient(hunting): my global_id is 4862 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f428c0065d0 con 0x7f428c0090f0 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1195806890 0 0) 0x7f428c0065d0 con 0x7f428c0090f0 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f428c009e30 con 0x7f428c0090f0 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a0006500 con 0x7f428c0090f0 2024-09-16T05:36:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 10 monclient: dump: 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3891643603 0 0) 0x7f428c009e30 con 0x7f428c0090f0 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.306+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f428c00c3a0 con 0x7f428c0090f0 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 3305198609) 0x7f42a0006360 con 0x7f428c0090f0 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f429dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f429dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 >> v1:172.21.15.6:6790/0 conn(0x7f428c0090f0 legacy=0x7f428c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f429dffb640 1 -- 172.21.15.6:0/3239086992 >> v1:172.21.15.6:6790/0 conn(0x7f428c0090f0 legacy=0x7f428c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:36:49.325 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 14, 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 138366, 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.327 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:36:49.328 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:36:49.329 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.330 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.332 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:36:49.333 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.334 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.335 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:36:49.336 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 >> v1:172.21.15.63:6812/3643207211 conn(0x7f42900368f0 legacy=0x7f4290038db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b3680640 10 monclient: shutdown 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b3680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 >> v1:172.21.15.6:6789/0 conn(0x7f42ac0a0da0 legacy=0x7f42ac1d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b1c1d640 1 -- 172.21.15.6:0/3239086992 reap_dead start 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b1c1d640 1 -- 172.21.15.6:0/3239086992 reap_dead start 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 shutdown_connections 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 >> 172.21.15.6:0/3239086992 conn(0x7f42ac0a1650 msgr2=0x7f42ac0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 shutdown_connections 2024-09-16T05:36:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.310+0000 7f42b3680640 1 -- 172.21.15.6:0/3239086992 wait complete. 2024-09-16T05:36:49.339 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:36:49.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:36:49.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:36:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient: get_monmap_and_config 2024-09-16T05:36:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient: build_initial_monmap 2024-09-16T05:36:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient: monmap: 2024-09-16T05:36:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:49.455159+0000 2024-09-16T05:36:49.455 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:49.455159+0000 2024-09-16T05:36:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:49.456 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:49.456 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 Processor -- start 2024-09-16T05:36:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- start start 2024-09-16T05:36:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient: init 2024-09-16T05:36:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:49.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient(hunting): picked mon.noname-c con 0x7f07d009bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient(hunting): picked mon.noname-b con 0x7f07d009bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient(hunting): picked mon.noname-a con 0x7f07d01397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d00a1350 con 0x7f07d01397f0 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d00a10d0 con 0x7f07d009bb40 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d009d940 con 0x7f07d009bf20 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient(hunting): authenticate will time out at 1071.261840s 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07ce59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f07d009bf20 0x7f07d009c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50870/0 (socket says 172.21.15.6:50870) 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07ce59c640 1 -- 172.21.15.6:0/2056819796 learned_addr learned my addr 172.21.15.6:0/2056819796 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713656793 0 0) 0x7f07d00a1350 con 0x7f07d01397f0 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): my global_id is 4878 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07bc005740 con 0x7f07d01397f0 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1025674006 0 0) 0x7f07d009d940 con 0x7f07d009bf20 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): my global_id is 4871 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d00a1350 con 0x7f07d009bf20 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994352132 0 0) 0x7f07d00a10d0 con 0x7f07d009bb40 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient(hunting): my global_id is 94117 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d009d940 con 0x7f07d009bb40 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4225740261 0 0) 0x7f07bc005740 con 0x7f07d01397f0 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 >> v1:172.21.15.6:6790/0 conn(0x7f07d009bf20 legacy=0x7f07d009c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 >> v1:172.21.15.63:6789/0 conn(0x7f07d009bb40 legacy=0x7f07d009e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 monclient: found mon.noname-a 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d009dc20 con 0x7f07d01397f0 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: _finish_auth 0 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:49.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07c80019a0 con 0x7f07d01397f0 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: dump: 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.463 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 5 monclient: authenticate success, global_id 4878 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07c8001c50 con 0x7f07d01397f0 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 1 -- 172.21.15.6:0/2056819796 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07c8002020 con 0x7f07d01397f0 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 4 set_mon_vals no callback set 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07cdd9b640 10 monclient: dump: 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.464 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 10 monclient: shutdown 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- 172.21.15.6:0/2056819796 >> v1:172.21.15.6:6789/0 conn(0x7f07d01397f0 legacy=0x7f07d013bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- 172.21.15.6:0/2056819796 shutdown_connections 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- 172.21.15.6:0/2056819796 >> 172.21.15.6:0/2056819796 conn(0x7f07d00a1650 msgr2=0x7f07d00a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- 172.21.15.6:0/2056819796 shutdown_connections 2024-09-16T05:36:49.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.454+0000 7f07d5208640 1 -- 172.21.15.6:0/2056819796 wait complete. 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: build_initial_monmap 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: monmap: 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 1 Processor -- start 2024-09-16T05:36:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 1 -- start start 2024-09-16T05:36:49.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: init 2024-09-16T05:36:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient(hunting): picked mon.noname-b con 0x7f07d009bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient(hunting): picked mon.noname-a con 0x7f07d009bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient(hunting): picked mon.noname-c con 0x7f07d01397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:49.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d00a3b10 con 0x7f07d009bf20 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d011d540 con 0x7f07d009bb40 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d00727b0 con 0x7f07d01397f0 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient(hunting): authenticate will time out at 1071.265070s 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07cf59e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f07d01397f0 0x7f07d01d3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50876/0 (socket says 172.21.15.6:50876) 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07cf59e640 1 -- 172.21.15.6:0/4233291047 learned_addr learned my addr 172.21.15.6:0/4233291047 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07ce59c640 1 --1- 172.21.15.6:0/4233291047 >> v1:172.21.15.6:6789/0 conn(0x7f07d009bf20 0x7f07d011b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59298/0 (socket says 172.21.15.6:59298) 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 462392252 0 0) 0x7f07d00727b0 con 0x7f07d01397f0 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): my global_id is 4874 2024-09-16T05:36:49.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07b0003530 con 0x7f07d01397f0 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3085156971 0 0) 0x7f07d00a1350 con 0x7f07d009bf20 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): my global_id is 4881 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d00727b0 con 0x7f07d009bf20 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 568499781 0 0) 0x7f07d011d540 con 0x7f07d009bb40 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): my global_id is 94120 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d00a1350 con 0x7f07d009bb40 2024-09-16T05:36:49.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4294800473 0 0) 0x7f07d00727b0 con 0x7f07d009bf20 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07d011d540 con 0x7f07d009bf20 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3052580206 0 0) 0x7f07b0003530 con 0x7f07d01397f0 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07d00727b0 con 0x7f07d01397f0 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07c8001fa0 con 0x7f07d01397f0 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): dump: 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.474 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07b8001900 con 0x7f07d009bf20 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): dump: 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.475 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3286595827 0 0) 0x7f07d00727b0 con 0x7f07d01397f0 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 >> v1:172.21.15.63:6789/0 conn(0x7f07d009bb40 legacy=0x7f07d011b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 >> v1:172.21.15.6:6789/0 conn(0x7f07d009bf20 legacy=0x7f07d011b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 monclient: found mon.c 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:49.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d011bfc0 con 0x7f07d01397f0 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 5 monclient: authenticate success, global_id 4874 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: _renew_subs 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07d011d820 con 0x7f07d01397f0 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: _renew_subs 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f07d011d260 con 0x7f07d01397f0 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07c8001c50 con 0x7f07d01397f0 2024-09-16T05:36:49.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07c8003eb0 con 0x7f07d01397f0 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07cd59a640 4 set_mon_vals no callback set 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient: dump: 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.478 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f07b40065d0 con 0x7f07b4002870 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f07c8004050 con 0x7f07d01397f0 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7f07c80446a0 con 0x7f07d01397f0 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266813932 0 0) 0x7f07b40065d0 con 0x7f07b4002870 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 10 monclient(hunting): my global_id is 94123 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d00727b0 con 0x7f07b4002870 2024-09-16T05:36:49.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07ce59c640 10 monclient: get_auth_request con 0x7f07b0036b00 auth_method 0 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.458+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3182178503 0 0) 0x7f07d00727b0 con 0x7f07b4002870 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07b40065d0 con 0x7f07b4002870 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07c00017e0 con 0x7f07b4002870 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 10 monclient: dump: 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.480 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2806609738 0 0) 0x7f07b40065d0 con 0x7f07b4002870 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07b400a160 con 0x7f07b4002870 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f07c0001ae0 con 0x7f07b4002870 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 >> v1:172.21.15.63:6789/0 conn(0x7f07b4002870 legacy=0x7f07b4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.462+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 >> v1:172.21.15.63:6789/0 conn(0x7f07b4002870 legacy=0x7f07b4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.466+0000 7f07d5208640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.466+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f07b4009e30 con 0x7f07b40090f0 2024-09-16T05:36:49.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4259127311 0 0) 0x7f07b4009e30 con 0x7f07b40090f0 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient(hunting): my global_id is 94126 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07b40065d0 con 0x7f07b40090f0 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4179864318 0 0) 0x7f07b40065d0 con 0x7f07b40090f0 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07b4009e30 con 0x7f07b40090f0 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07c00037a0 con 0x7f07b40090f0 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient: dump: 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.482 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2458979738 0 0) 0x7f07b4009e30 con 0x7f07b40090f0 2024-09-16T05:36:49.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07b400c3a0 con 0x7f07b40090f0 2024-09-16T05:36:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3959 (unknown 0 0 1431010330) 0x7f07c0005a60 con 0x7f07b40090f0 2024-09-16T05:36:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:36:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 >> v1:172.21.15.63:6789/0 conn(0x7f07b40090f0 legacy=0x7f07b4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07c77fe640 1 -- 172.21.15.6:0/4233291047 >> v1:172.21.15.63:6789/0 conn(0x7f07b40090f0 legacy=0x7f07b4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:49.484 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:36:49.484 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 14, 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 779, 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:36:49.485 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.486 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:36:49.488 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:49.489 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.490 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:36:49.491 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.492 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.494 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:36:49.495 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 >> v1:172.21.15.63:6812/3643207211 conn(0x7f07b0036b00 legacy=0x7f07b0038fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07d5208640 10 monclient: shutdown 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07d5208640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 >> v1:172.21.15.6:6790/0 conn(0x7f07d01397f0 legacy=0x7f07d01d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07cf59e640 1 -- 172.21.15.6:0/4233291047 reap_dead start 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07cf59e640 1 -- 172.21.15.6:0/4233291047 reap_dead start 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 shutdown_connections 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 >> 172.21.15.6:0/4233291047 conn(0x7f07d00a1650 msgr2=0x7f07d00a4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 shutdown_connections 2024-09-16T05:36:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.470+0000 7f07d5208640 1 -- 172.21.15.6:0/4233291047 wait complete. 2024-09-16T05:36:49.498 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:36:49.498 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:36:49.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:36:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.618+0000 7f1cb129c640 10 monclient: get_monmap_and_config 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.618+0000 7f1cb129c640 10 monclient: build_initial_monmap 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.618+0000 7f1cb129c640 10 monclient: monmap: 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:49.621668+0000 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:49.621668+0000 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:49.623 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.618+0000 7f1cb129c640 1 Processor -- start 2024-09-16T05:36:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.618+0000 7f1cb129c640 1 -- start start 2024-09-16T05:36:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.618+0000 7f1cb129c640 10 monclient: init 2024-09-16T05:36:49.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient(hunting): picked mon.noname-b con 0x7f1cac0a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient(hunting): picked mon.noname-c con 0x7f1cac0a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient(hunting): picked mon.noname-a con 0x7f1cac0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac06e520 con 0x7f1cac0a1a00 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac0a0780 con 0x7f1cac0a4290 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac0a4c70 con 0x7f1cac0a37a0 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient(hunting): authenticate will time out at 1071.428485s 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1caaffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1cac0a37a0 0x7f1cac0a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50886/0 (socket says 172.21.15.6:50886) 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1caaffd640 1 -- 172.21.15.6:0/254022191 learned_addr learned my addr 172.21.15.6:0/254022191 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:49.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471207900 0 0) 0x7f1cac0a4c70 con 0x7f1cac0a37a0 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): my global_id is 4877 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c94003530 con 0x7f1cac0a37a0 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523957130 0 0) 0x7f1cac06e520 con 0x7f1cac0a1a00 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): my global_id is 4884 2024-09-16T05:36:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cac0a4c70 con 0x7f1cac0a1a00 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4294087440 0 0) 0x7f1cac0a0780 con 0x7f1cac0a4290 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient(hunting): my global_id is 94129 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cac06e520 con 0x7f1cac0a4290 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2458123840 0 0) 0x7f1c94003530 con 0x7f1cac0a37a0 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 >> v1:172.21.15.63:6789/0 conn(0x7f1cac0a4290 legacy=0x7f1cac13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 >> v1:172.21.15.6:6789/0 conn(0x7f1cac0a1a00 legacy=0x7f1cac13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 monclient: found mon.noname-b 2024-09-16T05:36:49.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:49.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cac0a65f0 con 0x7f1cac0a37a0 2024-09-16T05:36:49.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:49.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1c9c001620 con 0x7f1cac0a37a0 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 5 monclient: authenticate success, global_id 4877 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: dump: 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.630 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c9c001880 con 0x7f1cac0a37a0 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 1 -- 172.21.15.6:0/254022191 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1c9c001c90 con 0x7f1cac0a37a0 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 4 set_mon_vals no callback set 2024-09-16T05:36:49.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1ca9ffb640 10 monclient: dump: 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient: shutdown 2024-09-16T05:36:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- 172.21.15.6:0/254022191 >> v1:172.21.15.6:6790/0 conn(0x7f1cac0a37a0 legacy=0x7f1cac0a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- 172.21.15.6:0/254022191 shutdown_connections 2024-09-16T05:36:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- 172.21.15.6:0/254022191 >> 172.21.15.6:0/254022191 conn(0x7f1cac0a58a0 msgr2=0x7f1cac0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- 172.21.15.6:0/254022191 shutdown_connections 2024-09-16T05:36:49.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- 172.21.15.6:0/254022191 wait complete. 2024-09-16T05:36:49.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient: build_initial_monmap 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient: monmap: 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 Processor -- start 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 1 -- start start 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient: init 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.622+0000 7f1cb129c640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient(hunting): picked mon.noname-a con 0x7f1cac0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient(hunting): picked mon.noname-b con 0x7f1cac0a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient(hunting): picked mon.noname-c con 0x7f1cac13f570 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac13ee70 con 0x7f1cac0a1a00 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac13bfe0 con 0x7f1cac0a4290 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac13d9c0 con 0x7f1cac13f570 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient(hunting): authenticate will time out at 1071.432139s 2024-09-16T05:36:49.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1caaffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1cac0a1a00 0x7f1cac13b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59316/0 (socket says 172.21.15.6:59316) 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1caaffd640 1 -- 172.21.15.6:0/1449112028 learned_addr learned my addr 172.21.15.6:0/1449112028 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4097103128 0 0) 0x7f1cac13d9c0 con 0x7f1cac13f570 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): my global_id is 4880 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c98003560 con 0x7f1cac13f570 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3454412467 0 0) 0x7f1cac13ee70 con 0x7f1cac0a1a00 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): my global_id is 4887 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cac13d9c0 con 0x7f1cac0a1a00 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2994744945 0 0) 0x7f1c98003560 con 0x7f1cac13f570 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cac13ee70 con 0x7f1cac13f570 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1ca4001da0 con 0x7f1cac13f570 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): dump: 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.641 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151234154 0 0) 0x7f1cac13bfe0 con 0x7f1cac0a4290 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:49.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): my global_id is 94132 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c98003560 con 0x7f1cac0a4290 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1964622814 0 0) 0x7f1cac13d9c0 con 0x7f1cac0a1a00 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cac13bfe0 con 0x7f1cac0a1a00 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1c9c001bd0 con 0x7f1cac0a1a00 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient(hunting): dump: 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.643 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3533027010 0 0) 0x7f1cac13ee70 con 0x7f1cac13f570 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 >> v1:172.21.15.63:6789/0 conn(0x7f1cac0a4290 legacy=0x7f1cac13b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 >> v1:172.21.15.6:6789/0 conn(0x7f1cac0a1a00 legacy=0x7f1cac13b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:49.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 monclient: found mon.c 2024-09-16T05:36:49.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:49.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cac13f150 con 0x7f1cac13f570 2024-09-16T05:36:49.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 5 monclient: authenticate success, global_id 4880 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient: _renew_subs 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1cac0a8020 con 0x7f1cac13f570 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient: _renew_subs 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1cac13eb90 con 0x7f1cac13f570 2024-09-16T05:36:49.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ca4002050 con 0x7f1cac13f570 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1ca4003dd0 con 0x7f1cac13f570 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1ca97fa640 4 set_mon_vals no callback set 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 10 monclient: dump: 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:49.647 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1ca400eb70 con 0x7f1cac13f570 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8c005090 con 0x7f1cac13f570 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7f1ca40436a0 con 0x7f1cac13f570 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.626+0000 7f1caaffd640 10 monclient: get_auth_request con 0x7f1c98036c70 auth_method 0 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.630+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1ca40145f0 con 0x7f1cac13f570 2024-09-16T05:36:49.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.630+0000 7f1c937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:49.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.630+0000 7f1c937fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:49.686 INFO:tasks.workunit.client.0.smithi063.stdout:2024-09-16T05:36:49.563+0000 7f8f0e7fc640 1 -- 172.21.15.63:0/678179289 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v61) ==== 103+0+0 (unknown 1012277187 0 0) 0x7f8f10015090 con 0x7f8f180a0280 2024-09-16T05:36:49.686 INFO:tasks.workunit.client.0.smithi063.stdout:2024-09-16T05:36:49.563+0000 7f8f0e7fc640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-16T05:36:49.686 INFO:tasks.workunit.client.0.smithi063.stdout:2024-09-16T05:36:49.667+0000 7f8f0e7fc640 1 -- 172.21.15.63:0/678179289 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v61) ==== 103+0+0 (unknown 1012277187 0 0) 0x7f8f1001dc40 con 0x7f8f180a0280 2024-09-16T05:36:49.686 INFO:tasks.workunit.client.0.smithi063.stdout:2024-09-16T05:36:49.667+0000 7f8f0e7fc640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T05:36:49.686 INFO:tasks.workunit.client.0.smithi063.stdout:rule foo does not exist 2024-09-16T05:36:49.686 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rm bar 2024-09-16T05:36:49.687 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'does not exist' 2024-09-16T05:36:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.858+0000 7f1cb129c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:36:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.858+0000 7f1cb129c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:36:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.858+0000 7f1cb129c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.858+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1c8c005b90 con 0x7f1cac13f570 2024-09-16T05:36:49.864 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.864 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=1911183372,config=3541143443,health=3897446751,kv=2361287224,logm=1195993906}) 2024-09-16T05:36:49.865 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=10,logm=61} crc {auth=3477334232,config=3541143443,health=2609900051,kv=2392046976,logm=4220528027}) 2024-09-16T05:36:49.865 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.865 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=1911183372,config=3541143443,health=3897446751,kv=2361287224,logm=1195993906}) 2024-09-16T05:36:49.865 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=11533800,config=3541143443,health=3897446751,kv=2361287224,logm=3939090652}) 2024-09-16T05:36:49.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f1c937fe640 1 -- 172.21.15.6:0/1449112028 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f1ca4013ef0 con 0x7f1cac13f570 2024-09-16T05:36:49.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f1c937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:36:49.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f1c937fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:49.866 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.866 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324988202}) 2024-09-16T05:36:49.866 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2736810217}) 2024-09-16T05:36:49.866 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.866 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324988202}) 2024-09-16T05:36:49.866 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=131631683}) 2024-09-16T05:36:49.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.862+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1c98036c70 legacy=0x7f1c98039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f1cb129c640 10 monclient: shutdown 2024-09-16T05:36:49.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f1cb129c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:49.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 >> v1:172.21.15.6:6790/0 conn(0x7f1cac13f570 legacy=0x7f1cac1d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:49.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 shutdown_connections 2024-09-16T05:36:49.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 >> 172.21.15.6:0/1449112028 conn(0x7f1cac0a58a0 msgr2=0x7f1cac1401e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:49.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 shutdown_connections 2024-09-16T05:36:49.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f1cb129c640 1 -- 172.21.15.6:0/1449112028 wait complete. 2024-09-16T05:36:49.869 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.869 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3813687275}) 2024-09-16T05:36:49.869 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011244615}) 2024-09-16T05:36:49.869 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.869 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3813687275}) 2024-09-16T05:36:49.869 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=255436235}) 2024-09-16T05:36:49.870 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.870 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972260488}) 2024-09-16T05:36:49.870 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669531326}) 2024-09-16T05:36:49.870 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.870 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972260488}) 2024-09-16T05:36:49.870 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.866+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798843057}) 2024-09-16T05:36:49.871 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.871 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48799687}) 2024-09-16T05:36:49.871 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=501940729}) 2024-09-16T05:36:49.871 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.871 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48799687}) 2024-09-16T05:36:49.871 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237328088}) 2024-09-16T05:36:49.873 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.873 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4018374298}) 2024-09-16T05:36:49.873 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1560285830}) 2024-09-16T05:36:49.873 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.873 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4018374298}) 2024-09-16T05:36:49.874 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477603630}) 2024-09-16T05:36:49.874 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.874 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=79,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9} crc {logm=1068182965,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgrstat=2228867744}) 2024-09-16T05:36:49.875 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=2165703782,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2228867744}) 2024-09-16T05:36:49.875 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.875 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=79,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9} crc {logm=1068182965,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgrstat=2228867744}) 2024-09-16T05:36:49.875 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.870+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=3811505596,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=353370607}) 2024-09-16T05:36:49.876 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.874+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.876 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.874+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=3,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=2500552818,mon_config_key=2928357758,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=496959941,osd_snap=723727393,osdmap=3791862026}) 2024-09-16T05:36:49.876 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.874+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=13,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mgrstat=2500552818,mon_config_key=3206761495,monmap=3625042559,osd_metadata=634326011,osd_pg_creating=496959941,osd_snap=723727393,osdmap=512178608}) 2024-09-16T05:36:49.876 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.874+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.876 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.874+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=3,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=2500552818,mon_config_key=2928357758,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=496959941,osd_snap=723727393,osdmap=3791862026}) 2024-09-16T05:36:49.877 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.874+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=3,mon_config_key=13,monmap=3,osd_metadata=5,osd_snap=2,osdmap=74} crc {mgrstat=2500552818,mon_config_key=2399792772,monmap=3625042558,osd_metadata=4164660051,osd_snap=1927752934,osdmap=864219365}) 2024-09-16T05:36:49.879 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.878+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.879 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.878+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=29} crc {osdmap=3862615585}) 2024-09-16T05:36:49.879 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.878+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=743576190}) 2024-09-16T05:36:49.879 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.878+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:36:49.879 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.878+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=29} crc {osdmap=3862615585}) 2024-09-16T05:36:49.879 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:36:49.878+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=30} crc {osdmap=3319078156}) 2024-09-16T05:36:49.885 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:36:50.137 INFO:tasks.workunit.client.0.smithi063.stdout:2024-09-16T05:36:49.975+0000 7f01ddffb640 1 -- 172.21.15.63:0/1073437548 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v61) ==== 103+0+0 (unknown 1563074952 0 0) 0x7f01d8019120 con 0x7f01e809b780 2024-09-16T05:36:50.138 INFO:tasks.workunit.client.0.smithi063.stdout:2024-09-16T05:36:49.979+0000 7f01ddffb640 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-16T05:36:50.138 INFO:tasks.workunit.client.0.smithi063.stdout:2024-09-16T05:36:50.119+0000 7f01ddffb640 1 -- 172.21.15.63:0/1073437548 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v61) ==== 103+0+0 (unknown 1563074952 0 0) 0x7f01d801d9e0 con 0x7f01e809b780 2024-09-16T05:36:50.138 INFO:tasks.workunit.client.0.smithi063.stdout:2024-09-16T05:36:50.119+0000 7f01ddffb640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-16T05:36:50.138 INFO:tasks.workunit.client.0.smithi063.stdout:rule bar does not exist 2024-09-16T05:36:50.138 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-16T05:36:50.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: get_monmap_and_config 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: build_initial_monmap 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: monmap: 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:50.256611+0000 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:50.256611+0000 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:50.257 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:50.258 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:50.258 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:50.258 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:50.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 Processor -- start 2024-09-16T05:36:50.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- start start 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: init 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient(hunting): picked mon.noname-a con 0x7f359c0a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient(hunting): picked mon.noname-c con 0x7f359c0a4d90 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient(hunting): picked mon.noname-b con 0x7f359c0a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359c0a51e0 con 0x7f359c0a3ca0 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359c0a0d60 con 0x7f359c0a4d90 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359c0a1040 con 0x7f359c0a4790 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient(hunting): authenticate will time out at 1067.033625s 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359bfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f359c0a4d90 0x7f359c13bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47298/0 (socket says 172.21.15.63:47298) 2024-09-16T05:36:50.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359bfff640 1 -- 172.21.15.63:0/1489721407 learned_addr learned my addr 172.21.15.63:0/1489721407 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092013974 0 0) 0x7f359c0a0d60 con 0x7f359c0a4d90 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient(hunting): my global_id is 94141 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3584003530 con 0x7f359c0a4d90 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3231476370 0 0) 0x7f3584003530 con 0x7f359c0a4d90 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 >> v1:172.21.15.6:6790/0 conn(0x7f359c0a4790 legacy=0x7f359c13f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 >> v1:172.21.15.6:6789/0 conn(0x7f359c0a3ca0 legacy=0x7f359c0a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 monclient: found mon.noname-c 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:50.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c0a0810 con 0x7f359c0a4d90 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3590001620 con 0x7f359c0a4d90 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: dump: 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:50.267 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 5 monclient: authenticate success, global_id 94141 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3590001910 con 0x7f359c0a4d90 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:50.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 1 -- 172.21.15.63:0/1489721407 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3590001ce0 con 0x7f359c0a4d90 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f359b7fe640 10 monclient: dump: 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:50.269 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 4 set_mon_vals no callback set 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: shutdown 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- 172.21.15.63:0/1489721407 >> v1:172.21.15.63:6789/0 conn(0x7f359c0a4d90 legacy=0x7f359c13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- 172.21.15.63:0/1489721407 shutdown_connections 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- 172.21.15.63:0/1489721407 >> 172.21.15.63:0/1489721407 conn(0x7f359c075470 msgr2=0x7f359c075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- 172.21.15.63:0/1489721407 shutdown_connections 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 1 -- 172.21.15.63:0/1489721407 wait complete. 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: build_initial_monmap 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.255+0000 7f35a2e17640 10 monclient: monmap: 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:50.270 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 1 Processor -- start 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 1 -- start start 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: init 2024-09-16T05:36:50.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient(hunting): picked mon.noname-a con 0x7f359c0a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient(hunting): picked mon.noname-b con 0x7f359c0a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient(hunting): picked mon.noname-c con 0x7f359c0a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359c13eaa0 con 0x7f359c0a3ca0 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359c13bef0 con 0x7f359c0a4790 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359c13ed80 con 0x7f359c0a4d90 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient(hunting): authenticate will time out at 1067.036722s 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f359bfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f359c0a4790 0x7f359c13b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47314/0 (socket says 172.21.15.63:47314) 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f359bfff640 1 -- 172.21.15.63:0/966920107 learned_addr learned my addr 172.21.15.63:0/966920107 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:50.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3263752876 0 0) 0x7f359c13bef0 con 0x7f359c0a4790 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): my global_id is 94144 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3588003530 con 0x7f359c0a4790 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 599391239 0 0) 0x7f3588003530 con 0x7f359c0a4790 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f359c13bef0 con 0x7f359c0a4790 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35900015e0 con 0x7f359c0a4790 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): dump: 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:50.273 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765141863 0 0) 0x7f359c13ed80 con 0x7f359c0a4d90 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): _negotiate 2024-09-16T05:36:50.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient(hunting): my global_id is 4892 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3588003530 con 0x7f359c0a4d90 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2856492038 0 0) 0x7f359c13bef0 con 0x7f359c0a4790 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 >> v1:172.21.15.6:6790/0 conn(0x7f359c0a4d90 legacy=0x7f359c1d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 >> v1:172.21.15.6:6789/0 conn(0x7f359c0a3ca0 legacy=0x7f359c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient: _finish_hunting 0 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 monclient: found mon.b 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c0a87b0 con 0x7f359c0a4790 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient: _finish_auth 0 2024-09-16T05:36:50.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient: _check_auth_tickets 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 5 monclient: authenticate success, global_id 94144 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: _renew_subs 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f359c13e7c0 con 0x7f359c0a4790 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: _renew_subs 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f359c13d530 con 0x7f359c0a4790 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3590004cc0 con 0x7f359c0a4790 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:50.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3590005010 con 0x7f359c0a4790 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f359affd640 4 set_mon_vals no callback set 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 10 monclient: dump: 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:50.277 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f357c005090 con 0x7f359c0a4790 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3590005230 con 0x7f359c0a4790 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7f35900446f0 con 0x7f359c0a4790 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.259+0000 7f35a0bb3640 10 monclient: get_auth_request con 0x7f3588036a60 auth_method 0 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.263+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3590015700 con 0x7f359c0a4790 2024-09-16T05:36:50.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.263+0000 7f35997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:50.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.263+0000 7f35997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:50.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.423+0000 7f35a2e17640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:50.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.423+0000 7f35a2e17640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:50.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.423+0000 7f35a2e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:50.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:50.423+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) -- 0x7f357c005380 con 0x7f359c0a4790 2024-09-16T05:36:50.885 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:36:50.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:36:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient: get_monmap_and_config 2024-09-16T05:36:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient: monmap: 2024-09-16T05:36:51.024 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:51.023731+0000 2024-09-16T05:36:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:51.023731+0000 2024-09-16T05:36:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.025 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:51.026 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:51.026 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 1 Processor -- start 2024-09-16T05:36:51.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 1 -- start start 2024-09-16T05:36:51.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient: init 2024-09-16T05:36:51.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:51.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient(hunting): picked mon.noname-a con 0x7f0bbc09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient(hunting): picked mon.noname-c con 0x7f0bbc09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient(hunting): picked mon.noname-b con 0x7f0bbc0a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bbc0a1370 con 0x7f0bbc09aa30 2024-09-16T05:36:51.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bbc09ead0 con 0x7f0bbc09e230 2024-09-16T05:36:51.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bbc09edb0 con 0x7f0bbc0a0da0 2024-09-16T05:36:51.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient(hunting): authenticate will time out at 1072.830428s 2024-09-16T05:36:51.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bbb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0bbc09aa30 0x7f0bbc09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59320/0 (socket says 172.21.15.6:59320) 2024-09-16T05:36:51.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bbb7fe640 1 -- 172.21.15.6:0/3760121719 learned_addr learned my addr 172.21.15.6:0/3760121719 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343746082 0 0) 0x7f0bbc0a1370 con 0x7f0bbc09aa30 2024-09-16T05:36:51.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient(hunting): my global_id is 4902 2024-09-16T05:36:51.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ba4003530 con 0x7f0bbc09aa30 2024-09-16T05:36:51.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3452733842 0 0) 0x7f0bbc09edb0 con 0x7f0bbc0a0da0 2024-09-16T05:36:51.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient(hunting): my global_id is 4895 2024-09-16T05:36:51.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bbc0a1370 con 0x7f0bbc0a0da0 2024-09-16T05:36:51.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3244823554 0 0) 0x7f0ba4003530 con 0x7f0bbc09aa30 2024-09-16T05:36:51.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 >> v1:172.21.15.6:6790/0 conn(0x7f0bbc0a0da0 legacy=0x7f0bbc13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 >> v1:172.21.15.63:6789/0 conn(0x7f0bbc09e230 legacy=0x7f0bbc1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 monclient: found mon.noname-a 2024-09-16T05:36:51.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:51.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc09dd50 con 0x7f0bbc09aa30 2024-09-16T05:36:51.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bac001620 con 0x7f0bbc09aa30 2024-09-16T05:36:51.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:51.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: dump: 2024-09-16T05:36:51.045 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.045 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.046 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.046 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.046 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.047 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.047 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.047 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.047 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.047 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 5 monclient: authenticate success, global_id 4902 2024-09-16T05:36:51.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bac001890 con 0x7f0bbc09aa30 2024-09-16T05:36:51.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 1 -- 172.21.15.6:0/3760121719 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bac001ca0 con 0x7f0bbc09aa30 2024-09-16T05:36:51.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bba7fc640 10 monclient: dump: 2024-09-16T05:36:51.049 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.049 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.049 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.049 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.050 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.050 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.050 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.050 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.050 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.050 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:51.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 4 set_mon_vals no callback set 2024-09-16T05:36:51.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 10 monclient: shutdown 2024-09-16T05:36:51.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.022+0000 7f0bc2140640 1 -- 172.21.15.6:0/3760121719 >> v1:172.21.15.6:6789/0 conn(0x7f0bbc09aa30 legacy=0x7f0bbc09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- 172.21.15.6:0/3760121719 shutdown_connections 2024-09-16T05:36:51.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- 172.21.15.6:0/3760121719 >> 172.21.15.6:0/3760121719 conn(0x7f0bbc0a1650 msgr2=0x7f0bbc0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- 172.21.15.6:0/3760121719 shutdown_connections 2024-09-16T05:36:51.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- 172.21.15.6:0/3760121719 wait complete. 2024-09-16T05:36:51.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: monmap: 2024-09-16T05:36:51.052 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.053 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:51.053 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:51.053 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:51.053 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.053 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.053 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.054 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:51.054 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:51.054 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 Processor -- start 2024-09-16T05:36:51.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- start start 2024-09-16T05:36:51.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: init 2024-09-16T05:36:51.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:36:51.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient(hunting): picked mon.noname-a con 0x7f0bbc09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient(hunting): picked mon.noname-c con 0x7f0bbc09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient(hunting): picked mon.noname-b con 0x7f0bbc0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bbc09c1b0 con 0x7f0bbc09aa30 2024-09-16T05:36:51.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bbc09bcf0 con 0x7f0bbc0a0da0 2024-09-16T05:36:51.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bbc120c70 con 0x7f0bbc09e230 2024-09-16T05:36:51.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient(hunting): authenticate will time out at 1072.833523s 2024-09-16T05:36:51.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bbb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0bbc09aa30 0x7f0bbc133870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59334/0 (socket says 172.21.15.6:59334) 2024-09-16T05:36:51.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bbb7fe640 1 -- 172.21.15.6:0/4102280743 learned_addr learned my addr 172.21.15.6:0/4102280743 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bbaffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0bbc09e230 0x7f0bbc1cbfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50914/0 (socket says 172.21.15.6:50914) 2024-09-16T05:36:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352918375 0 0) 0x7f0bbc120c70 con 0x7f0bbc09e230 2024-09-16T05:36:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): my global_id is 4898 2024-09-16T05:36:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ba8003530 con 0x7f0bbc09e230 2024-09-16T05:36:51.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1028996601 0 0) 0x7f0bbc09c1b0 con 0x7f0bbc09aa30 2024-09-16T05:36:51.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): my global_id is 4905 2024-09-16T05:36:51.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bbc120c70 con 0x7f0bbc09aa30 2024-09-16T05:36:51.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2695961403 0 0) 0x7f0ba8003530 con 0x7f0bbc09e230 2024-09-16T05:36:51.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0bbc09c1b0 con 0x7f0bbc09e230 2024-09-16T05:36:51.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bb00018c0 con 0x7f0bbc09e230 2024-09-16T05:36:51.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:51.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): dump: 2024-09-16T05:36:51.060 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.061 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.061 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.061 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.061 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.061 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.061 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.062 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.062 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.062 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:51.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2374424591 0 0) 0x7f0bbc120c70 con 0x7f0bbc09aa30 2024-09-16T05:36:51.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ba8003530 con 0x7f0bbc09aa30 2024-09-16T05:36:51.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bac001be0 con 0x7f0bbc09aa30 2024-09-16T05:36:51.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): dump: 2024-09-16T05:36:51.063 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.063 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.064 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.064 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.064 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.064 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.064 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.064 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.065 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.065 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418266713 0 0) 0x7f0bbc09bcf0 con 0x7f0bbc0a0da0 2024-09-16T05:36:51.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient(hunting): my global_id is 94150 2024-09-16T05:36:51.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bbc120c70 con 0x7f0bbc0a0da0 2024-09-16T05:36:51.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3335305856 0 0) 0x7f0bbc09c1b0 con 0x7f0bbc09e230 2024-09-16T05:36:51.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 >> v1:172.21.15.63:6789/0 conn(0x7f0bbc0a0da0 legacy=0x7f0bbc1cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 >> v1:172.21.15.6:6789/0 conn(0x7f0bbc09aa30 legacy=0x7f0bbc133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 monclient: found mon.c 2024-09-16T05:36:51.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc120f50 con 0x7f0bbc09e230 2024-09-16T05:36:51.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 5 monclient: authenticate success, global_id 4898 2024-09-16T05:36:51.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: _renew_subs 2024-09-16T05:36:51.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0bbc121230 con 0x7f0bbc09e230 2024-09-16T05:36:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: _renew_subs 2024-09-16T05:36:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0bbc133f80 con 0x7f0bbc09e230 2024-09-16T05:36:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bb0003620 con 0x7f0bbc09e230 2024-09-16T05:36:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bb0003970 con 0x7f0bbc09e230 2024-09-16T05:36:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bb9ffb640 4 set_mon_vals no callback set 2024-09-16T05:36:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 10 monclient: dump: 2024-09-16T05:36:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.071 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.072 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b9c005090 con 0x7f0bbc09e230 2024-09-16T05:36:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0bb0003b90 con 0x7f0bbc09e230 2024-09-16T05:36:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.026+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4387+0+0 (unknown 4150075110 0 0) 0x7f0bb00443e0 con 0x7f0bbc09e230 2024-09-16T05:36:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.030+0000 7f0bbb7fe640 10 monclient: get_auth_request con 0x7f0ba8036be0 auth_method 0 2024-09-16T05:36:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.034+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0bb0018fe0 con 0x7f0bbc09e230 2024-09-16T05:36:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.034+0000 7f0ba3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.034+0000 7f0ba3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:51.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.258+0000 7f0bc2140640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:36:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.258+0000 7f0bc2140640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:36:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.258+0000 7f0bc2140640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.258+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0b9c005380 con 0x7f0bbc09e230 2024-09-16T05:36:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0ba3fff640 1 -- 172.21.15.6:0/4102280743 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2612530308) 0x7f0bb001d8a0 con 0x7f0bbc09e230 2024-09-16T05:36:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0ba3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:36:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0ba3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:51.265 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:36:51.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0ba8036be0 legacy=0x7f0ba80390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0bc2140640 10 monclient: shutdown 2024-09-16T05:36:51.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0bc2140640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 >> v1:172.21.15.6:6790/0 conn(0x7f0bbc09e230 legacy=0x7f0bbc1cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 shutdown_connections 2024-09-16T05:36:51.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 >> 172.21.15.6:0/4102280743 conn(0x7f0bbc0a1650 msgr2=0x7f0bbc134610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 shutdown_connections 2024-09-16T05:36:51.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.262+0000 7f0bc2140640 1 -- 172.21.15.6:0/4102280743 wait complete. 2024-09-16T05:36:51.282 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:36:51.283 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:36:51.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:36:51.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:36:51.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.343+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 v62) ==== 110+0+0 (unknown 4111457707 0 0) 0x7f3590015000 con 0x7f359c0a4790 2024-09-16T05:36:51.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.343+0000 7f35997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.343+0000 7f35997fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.402+0000 7fcff0d74640 10 monclient: get_monmap_and_config 2024-09-16T05:36:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.402+0000 7fcff0d74640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.402+0000 7fcff0d74640 10 monclient: monmap: 2024-09-16T05:36:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:51.406382+0000 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:51.406382+0000 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:51.407 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 Processor -- start 2024-09-16T05:36:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- start start 2024-09-16T05:36:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient: init 2024-09-16T05:36:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient(hunting): picked mon.noname-b con 0x7fcfec09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient(hunting): picked mon.noname-c con 0x7fcfec09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient(hunting): picked mon.noname-a con 0x7fcfec0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfec0a1370 con 0x7fcfec0a0da0 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfec09ead0 con 0x7fcfec09e230 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfec09edb0 con 0x7fcfec09aa30 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient(hunting): authenticate will time out at 1073.213439s 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfead9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcfec0a0da0 0x7fcfec13d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59346/0 (socket says 172.21.15.6:59346) 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfea59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcfec09aa30 0x7fcfec09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50920/0 (socket says 172.21.15.6:50920) 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfead9d640 1 -- 172.21.15.6:0/2130638801 learned_addr learned my addr 172.21.15.6:0/2130638801 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134833794 0 0) 0x7fcfec0a1370 con 0x7fcfec0a0da0 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient(hunting): my global_id is 4908 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfd8003530 con 0x7fcfec0a0da0 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308795877 0 0) 0x7fcfec09edb0 con 0x7fcfec09aa30 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient(hunting): my global_id is 4901 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfec0a1370 con 0x7fcfec09aa30 2024-09-16T05:36:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2474726743 0 0) 0x7fcfd8003530 con 0x7fcfec0a0da0 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 >> v1:172.21.15.6:6790/0 conn(0x7fcfec09aa30 legacy=0x7fcfec09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 >> v1:172.21.15.63:6789/0 conn(0x7fcfec09e230 legacy=0x7fcfec1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 monclient: found mon.noname-a 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfec09dd50 con 0x7fcfec0a0da0 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfe40019a0 con 0x7fcfec0a0da0 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 5 monclient: authenticate success, global_id 4908 2024-09-16T05:36:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: dump: 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.419 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcfe4001c50 con 0x7fcfec0a0da0 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 1 -- 172.21.15.6:0/2130638801 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfe4002060 con 0x7fcfec0a0da0 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcfe959a640 10 monclient: dump: 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 4 set_mon_vals no callback set 2024-09-16T05:36:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient: shutdown 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- 172.21.15.6:0/2130638801 >> v1:172.21.15.6:6789/0 conn(0x7fcfec0a0da0 legacy=0x7fcfec13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- 172.21.15.6:0/2130638801 shutdown_connections 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- 172.21.15.6:0/2130638801 >> 172.21.15.6:0/2130638801 conn(0x7fcfec0a1650 msgr2=0x7fcfec0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- 172.21.15.6:0/2130638801 shutdown_connections 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 1 -- 172.21.15.6:0/2130638801 wait complete. 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.406+0000 7fcff0d74640 10 monclient: monmap: 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 1 Processor -- start 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 1 -- start start 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient: init 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient(hunting): picked mon.noname-c con 0x7fcfec09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient(hunting): picked mon.noname-b con 0x7fcfec09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient(hunting): picked mon.noname-a con 0x7fcfec0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfec0a0230 con 0x7fcfec0a0da0 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfec0740c0 con 0x7fcfec09e230 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfec13b5a0 con 0x7fcfec09aa30 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient(hunting): authenticate will time out at 1073.216758s 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfea59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcfec09aa30 0x7fcfec09bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50932/0 (socket says 172.21.15.6:50932) 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfea59c640 1 -- 172.21.15.6:0/2479894924 learned_addr learned my addr 172.21.15.6:0/2479894924 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfead9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcfec0a0da0 0x7fcfec09c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59350/0 (socket says 172.21.15.6:59350) 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923834032 0 0) 0x7fcfec13b5a0 con 0x7fcfec09aa30 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): my global_id is 4904 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfcc003530 con 0x7fcfec09aa30 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1534572505 0 0) 0x7fcfec09dd50 con 0x7fcfec0a0da0 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): my global_id is 4911 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfec13b5a0 con 0x7fcfec0a0da0 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 469536301 0 0) 0x7fcfcc003530 con 0x7fcfec09aa30 2024-09-16T05:36:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfec09dd50 con 0x7fcfec09aa30 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfd40019a0 con 0x7fcfec09aa30 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.426 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3870142632 0 0) 0x7fcfdc001240 con 0x7fcfec09e230 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): my global_id is 94156 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfcc003530 con 0x7fcfec09e230 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 325730904 0 0) 0x7fcfec13b5a0 con 0x7fcfec0a0da0 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfdc001240 con 0x7fcfec0a0da0 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfe4001fa0 con 0x7fcfec0a0da0 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 709498428 0 0) 0x7fcfec09dd50 con 0x7fcfec09aa30 2024-09-16T05:36:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 >> v1:172.21.15.63:6789/0 conn(0x7fcfec09e230 legacy=0x7fcfec09c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 >> v1:172.21.15.6:6789/0 conn(0x7fcfec0a0da0 legacy=0x7fcfec09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 monclient: found mon.c 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfec13b880 con 0x7fcfec09aa30 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 5 monclient: authenticate success, global_id 4904 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient: _renew_subs 2024-09-16T05:36:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcfec1351c0 con 0x7fcfec09aa30 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient: _renew_subs 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcfec1d4900 con 0x7fcfec09aa30 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcfd40047f0 con 0x7fcfec09aa30 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfd4004b40 con 0x7fcfec09aa30 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe8d99640 4 set_mon_vals no callback set 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: dump: 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.430 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.431 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.431 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcfd00065d0 con 0x7fcfd0002870 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcfd400f8e0 con 0x7fcfec09aa30 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7fcfd40444f0 con 0x7fcfec09aa30 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1402148403 0 0) 0x7fcfd00065d0 con 0x7fcfd0002870 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient(hunting): my global_id is 4914 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfec09dd50 con 0x7fcfd0002870 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfead9d640 10 monclient: get_auth_request con 0x7fcfcc036c40 auth_method 0 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2626687419 0 0) 0x7fcfec09dd50 con 0x7fcfd0002870 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfd00065d0 con 0x7fcfd0002870 2024-09-16T05:36:51.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfdc0018a0 con 0x7fcfd0002870 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: dump: 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.433 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3445968403 0 0) 0x7fcfd00065d0 con 0x7fcfd0002870 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfd000a160 con 0x7fcfd0002870 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcfdc0016c0 con 0x7fcfd0002870 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 >> v1:172.21.15.6:6789/0 conn(0x7fcfd0002870 legacy=0x7fcfd0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.410+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 >> v1:172.21.15.6:6789/0 conn(0x7fcfd0002870 legacy=0x7fcfd0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcfd0009e50 con 0x7fcfd0009110 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114059588 0 0) 0x7fcfd0009e50 con 0x7fcfd0009110 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient(hunting): my global_id is 4917 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfd00065d0 con 0x7fcfd0009110 2024-09-16T05:36:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4187703057 0 0) 0x7fcfd00065d0 con 0x7fcfd0009110 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfd0009e50 con 0x7fcfd0009110 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfdc003820 con 0x7fcfd0009110 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient: dump: 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.437 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3073102237 0 0) 0x7fcfd0009e50 con 0x7fcfd0009110 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcfd000c3e0 con 0x7fcfd0009110 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 2231373731) 0x7fcfdc003ae0 con 0x7fcfd0009110 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 >> v1:172.21.15.6:6789/0 conn(0x7fcfd0009110 legacy=0x7fcfd00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfe2ffd640 1 -- 172.21.15.6:0/2479894924 >> v1:172.21.15.6:6789/0 conn(0x7fcfd0009110 legacy=0x7fcfd00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 14, 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 140474, 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.439 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.440 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.441 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:36:51.442 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.443 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:36:51.448 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.449 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.450 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.452 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:36:51.452 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:36:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 >> v1:172.21.15.63:6812/3643207211 conn(0x7fcfcc036c40 legacy=0x7fcfcc039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 10 monclient: shutdown 2024-09-16T05:36:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 >> v1:172.21.15.6:6790/0 conn(0x7fcfec09aa30 legacy=0x7fcfec09bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfead9d640 1 -- 172.21.15.6:0/2479894924 reap_dead start 2024-09-16T05:36:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcfead9d640 1 -- 172.21.15.6:0/2479894924 reap_dead start 2024-09-16T05:36:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 shutdown_connections 2024-09-16T05:36:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 >> 172.21.15.6:0/2479894924 conn(0x7fcfec0a1650 msgr2=0x7fcfec13dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 shutdown_connections 2024-09-16T05:36:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.418+0000 7fcff0d74640 1 -- 172.21.15.6:0/2479894924 wait complete. 2024-09-16T05:36:51.454 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:36:51.455 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:36:51.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:36:51.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.447+0000 7f35a2e17640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.447+0000 7f35a2e17640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.447+0000 7f35a2e17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:51.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.447+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) -- 0x7f357c005be0 con 0x7f359c0a4790 2024-09-16T05:36:51.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.447+0000 7f35997fa640 1 -- 172.21.15.63:0/966920107 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v62) ==== 134+0+0 (unknown 2382817871 0 0) 0x7f359001db50 con 0x7f359c0a4790 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.447+0000 7f35997fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.447+0000 7f35997fa640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:already renamed to 'foo' 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.451+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3588036a60 legacy=0x7f3588038f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.451+0000 7f35a2e17640 10 monclient: shutdown 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.451+0000 7f35a2e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.451+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 >> v1:172.21.15.63:6789/0 conn(0x7f359c0a4790 legacy=0x7f359c13b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.451+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 shutdown_connections 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.451+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 >> 172.21.15.63:0/966920107 conn(0x7f359c075470 msgr2=0x7f359c072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.451+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 shutdown_connections 2024-09-16T05:36:51.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.451+0000 7f35a2e17640 1 -- 172.21.15.63:0/966920107 wait complete. 2024-09-16T05:36:51.470 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-16T05:36:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient: get_monmap_and_config 2024-09-16T05:36:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient: monmap: 2024-09-16T05:36:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:51.571878+0000 2024-09-16T05:36:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:51.571878+0000 2024-09-16T05:36:51.572 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:51.573 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:51.573 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 1 Processor -- start 2024-09-16T05:36:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 1 -- start start 2024-09-16T05:36:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient: init 2024-09-16T05:36:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:36:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient(hunting): picked mon.noname-c con 0x7fcf9c0a0df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient(hunting): picked mon.noname-a con 0x7fcf9c135330 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient(hunting): picked mon.noname-b con 0x7fcf9c138e40 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf9c09d2f0 con 0x7fcf9c135330 2024-09-16T05:36:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf9c13d960 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf9c0a0560 con 0x7fcf9c138e40 2024-09-16T05:36:51.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 10 monclient(hunting): authenticate will time out at 1073.378984s 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf99d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcf9c135330 0x7fcf9c137720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59374/0 (socket says 172.21.15.6:59374) 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf99d9b640 1 -- 172.21.15.6:0/3888850920 learned_addr learned my addr 172.21.15.6:0/3888850920 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9ad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcf9c138e40 0x7fcf9c13b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50944/0 (socket says 172.21.15.6:50944) 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1014917972 0 0) 0x7fcf9c09d2f0 con 0x7fcf9c135330 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient(hunting): my global_id is 4920 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf84003530 con 0x7fcf9c135330 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2748901285 0 0) 0x7fcf9c0a0560 con 0x7fcf9c138e40 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient(hunting): my global_id is 4907 2024-09-16T05:36:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf9c09d2f0 con 0x7fcf9c138e40 2024-09-16T05:36:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 571098554 0 0) 0x7fcf9c09d2f0 con 0x7fcf9c138e40 2024-09-16T05:36:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 >> v1:172.21.15.63:6789/0 conn(0x7fcf9c0a0df0 legacy=0x7fcf9c09de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 >> v1:172.21.15.6:6789/0 conn(0x7fcf9c135330 legacy=0x7fcf9c137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 monclient: found mon.noname-b 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf9c0a0730 con 0x7fcf9c138e40 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 5 monclient: authenticate success, global_id 4907 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf94001600 con 0x7fcf9c138e40 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:51.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcf9959a640 10 monclient: dump: 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.570+0000 7fcfa0b76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf940018e0 con 0x7fcf9c138e40 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf9959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 4 set_mon_vals no callback set 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf9959a640 1 -- 172.21.15.6:0/3888850920 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf94001cf0 con 0x7fcf9c138e40 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf9959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf9959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf9959a640 10 monclient: dump: 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.589 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: shutdown 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- 172.21.15.6:0/3888850920 >> v1:172.21.15.6:6790/0 conn(0x7fcf9c138e40 legacy=0x7fcf9c13b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- 172.21.15.6:0/3888850920 shutdown_connections 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- 172.21.15.6:0/3888850920 >> 172.21.15.6:0/3888850920 conn(0x7fcf9c0a1650 msgr2=0x7fcf9c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- 172.21.15.6:0/3888850920 shutdown_connections 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- 172.21.15.6:0/3888850920 wait complete. 2024-09-16T05:36:51.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: monmap: 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 Processor -- start 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- start start 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: init 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient(hunting): picked mon.noname-c con 0x7fcf9c0a0df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient(hunting): picked mon.noname-b con 0x7fcf9c135330 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient(hunting): picked mon.noname-a con 0x7fcf9c138e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf9c13cc20 con 0x7fcf9c138e40 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf9c13cf00 con 0x7fcf9c135330 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf9c1341b0 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient(hunting): authenticate will time out at 1073.382120s 2024-09-16T05:36:51.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf9a59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcf9c0a0df0 0x7fcf9c133390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50958/0 (socket says 172.21.15.6:50958) 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf9a59c640 1 -- 172.21.15.6:0/4092282296 learned_addr learned my addr 172.21.15.6:0/4092282296 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4102455513 0 0) 0x7fcf9c1341b0 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): my global_id is 4910 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf7c003530 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516578019 0 0) 0x7fcf9c13cc20 con 0x7fcf9c138e40 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): my global_id is 4923 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf9c1341b0 con 0x7fcf9c138e40 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 797507999 0 0) 0x7fcf7c003530 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf9c13cc20 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf8c001ca0 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): dump: 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.594 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.595 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.595 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.595 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999711026 0 0) 0x7fcf9c13cf00 con 0x7fcf9c135330 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient(hunting): my global_id is 94162 2024-09-16T05:36:51.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf7c003530 con 0x7fcf9c135330 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3494148702 0 0) 0x7fcf9c1341b0 con 0x7fcf9c138e40 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf9c13cf00 con 0x7fcf9c138e40 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1041660195 0 0) 0x7fcf9c13cc20 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 >> v1:172.21.15.63:6789/0 conn(0x7fcf9c135330 legacy=0x7fcf9c133aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 >> v1:172.21.15.6:6789/0 conn(0x7fcf9c138e40 legacy=0x7fcf9c1d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 monclient: found mon.c 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf9c13bcc0 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 5 monclient: authenticate success, global_id 4910 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: _renew_subs 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf9c13c4c0 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: _renew_subs 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf9c134e10 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcf9c0740c0 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf8c001ff0 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf8c003e70 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf8affd640 10 monclient: dump: 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.600 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.574+0000 7fcf98d99640 4 set_mon_vals no callback set 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcf8c004010 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797406325 0 0) 0x7fcf9c0740c0 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient(hunting): my global_id is 4913 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf9c13cc20 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7fcf8c0447a0 con 0x7fcf9c0a0df0 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf9ad9d640 10 monclient: get_auth_request con 0x7fcf7c036c10 auth_method 0 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 407268534 0 0) 0x7fcf9c13cc20 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf9c0740c0 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf900018a0 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient: dump: 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.602 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 122942337 0 0) 0x7fcf9c0740c0 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf9c02dbb0 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcf90001b00 con 0x7fcf9c02d7d0 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 >> v1:172.21.15.6:6790/0 conn(0x7fcf9c02d7d0 legacy=0x7fcf9c062ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.578+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 >> v1:172.21.15.6:6790/0 conn(0x7fcf9c02d7d0 legacy=0x7fcf9c062ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:51.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcfa0b76640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcf9c09b180 con 0x7fcf9c06eee0 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006246097 0 0) 0x7fcf9c09b180 con 0x7fcf9c06eee0 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 10 monclient(hunting): my global_id is 4916 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf9c0740c0 con 0x7fcf9c06eee0 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 87088409 0 0) 0x7fcf9c0740c0 con 0x7fcf9c06eee0 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf9c09b180 con 0x7fcf9c06eee0 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcf900049e0 con 0x7fcf9c06eee0 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 10 monclient: dump: 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 169318710 0 0) 0x7fcf9c09b180 con 0x7fcf9c06eee0 2024-09-16T05:36:51.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.582+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf9c0a3cf0 con 0x7fcf9c06eee0 2024-09-16T05:36:51.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 2668356257) 0x7fcf90006030 con 0x7fcf9c06eee0 2024-09-16T05:36:51.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcf8affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:36:51.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcf8affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:51.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 >> v1:172.21.15.6:6790/0 conn(0x7fcf9c06eee0 legacy=0x7fcf9c1d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcf8affd640 1 -- 172.21.15.6:0/4092282296 >> v1:172.21.15.6:6790/0 conn(0x7fcf9c06eee0 legacy=0x7fcf9c1d7330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:51.609 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:36:51.609 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:36:51.609 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 14, 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 140642, 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:36:51.610 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:36:51.611 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.612 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:36:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.614 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.615 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.617 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.618 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.619 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.620 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:36:51.621 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:36:51.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 >> v1:172.21.15.63:6812/3643207211 conn(0x7fcf7c036c10 legacy=0x7fcf7c0390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcfa0b76640 10 monclient: shutdown 2024-09-16T05:36:51.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcfa0b76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 >> v1:172.21.15.6:6790/0 conn(0x7fcf9c0a0df0 legacy=0x7fcf9c133390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcf9ad9d640 1 -- 172.21.15.6:0/4092282296 reap_dead start 2024-09-16T05:36:51.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcf9ad9d640 1 -- 172.21.15.6:0/4092282296 reap_dead start 2024-09-16T05:36:51.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 shutdown_connections 2024-09-16T05:36:51.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 >> 172.21.15.6:0/4092282296 conn(0x7fcf9c0a1650 msgr2=0x7fcf9c0a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 shutdown_connections 2024-09-16T05:36:51.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.586+0000 7fcfa0b76640 1 -- 172.21.15.6:0/4092282296 wait complete. 2024-09-16T05:36:51.625 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:36:51.625 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:36:51.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:36:51.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient: get_monmap_and_config 2024-09-16T05:36:51.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient: monmap: 2024-09-16T05:36:51.630 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:51.593785+0000 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:51.593785+0000 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 1 Processor -- start 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 1 -- start start 2024-09-16T05:36:51.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient: init 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient(hunting): picked mon.noname-b con 0x7f23540a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient(hunting): picked mon.noname-c con 0x7f23540a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient(hunting): picked mon.noname-a con 0x7f23540a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23540a53b0 con 0x7f23540a4600 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23540a0290 con 0x7f23540a07a0 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23540a0dc0 con 0x7f23540a3b10 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f235be85640 10 monclient(hunting): authenticate will time out at 1068.371337s 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2359420640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f23540a07a0 0x7f235413be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47322/0 (socket says 172.21.15.63:47322) 2024-09-16T05:36:51.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2359420640 1 -- 172.21.15.63:0/1545818060 learned_addr learned my addr 172.21.15.63:0/1545818060 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400695604 0 0) 0x7f23540a0290 con 0x7f23540a07a0 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): my global_id is 94165 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2340003530 con 0x7f23540a07a0 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1408796200 0 0) 0x7f23540a53b0 con 0x7f23540a4600 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): my global_id is 4926 2024-09-16T05:36:51.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23540a0290 con 0x7f23540a4600 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799002470 0 0) 0x7f23540a0dc0 con 0x7f23540a3b10 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 10 monclient(hunting): my global_id is 4919 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.591+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23540a53b0 con 0x7f23540a3b10 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1611915810 0 0) 0x7f2340003530 con 0x7f23540a07a0 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 >> v1:172.21.15.6:6790/0 conn(0x7f23540a3b10 legacy=0x7f23540a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 >> v1:172.21.15.6:6789/0 conn(0x7f23540a4600 legacy=0x7f235413f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 1 monclient: found mon.noname-c 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:36:51.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23540a10a0 con 0x7f23540a07a0 2024-09-16T05:36:51.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2344001620 con 0x7f23540a07a0 2024-09-16T05:36:51.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 5 monclient: authenticate success, global_id 94165 2024-09-16T05:36:51.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: dump: 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23440018d0 con 0x7f23540a07a0 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 1 -- 172.21.15.63:0/1545818060 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2344001ce0 con 0x7f23540a07a0 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 4 set_mon_vals no callback set 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2358c1f640 10 monclient: dump: 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.637 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: shutdown 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- 172.21.15.63:0/1545818060 >> v1:172.21.15.63:6789/0 conn(0x7f23540a07a0 legacy=0x7f235413be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- 172.21.15.63:0/1545818060 shutdown_connections 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- 172.21.15.63:0/1545818060 >> 172.21.15.63:0/1545818060 conn(0x7f2354075470 msgr2=0x7f2354075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- 172.21.15.63:0/1545818060 shutdown_connections 2024-09-16T05:36:51.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- 172.21.15.63:0/1545818060 wait complete. 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: monmap: 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 Processor -- start 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- start start 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: init 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient(hunting): picked mon.noname-a con 0x7f23540a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient(hunting): picked mon.noname-b con 0x7f23540a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient(hunting): picked mon.noname-c con 0x7f23540a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23541237f0 con 0x7f23540a07a0 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2354122ff0 con 0x7f23540a3b10 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2354125ae0 con 0x7f23540a4600 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient(hunting): authenticate will time out at 1068.374478s 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2359420640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f23540a3b10 0x7f23541d0150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47338/0 (socket says 172.21.15.63:47338) 2024-09-16T05:36:51.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f2359420640 1 -- 172.21.15.63:0/949326850 learned_addr learned my addr 172.21.15.63:0/949326850 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3479565530 0 0) 0x7f2354122ff0 con 0x7f23540a3b10 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): my global_id is 94168 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2338003530 con 0x7f23540a3b10 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155424912 0 0) 0x7f2354125ae0 con 0x7f23540a4600 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): my global_id is 4922 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2354122ff0 con 0x7f23540a4600 2024-09-16T05:36:51.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1308299462 0 0) 0x7f23541237f0 con 0x7f23540a07a0 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): my global_id is 4929 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2354125ae0 con 0x7f23540a07a0 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1944964024 0 0) 0x7f2338003530 con 0x7f23540a3b10 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23541237f0 con 0x7f23540a3b10 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2344001a80 con 0x7f23540a3b10 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): dump: 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:51.642 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 503356571 0 0) 0x7f2354122ff0 con 0x7f23540a4600 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2338003530 con 0x7f23540a4600 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f234c001140 con 0x7f23540a4600 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): dump: 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.644 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2635862785 0 0) 0x7f2354125ae0 con 0x7f23540a07a0 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2354122ff0 con 0x7f23540a07a0 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2350001af0 con 0x7f23540a07a0 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient(hunting): dump: 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.645 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3929733726 0 0) 0x7f23541237f0 con 0x7f23540a3b10 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.6:6790/0 conn(0x7f23540a4600 legacy=0x7f23541d3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.6:6789/0 conn(0x7f23540a07a0 legacy=0x7f2354137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 monclient: found mon.b 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2354137820 con 0x7f23540a3b10 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 5 monclient: authenticate success, global_id 94168 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: _renew_subs 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2354124350 con 0x7f23540a3b10 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: _renew_subs 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f23541c7a60 con 0x7f23540a3b10 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23440018d0 con 0x7f23540a3b10 2024-09-16T05:36:51.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2344004c40 con 0x7f23540a3b10 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234bfff640 4 set_mon_vals no callback set 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.595+0000 7f234a7fc640 10 monclient: dump: 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.648 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.599+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2344004de0 con 0x7f23540a3b10 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.599+0000 7f235be85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.599+0000 7f235be85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.599+0000 7f235be85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.599+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f233c005090 con 0x7f23540a3b10 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.599+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7f2344044490 con 0x7f23540a3b10 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.599+0000 7f2359c21640 10 monclient: get_auth_request con 0x7f2338036dc0 auth_method 0 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.603+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2344015450 con 0x7f23540a3b10 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.603+0000 7f234a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.603+0000 7f234a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient: get_monmap_and_config 2024-09-16T05:36:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient: monmap: 2024-09-16T05:36:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:51.767245+0000 2024-09-16T05:36:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:51.767245+0000 2024-09-16T05:36:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:51.768 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:51.768 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.767+0000 7f235be85640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.767+0000 7f235be85640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.767+0000 7f235be85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:36:51.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.767+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) -- 0x7f233c005380 con 0x7f23540a3b10 2024-09-16T05:36:51.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.771+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v62) ==== 134+0+0 (unknown 2382817871 0 0) 0x7f2344014d50 con 0x7f23540a3b10 2024-09-16T05:36:51.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.771+0000 7f234a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.771+0000 7f234a7fc640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo' 2024-09-16T05:36:51.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 Processor -- start 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- start start 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient: init 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient(hunting): picked mon.noname-a con 0x7ff82009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient(hunting): picked mon.noname-c con 0x7ff82009b520 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient(hunting): picked mon.noname-b con 0x7ff820141690 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8200a1350 con 0x7ff82009aa30 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82009cdf0 con 0x7ff82009b520 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82009dad0 con 0x7ff820141690 2024-09-16T05:36:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient(hunting): authenticate will time out at 1073.573893s 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81f7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff820141690 0x7ff820143a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50994/0 (socket says 172.21.15.6:50994) 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81f7fe640 1 -- 172.21.15.6:0/3753425546 learned_addr learned my addr 172.21.15.6:0/3753425546 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff82009aa30 0x7ff82009ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59402/0 (socket says 172.21.15.6:59402) 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902067888 0 0) 0x7ff82009dad0 con 0x7ff820141690 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient(hunting): my global_id is 4925 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff808003530 con 0x7ff820141690 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4128041729 0 0) 0x7ff810001050 con 0x7ff82009aa30 2024-09-16T05:36:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient(hunting): my global_id is 4932 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff82009dad0 con 0x7ff82009aa30 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2342622007 0 0) 0x7ff808003530 con 0x7ff820141690 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 >> v1:172.21.15.63:6789/0 conn(0x7ff82009b520 legacy=0x7ff82013ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 >> v1:172.21.15.6:6789/0 conn(0x7ff82009aa30 legacy=0x7ff82009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 monclient: found mon.noname-b 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8200a10d0 con 0x7ff820141690 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff8180019a0 con 0x7ff820141690 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: dump: 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.781 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 5 monclient: authenticate success, global_id 4925 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff818001c50 con 0x7ff820141690 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 1 -- 172.21.15.6:0/3753425546 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff818002060 con 0x7ff820141690 2024-09-16T05:36:51.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 4 set_mon_vals no callback set 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff81dffb640 10 monclient: dump: 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.783 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 10 monclient: shutdown 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- 172.21.15.6:0/3753425546 >> v1:172.21.15.6:6790/0 conn(0x7ff820141690 legacy=0x7ff820143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- 172.21.15.6:0/3753425546 shutdown_connections 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- 172.21.15.6:0/3753425546 >> 172.21.15.6:0/3753425546 conn(0x7ff8200a1650 msgr2=0x7ff8200a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- 172.21.15.6:0/3753425546 shutdown_connections 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.766+0000 7ff82561d640 1 -- 172.21.15.6:0/3753425546 wait complete. 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: monmap: 2024-09-16T05:36:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 1 Processor -- start 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 1 -- start start 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: init 2024-09-16T05:36:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient(hunting): picked mon.noname-b con 0x7ff82009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient(hunting): picked mon.noname-c con 0x7ff82009b520 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient(hunting): picked mon.noname-a con 0x7ff820141690 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff820141230 con 0x7ff820141690 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8200a3b60 con 0x7ff82009aa30 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8200740c0 con 0x7ff82009b520 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient(hunting): authenticate will time out at 1073.577057s 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff81f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff820141690 0x7ff8201d83d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59416/0 (socket says 172.21.15.6:59416) 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff81e7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff82009b520 0x7ff8201d4ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51004/0 (socket says 172.21.15.6:51004) 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff81f7fe640 1 -- 172.21.15.6:0/3717640494 learned_addr learned my addr 172.21.15.6:0/3717640494 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225370488 0 0) 0x7ff820141230 con 0x7ff820141690 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): my global_id is 4935 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff80c003530 con 0x7ff820141690 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1202958929 0 0) 0x7ff8200740c0 con 0x7ff82009b520 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): my global_id is 4928 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff820141230 con 0x7ff82009b520 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 716697489 0 0) 0x7ff820141230 con 0x7ff82009b520 2024-09-16T05:36:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8200740c0 con 0x7ff82009b520 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff814001840 con 0x7ff82009b520 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): dump: 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2450222630 0 0) 0x7ff80c003530 con 0x7ff820141690 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff820141230 con 0x7ff820141690 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff818001fa0 con 0x7ff820141690 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): dump: 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 33867878 0 0) 0x7ff8200740c0 con 0x7ff82009b520 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 >> v1:172.21.15.63:6789/0 conn(0x7ff82009aa30 legacy=0x7ff82013ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 >> v1:172.21.15.6:6789/0 conn(0x7ff820141690 legacy=0x7ff8201d83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 monclient: found mon.c 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff820140ad0 con 0x7ff82009b520 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 5 monclient: authenticate success, global_id 4928 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: _renew_subs 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff820126810 con 0x7ff82009b520 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: _renew_subs 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff820125080 con 0x7ff82009b520 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff814003620 con 0x7ff82009b520 2024-09-16T05:36:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff8140037d0 con 0x7ff82009b520 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff81d7fa640 4 set_mon_vals no callback set 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient: dump: 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff814003970 con 0x7ff82009b520 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff8000065d0 con 0x7ff800002870 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7ff814044260 con 0x7ff82009b520 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff81f7fe640 10 monclient: get_auth_request con 0x7ff80c036af0 auth_method 0 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2455541461 0 0) 0x7ff8000065d0 con 0x7ff800002870 2024-09-16T05:36:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 10 monclient(hunting): my global_id is 94177 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.770+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8200740c0 con 0x7ff800002870 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2852139364 0 0) 0x7ff8200740c0 con 0x7ff800002870 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8000065d0 con 0x7ff800002870 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff810001560 con 0x7ff800002870 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 10 monclient: dump: 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3874573274 0 0) 0x7ff8000065d0 con 0x7ff800002870 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff80000a160 con 0x7ff800002870 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff810001950 con 0x7ff800002870 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 >> v1:172.21.15.63:6789/0 conn(0x7ff800002870 legacy=0x7ff800004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.774+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 >> v1:172.21.15.63:6789/0 conn(0x7ff800002870 legacy=0x7ff800004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.778+0000 7ff82561d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.778+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff800009e50 con 0x7ff800009110 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713932606 0 0) 0x7ff800009e50 con 0x7ff800009110 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient(hunting): my global_id is 94180 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8000065d0 con 0x7ff800009110 2024-09-16T05:36:51.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 212768279 0 0) 0x7ff8000065d0 con 0x7ff800009110 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff800009e50 con 0x7ff800009110 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff810004990 con 0x7ff800009110 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient: dump: 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.798 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2682156981 0 0) 0x7ff800009e50 con 0x7ff800009110 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff80000c3e0 con 0x7ff800009110 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3960 (unknown 0 0 16287953) 0x7ff8100047f0 con 0x7ff800009110 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 >> v1:172.21.15.63:6789/0 conn(0x7ff800009110 legacy=0x7ff8000095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff8077fe640 1 -- 172.21.15.6:0/3717640494 >> v1:172.21.15.63:6789/0 conn(0x7ff800009110 legacy=0x7ff8000095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:51.799 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 14, 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 3091, 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 3, 2024-09-16T05:36:51.800 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.801 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.802 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:51.803 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:36:51.804 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:36:51.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.807 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:36:51.808 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:36:51.809 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:36:51.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:36:51.811 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:36:51.811 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:36:51.811 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:36:51.811 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:36:51.811 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:36:51.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:36:51.811 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:36:51.811 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:36:51.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff80c036af0 legacy=0x7ff80c038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff82561d640 10 monclient: shutdown 2024-09-16T05:36:51.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff82561d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 >> v1:172.21.15.6:6790/0 conn(0x7ff82009b520 legacy=0x7ff8201d4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff81f7fe640 1 -- 172.21.15.6:0/3717640494 reap_dead start 2024-09-16T05:36:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff81f7fe640 1 -- 172.21.15.6:0/3717640494 reap_dead start 2024-09-16T05:36:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 shutdown_connections 2024-09-16T05:36:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 >> 172.21.15.6:0/3717640494 conn(0x7ff8200a1650 msgr2=0x7ff8200a2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 shutdown_connections 2024-09-16T05:36:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.782+0000 7ff82561d640 1 -- 172.21.15.6:0/3717640494 wait complete. 2024-09-16T05:36:51.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:36:51.814 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:36:51.814 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:36:51.814 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:36:51.814 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:36:51.814 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:36:51.814 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:36:51.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f2359420640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.63:6789/0 conn(0x7f23540a3b10 legacy=0x7f23541d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:36:51.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f2359420640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.63:6789/0 conn(0x7f23540a3b10 legacy=0x7f23541d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:36:51.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f2359420640 1 --1- 172.21.15.63:0/949326850 >> v1:172.21.15.63:6789/0 conn(0x7f23540a3b10 0x7f23541d0150 :-1 s=OPENED pgs=46 cs=1 l=1).handle_message read tag failed 2024-09-16T05:36:51.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f2359420640 1 --1- 172.21.15.63:0/949326850 >> v1:172.21.15.63:6789/0 conn(0x7f23540a3b10 0x7f23541d0150 :-1 s=OPENED pgs=46 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:36:51.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.63:6789/0 2024-09-16T05:36:51.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.63:6789/0 conn(0x7f23540a3b10 legacy=0x7f23541d0150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:36:51.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): picked mon.b con 0x7f233803e950 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): picked mon.c con 0x7f23540a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): picked mon.a con 0x7f23540a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f23541237f0 con 0x7f23540a07a0 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2354125ae0 con 0x7f233803e950 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f233804be20 con 0x7f23540a4600 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f2359420640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.63:6789/0 conn(0x7f233803e950 legacy=0x7f2338040d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649413782 0 0) 0x7f23541237f0 con 0x7f23540a07a0 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:36:51.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f233804bff0 con 0x7f23540a07a0 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269197851 0 0) 0x7f233804be20 con 0x7f23540a4600 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f23541237f0 con 0x7f23540a4600 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 2265671531 0 0) 0x7f233804bff0 con 0x7f23540a07a0 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.6:6790/0 conn(0x7f23540a4600 legacy=0x7f2338044720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.63:6789/0 conn(0x7f233803e950 legacy=0x7f2338040d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 monclient: found mon.a 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:51.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7f233804d020 con 0x7f23540a07a0 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2350005f00 con 0x7f23540a07a0 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: dump: 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.824 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:51.825 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.825 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.825 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.825 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:51.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2350005d60 con 0x7f23540a07a0 2024-09-16T05:36:51.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.815+0000 7f234b7fe640 4 set_mon_vals no callback set 2024-09-16T05:36:51.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.819+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2350006130 con 0x7f23540a07a0 2024-09-16T05:36:51.828 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:36:51.828 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:36:51.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:36:51.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.867+0000 7f235be85640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.867+0000 7f235be85640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.867+0000 7f235be85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:51.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.867+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) -- 0x7f23300050f0 con 0x7f23540a07a0 2024-09-16T05:36:51.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.867+0000 7f234a7fc640 1 -- 172.21.15.63:0/949326850 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v62) ==== 134+0+0 (unknown 2382817871 0 0) 0x7f235001c520 con 0x7f23540a07a0 2024-09-16T05:36:51.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.867+0000 7f234a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T05:36:51.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.867+0000 7f234a7fc640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-16T05:36:51.871 INFO:tasks.workunit.client.0.smithi063.stderr:already renamed to 'foo' 2024-09-16T05:36:51.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.871+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2338036dc0 legacy=0x7f2338039280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.871+0000 7f235be85640 10 monclient: shutdown 2024-09-16T05:36:51.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.871+0000 7f235be85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.871+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 >> v1:172.21.15.6:6789/0 conn(0x7f23540a07a0 legacy=0x7f2338048260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.871+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 shutdown_connections 2024-09-16T05:36:51.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.871+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 >> 172.21.15.63:0/949326850 conn(0x7f2354075470 msgr2=0x7f23541bcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.871+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 shutdown_connections 2024-09-16T05:36:51.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:51.871+0000 7f235be85640 1 -- 172.21.15.63:0/949326850 wait complete. 2024-09-16T05:36:51.886 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rename bar-asdf bar 2024-09-16T05:36:51.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 10 monclient: get_monmap_and_config 2024-09-16T05:36:51.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 10 monclient: monmap: 2024-09-16T05:36:51.941 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.941 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.941 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:51.941159+0000 2024-09-16T05:36:51.942 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:51.941159+0000 2024-09-16T05:36:51.942 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.942 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.942 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.942 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:51.942 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:51.942 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 1 Processor -- start 2024-09-16T05:36:51.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 1 -- start start 2024-09-16T05:36:51.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 10 monclient: init 2024-09-16T05:36:51.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.938+0000 7f5574505640 10 monclient(hunting): picked mon.noname-b con 0x7f556c0a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): picked mon.noname-c con 0x7f556c0a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): picked mon.noname-a con 0x7f556c0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556c06e520 con 0x7f556c0a1a00 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556c0a0780 con 0x7f556c0a4290 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556c0a4c70 con 0x7f556c0a37a0 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): authenticate will time out at 1073.748141s 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5571aa0640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f556c0a4290 legacy=0x7f556c13bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5572aa2640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f556c0a1a00 0x7f556c13f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59428/0 (socket says 172.21.15.6:59428) 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5572aa2640 1 -- 172.21.15.6:0/1615770335 learned_addr learned my addr 172.21.15.6:0/1615770335 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f55722a1640 1 --1- 172.21.15.6:0/1615770335 >> v1:172.21.15.6:6790/0 conn(0x7f556c0a37a0 0x7f556c0a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51006/0 (socket says 172.21.15.6:51006) 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799281733 0 0) 0x7f556c0a4c70 con 0x7f556c0a37a0 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient(hunting): my global_id is 4931 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5558003530 con 0x7f556c0a37a0 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1117651146 0 0) 0x7f556c06e520 con 0x7f556c0a1a00 2024-09-16T05:36:51.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient(hunting): my global_id is 4938 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f556c0a4c70 con 0x7f556c0a1a00 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2914854962 0 0) 0x7f556c0a4c70 con 0x7f556c0a1a00 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 >> v1:172.21.15.6:6790/0 conn(0x7f556c0a37a0 legacy=0x7f556c0a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 >> v1:172.21.15.63:6789/0 conn(0x7f556c0a4290 legacy=0x7f556c13bab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 monclient: found mon.noname-a 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556c0a65f0 con 0x7f556c0a1a00 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5564001900 con 0x7f556c0a1a00 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 5 monclient: authenticate success, global_id 4938 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: dump: 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.949 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5564001be0 con 0x7f556c0a1a00 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 1 -- 172.21.15.6:0/1615770335 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5564001ff0 con 0x7f556c0a1a00 2024-09-16T05:36:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f557129f640 10 monclient: dump: 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.951 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 4 set_mon_vals no callback set 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient: shutdown 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- 172.21.15.6:0/1615770335 >> v1:172.21.15.6:6789/0 conn(0x7f556c0a1a00 legacy=0x7f556c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- 172.21.15.6:0/1615770335 shutdown_connections 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- 172.21.15.6:0/1615770335 >> 172.21.15.6:0/1615770335 conn(0x7f556c0a58a0 msgr2=0x7f556c0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- 172.21.15.6:0/1615770335 shutdown_connections 2024-09-16T05:36:51.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- 172.21.15.6:0/1615770335 wait complete. 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient: build_initial_monmap 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient: monmap: 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 Processor -- start 2024-09-16T05:36:51.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- start start 2024-09-16T05:36:51.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient: init 2024-09-16T05:36:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): picked mon.noname-c con 0x7f556c0a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): picked mon.noname-b con 0x7f556c0a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): picked mon.noname-a con 0x7f556c0a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556c132e20 con 0x7f556c0a4290 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556c135d70 con 0x7f556c0a37a0 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556c0a0100 con 0x7f556c0a1a00 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5574505640 10 monclient(hunting): authenticate will time out at 1073.751691s 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f55722a1640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f556c0a1a00 0x7f556c132000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51008/0 (socket says 172.21.15.6:51008) 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f55722a1640 1 -- 172.21.15.6:0/508463878 learned_addr learned my addr 172.21.15.6:0/508463878 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5571aa0640 1 -- 172.21.15.6:0/508463878 >> v1:172.21.15.63:6789/0 conn(0x7f556c0a37a0 legacy=0x7f556c132710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.942+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988089670 0 0) 0x7f556c0a0100 con 0x7f556c0a1a00 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): my global_id is 4934 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5550003530 con 0x7f556c0a1a00 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2207396260 0 0) 0x7f556c132e20 con 0x7f556c0a4290 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): my global_id is 4941 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f556c0a0100 con 0x7f556c0a4290 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2672556640 0 0) 0x7f5550003530 con 0x7f556c0a1a00 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f556c132e20 con 0x7f556c0a1a00 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1814699987 0 0) 0x7f556c0a0100 con 0x7f556c0a4290 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5550003530 con 0x7f556c0a4290 2024-09-16T05:36:51.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5568001a40 con 0x7f556c0a1a00 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5564001f30 con 0x7f556c0a4290 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient(hunting): dump: 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1447443282 0 0) 0x7f556c132e20 con 0x7f556c0a1a00 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 >> v1:172.21.15.63:6789/0 conn(0x7f556c0a37a0 legacy=0x7f556c132710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 >> v1:172.21.15.6:6789/0 conn(0x7f556c0a4290 legacy=0x7f556c1d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 monclient: found mon.c 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556c0a8120 con 0x7f556c0a1a00 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient: _finish_auth 0 2024-09-16T05:36:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 5 monclient: authenticate success, global_id 4934 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 10 monclient: _renew_subs 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f556c0a7bd0 con 0x7f556c0a1a00 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 10 monclient: _renew_subs 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f556c135a90 con 0x7f556c0a1a00 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5568004aa0 con 0x7f556c0a1a00 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5568004c50 con 0x7f556c0a1a00 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5570a9e640 4 set_mon_vals no callback set 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 10 monclient: dump: 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f556800f9f0 con 0x7f556c0a1a00 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5554005090 con 0x7f556c0a1a00 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7f5568044570 con 0x7f556c0a1a00 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.946+0000 7f5571aa0640 10 monclient: get_auth_request con 0x7f5550036a10 auth_method 0 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.950+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f55680154d0 con 0x7f556c0a1a00 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.950+0000 7f5562ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:51.950+0000 7f5562ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient: get_monmap_and_config 2024-09-16T05:36:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient: build_initial_monmap 2024-09-16T05:36:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient: monmap: 2024-09-16T05:36:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:36:52.009170+0000 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:36:52.009170+0000 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:52.010 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 1 Processor -- start 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 1 -- start start 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient: init 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient(hunting): picked mon.noname-b con 0x7f62bc09ca20 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient(hunting): picked mon.noname-c con 0x7f62bc09ce00 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient(hunting): picked mon.noname-a con 0x7f62bc139250 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62bc0a1bd0 con 0x7f62bc139250 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62bc09dc40 con 0x7f62bc09ce00 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62bc09e2a0 con 0x7f62bc09ca20 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62c15ea640 10 monclient(hunting): authenticate will time out at 1068.787153s 2024-09-16T05:36:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62ba7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f62bc09ce00 legacy=0x7f62bc09d210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62bb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62bc139250 0x7f62bc13b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37344/0 (socket says 172.21.15.63:37344) 2024-09-16T05:36:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62bb7fe640 1 -- 172.21.15.63:0/3247376430 learned_addr learned my addr 172.21.15.63:0/3247376430 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435508445 0 0) 0x7f62bc0a1bd0 con 0x7f62bc139250 2024-09-16T05:36:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62b9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62b9ffb640 10 monclient(hunting): my global_id is 4944 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.007+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62a4003530 con 0x7f62bc139250 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1994423132 0 0) 0x7f62a4003530 con 0x7f62bc139250 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 >> v1:172.21.15.6:6790/0 conn(0x7f62bc09ca20 legacy=0x7f62bc0a14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 >> v1:172.21.15.63:6789/0 conn(0x7f62bc09ce00 legacy=0x7f62bc09d210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 1 monclient: found mon.noname-a 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62bc09f2c0 con 0x7f62bc139250 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 5 monclient: authenticate success, global_id 4944 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62b4001620 con 0x7f62bc139250 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: dump: 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:52.019 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62b4001980 con 0x7f62bc139250 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 1 -- 172.21.15.63:0/3247376430 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62b40048b0 con 0x7f62bc139250 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:52.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 4 set_mon_vals no callback set 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b9ffb640 10 monclient: dump: 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:52.021 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: shutdown 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- 172.21.15.63:0/3247376430 >> v1:172.21.15.6:6789/0 conn(0x7f62bc139250 legacy=0x7f62bc13b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- 172.21.15.63:0/3247376430 shutdown_connections 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- 172.21.15.63:0/3247376430 >> 172.21.15.63:0/3247376430 conn(0x7f62bc0a2420 msgr2=0x7f62bc0a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- 172.21.15.63:0/3247376430 shutdown_connections 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- 172.21.15.63:0/3247376430 wait complete. 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: build_initial_monmap 2024-09-16T05:36:52.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: monmap: 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 Processor -- start 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- start start 2024-09-16T05:36:52.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: init 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient(hunting): picked mon.noname-a con 0x7f62bc09ca20 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient(hunting): picked mon.noname-b con 0x7f62bc09ce00 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient(hunting): picked mon.noname-c con 0x7f62bc139250 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62bc133c60 con 0x7f62bc09ca20 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62bc134f80 con 0x7f62bc09ce00 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62bc12ead0 con 0x7f62bc139250 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient(hunting): authenticate will time out at 1068.790460s 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62ba7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f62bc09ce00 legacy=0x7f62bc12e3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:52.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62bb7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f62bc139250 0x7f62bc1d3c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55280/0 (socket says 172.21.15.63:55280) 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62bb7fe640 1 -- 172.21.15.63:0/2410880128 learned_addr learned my addr 172.21.15.63:0/2410880128 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310579183 0 0) 0x7f62bc09e2a0 con 0x7f62bc09ca20 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient(hunting): my global_id is 4947 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62a8003530 con 0x7f62bc09ca20 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4283761398 0 0) 0x7f62a8003530 con 0x7f62bc09ca20 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62bc09e2a0 con 0x7f62bc09ca20 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62ac001b70 con 0x7f62bc09ca20 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:52.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient(hunting): dump: 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 663361462 0 0) 0x7f62bc09e2a0 con 0x7f62bc09ca20 2024-09-16T05:36:52.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 >> v1:172.21.15.6:6790/0 conn(0x7f62bc139250 legacy=0x7f62bc1d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 >> v1:172.21.15.63:6789/0 conn(0x7f62bc09ce00 legacy=0x7f62bc12e3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 monclient: found mon.a 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62bc135540 con 0x7f62bc09ca20 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 5 monclient: authenticate success, global_id 4947 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: _renew_subs 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:52.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f62bc1381a0 con 0x7f62bc09ca20 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: _renew_subs 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f62bc135260 con 0x7f62bc09ca20 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62ac0039a0 con 0x7f62bc09ca20 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62ac003cf0 con 0x7f62bc09ca20 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62b8ff9640 4 set_mon_vals no callback set 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.011+0000 7f62a37fe640 10 monclient: dump: 2024-09-16T05:36:52.028 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:36:52.029 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:52.029 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:52.029 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.015+0000 7f62c15ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.015+0000 7f62c15ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.015+0000 7f62c15ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.015+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f629c005090 con 0x7f62bc09ca20 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.015+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f62ac003e90 con 0x7f62bc09ca20 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.015+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7f62ac0435b0 con 0x7f62bc09ca20 2024-09-16T05:36:52.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.015+0000 7f62ba7fc640 10 monclient: get_auth_request con 0x7f62a8036a40 auth_method 0 2024-09-16T05:36:52.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.023+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f62bc133c60 con 0x7f62bc09ca20 2024-09-16T05:36:52.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.023+0000 7f62a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:52.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.023+0000 7f62a37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:36:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:36:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:36:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5554005380 con 0x7f556c0a1a00 2024-09-16T05:36:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5562ffd640 1 -- 172.21.15.6:0/508463878 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2550364847) 0x7f5568014dd0 con 0x7f556c0a1a00 2024-09-16T05:36:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5562ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:36:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5562ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:52.185 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:36:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5550036a10 legacy=0x7f5550038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 10 monclient: shutdown 2024-09-16T05:36:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 >> v1:172.21.15.6:6790/0 conn(0x7f556c0a1a00 legacy=0x7f556c132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 shutdown_connections 2024-09-16T05:36:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 >> 172.21.15.6:0/508463878 conn(0x7f556c0a58a0 msgr2=0x7f556c0a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 shutdown_connections 2024-09-16T05:36:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:52.182+0000 7f5574505640 1 -- 172.21.15.6:0/508463878 wait complete. 2024-09-16T05:36:52.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.199+0000 7f62c15ea640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T05:36:52.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.199+0000 7f62c15ea640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T05:36:52.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.199+0000 7f62c15ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:52.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:52.199+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) -- 0x7f629c005380 con 0x7f62bc09ca20 2024-09-16T05:36:55.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:55.011+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:36:55.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:55.011+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:55.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:55.011+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:55.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:55.011+0000 7f62a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:55.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:36:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient: get_monmap_and_config 2024-09-16T05:36:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient: build_initial_monmap 2024-09-16T05:36:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient: monmap: 2024-09-16T05:36:55.337 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:55.337 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:55.337 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:55.336011+0000 2024-09-16T05:36:55.337 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:55.336011+0000 2024-09-16T05:36:55.337 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:55.337 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:55.338 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:55.338 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:55.338 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:55.338 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 1 Processor -- start 2024-09-16T05:36:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 1 -- start start 2024-09-16T05:36:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient: init 2024-09-16T05:36:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient(hunting): picked mon.noname-c con 0x7fbc8809bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient(hunting): picked mon.noname-b con 0x7fbc8809bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient(hunting): picked mon.noname-a con 0x7fbc881397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc880a1350 con 0x7fbc881397f0 2024-09-16T05:36:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc880a10d0 con 0x7fbc8809bb40 2024-09-16T05:36:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc8809d940 con 0x7fbc8809bf20 2024-09-16T05:36:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient(hunting): authenticate will time out at 1077.142643s 2024-09-16T05:36:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc877fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbc881397f0 0x7fbc8813bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59436/0 (socket says 172.21.15.6:59436) 2024-09-16T05:36:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc867fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbc8809bf20 0x7fbc8809c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51012/0 (socket says 172.21.15.6:51012) 2024-09-16T05:36:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc877fe640 1 -- 172.21.15.6:0/1141433524 learned_addr learned my addr 172.21.15.6:0/1141433524 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc86ffd640 1 -- 172.21.15.6:0/1141433524 >> v1:172.21.15.63:6789/0 conn(0x7fbc8809bb40 legacy=0x7fbc8809e600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 964311551 0 0) 0x7fbc880a1350 con 0x7fbc881397f0 2024-09-16T05:36:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:36:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient(hunting): my global_id is 4950 2024-09-16T05:36:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc74003530 con 0x7fbc881397f0 2024-09-16T05:36:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 243689936 0 0) 0x7fbc74003530 con 0x7fbc881397f0 2024-09-16T05:36:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 >> v1:172.21.15.6:6790/0 conn(0x7fbc8809bf20 legacy=0x7fbc8809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 >> v1:172.21.15.63:6789/0 conn(0x7fbc8809bb40 legacy=0x7fbc8809e600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:36:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:36:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 monclient: found mon.noname-a 2024-09-16T05:36:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:55.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:36:55.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc8809dc20 con 0x7fbc881397f0 2024-09-16T05:36:55.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: _finish_auth 0 2024-09-16T05:36:55.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:36:55.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:55.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc800019a0 con 0x7fbc881397f0 2024-09-16T05:36:55.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:55.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:55.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: dump: 2024-09-16T05:36:55.358 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:55.358 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:55.358 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:55.358 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:55.359 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:55.359 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:55.359 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:55.359 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:55.359 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:55.359 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 5 monclient: authenticate success, global_id 4950 2024-09-16T05:36:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:36:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc80001c90 con 0x7fbc881397f0 2024-09-16T05:36:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 1 -- 172.21.15.6:0/1141433524 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc800020a0 con 0x7fbc881397f0 2024-09-16T05:36:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient: get_monmap_and_config success 2024-09-16T05:36:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 4 set_mon_vals no callback set 2024-09-16T05:36:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc85ffb640 10 monclient: dump: 2024-09-16T05:36:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 10 monclient: shutdown 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.334+0000 7fbc8d79f640 1 -- 172.21.15.6:0/1141433524 >> v1:172.21.15.6:6789/0 conn(0x7fbc881397f0 legacy=0x7fbc8813bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- 172.21.15.6:0/1141433524 shutdown_connections 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- 172.21.15.6:0/1141433524 >> 172.21.15.6:0/1141433524 conn(0x7fbc880a1650 msgr2=0x7fbc880a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- 172.21.15.6:0/1141433524 shutdown_connections 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- 172.21.15.6:0/1141433524 wait complete. 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: build_initial_monmap 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: monmap: 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 Processor -- start 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- start start 2024-09-16T05:36:55.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: init 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient(hunting): picked mon.noname-c con 0x7fbc8809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient(hunting): picked mon.noname-a con 0x7fbc8809bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient(hunting): picked mon.noname-b con 0x7fbc881397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc8811b910 con 0x7fbc8809bf20 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc8811c7f0 con 0x7fbc881397f0 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc880740c0 con 0x7fbc8809bb40 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient(hunting): authenticate will time out at 1077.145555s 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc86ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbc8809bb40 0x7fbc880706c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51014/0 (socket says 172.21.15.6:51014) 2024-09-16T05:36:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc86ffd640 1 -- 172.21.15.6:0/2026031809 learned_addr learned my addr 172.21.15.6:0/2026031809 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc877fe640 1 -- 172.21.15.6:0/2026031809 >> v1:172.21.15.63:6789/0 conn(0x7fbc881397f0 legacy=0x7fbc880714e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744588188 0 0) 0x7fbc8811b910 con 0x7fbc8809bf20 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient(hunting): my global_id is 4953 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc68003530 con 0x7fbc8809bf20 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3605767607 0 0) 0x7fbc68003530 con 0x7fbc8809bf20 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc8811b910 con 0x7fbc8809bf20 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc700017f0 con 0x7fbc8809bf20 2024-09-16T05:36:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient(hunting): dump: 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1839079889 0 0) 0x7fbc8811b910 con 0x7fbc8809bf20 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 >> v1:172.21.15.6:6790/0 conn(0x7fbc8809bb40 legacy=0x7fbc880706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 >> v1:172.21.15.63:6789/0 conn(0x7fbc881397f0 legacy=0x7fbc880714e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 monclient: found mon.a 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc88072310 con 0x7fbc8809bf20 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:55.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 5 monclient: authenticate success, global_id 4953 2024-09-16T05:36:55.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: _renew_subs 2024-09-16T05:36:55.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:55.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc8811c510 con 0x7fbc8809bf20 2024-09-16T05:36:55.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: _renew_subs 2024-09-16T05:36:55.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:55.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc8811b370 con 0x7fbc8809bf20 2024-09-16T05:36:55.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc700047f0 con 0x7fbc8809bf20 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbc70004bc0 con 0x7fbc8809bf20 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc857fa640 4 set_mon_vals no callback set 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 10 monclient: dump: 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc6c005090 con 0x7fbc8809bf20 2024-09-16T05:36:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbc70004d60 con 0x7fbc8809bf20 2024-09-16T05:36:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.338+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7fbc70044490 con 0x7fbc8809bf20 2024-09-16T05:36:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.342+0000 7fbc86ffd640 10 monclient: get_auth_request con 0x7fbc680369d0 auth_method 0 2024-09-16T05:36:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.342+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbc70015390 con 0x7fbc8809bf20 2024-09-16T05:36:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.342+0000 7fbc7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:36:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.342+0000 7fbc7f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:36:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.570+0000 7fbc8d79f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:36:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.570+0000 7fbc8d79f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:36:55.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.570+0000 7fbc8d79f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:36:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.570+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbc6c005380 con 0x7fbc8809bf20 2024-09-16T05:36:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.570+0000 7fbc7f7fe640 1 -- 172.21.15.6:0/2026031809 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2505430056) 0x7fbc70019040 con 0x7fbc8809bf20 2024-09-16T05:36:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.570+0000 7fbc7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:36:55.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.570+0000 7fbc7f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:36:55.575 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:36:55.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.574+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbc680369d0 legacy=0x7fbc68038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:55.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.574+0000 7fbc8d79f640 10 monclient: shutdown 2024-09-16T05:36:55.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.574+0000 7fbc8d79f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:36:55.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.574+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 >> v1:172.21.15.6:6789/0 conn(0x7fbc8809bf20 legacy=0x7fbc88070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:36:55.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.574+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 shutdown_connections 2024-09-16T05:36:55.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.574+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 >> 172.21.15.6:0/2026031809 conn(0x7fbc880a1650 msgr2=0x7fbc880a41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:36:55.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.574+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 shutdown_connections 2024-09-16T05:36:55.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:55.574+0000 7fbc8d79f640 1 -- 172.21.15.6:0/2026031809 wait complete. 2024-09-16T05:36:56.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:56.011+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:36:56.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:56.011+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:56.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:56.011+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:57.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:57.011+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:36:57.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:57.011+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:57.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:57.011+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:58.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:58.011+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:36:58.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:58.011+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:58.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:58.011+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:58.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:36:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient: get_monmap_and_config 2024-09-16T05:36:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient: build_initial_monmap 2024-09-16T05:36:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient: monmap: 2024-09-16T05:36:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:36:58.724521+0000 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:36:58.724521+0000 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:36:58.725 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:36:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 1 Processor -- start 2024-09-16T05:36:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 1 -- start start 2024-09-16T05:36:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient: init 2024-09-16T05:36:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient: _reopen_session rank -1 2024-09-16T05:36:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:36:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient(hunting): picked mon.noname-c con 0x7f8b6409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:36:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient(hunting): picked mon.noname-a con 0x7f8b640a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b640a1370 con 0x7f8b640a0da0 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6409ead0 con 0x7f8b6409aa30 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6409edb0 con 0x7f8b6409e230 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient(hunting): _renew_subs 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6bc01640 10 monclient(hunting): authenticate will time out at 1080.531135s 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6999d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b6409ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6919c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8b6409e230 0x7f8b641399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51028/0 (socket says 172.21.15.6:51028) 2024-09-16T05:36:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.722+0000 7f8b6919c640 1 -- 172.21.15.6:0/3835877622 learned_addr learned my addr 172.21.15.6:0/3835877622 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:36:58.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:58.926+0000 7f8b6999d640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b6409ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:36:59.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:59.015+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:36:59.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:59.015+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:36:59.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:36:59.015+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:36:59.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:36:59.330+0000 7f8b6999d640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b6409ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:00.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:00.011+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:37:00.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:00.011+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:00.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:00.011+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:00.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:00.134+0000 7f8b6999d640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b6409ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:01.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:01.015+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:37:01.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:01.015+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:01.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:01.015+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 10 monclient(hunting): tick 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 1 monclient(hunting): continuing hunt 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.6:6790/0 conn(0x7f8b6409e230 legacy=0x7f8b641399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b6409ae10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.6:6789/0 conn(0x7f8b640a0da0 legacy=0x7f8b6413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 10 monclient(hunting): picked mon.noname-c con 0x7f8b6409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:01.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 10 monclient(hunting): picked mon.noname-a con 0x7f8b50006e60 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b6999d640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6409dd50 con 0x7f8b50006e60 2024-09-16T05:37:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6409ead0 con 0x7f8b6409aa30 2024-09-16T05:37:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b500103b0 con 0x7f8b6409e230 2024-09-16T05:37:01.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.722+0000 7f8b637fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:01.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:01.926+0000 7f8b6999d640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:02.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:02.011+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:37:02.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:02.011+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:02.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:02.011+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:02.330+0000 7f8b6999d640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:03.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:03.011+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:37:03.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:03.011+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:03.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:03.011+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:03.134+0000 7f8b6999d640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:04.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:04.011+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:37:04.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:04.011+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:04.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:04.011+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:04.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 10 monclient(hunting): tick 2024-09-16T05:37:04.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 1 monclient(hunting): continuing hunt 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.6:6790/0 conn(0x7f8b6409e230 legacy=0x7f8b5000cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.6:6789/0 conn(0x7f8b50006e60 legacy=0x7f8b500092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 10 monclient(hunting): picked mon.noname-a con 0x7f8b50010660 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 10 monclient(hunting): picked mon.noname-c con 0x7f8b6409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:04.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b50011660 con 0x7f8b50010660 2024-09-16T05:37:04.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6409ead0 con 0x7f8b6409aa30 2024-09-16T05:37:04.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 1 -- 172.21.15.6:0/3835877622 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b50011370 con 0x7f8b6409e230 2024-09-16T05:37:04.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b637fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:04.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.726+0000 7f8b6a19e640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:04.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:04.930+0000 7f8b6a19e640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:05.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:05.015+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:37:05.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:05.015+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:05.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:05.015+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:05.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:05.015+0000 7f62a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:05.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:05.334+0000 7f8b6a19e640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:06.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:06.015+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:37:06.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:06.015+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:06.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:06.015+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.138+0000 7f8b6a19e640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:06.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.382+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583158927 0 0) 0x7f8b50011370 con 0x7f8b6409e230 2024-09-16T05:37:06.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.382+0000 7f8b6899b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.382+0000 7f8b6899b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.382+0000 7f8b6899b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.382+0000 7f8b6899b640 10 monclient(hunting): my global_id is 4937 2024-09-16T05:37:06.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.382+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b4c003530 con 0x7f8b6409e230 2024-09-16T05:37:06.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3215283286 0 0) 0x7f8b4c003530 con 0x7f8b6409e230 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b50005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.6:6789/0 conn(0x7f8b50010660 legacy=0x7f8b50010a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: _finish_hunting 0 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 1 monclient: found mon.noname-b 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b500092f0 con 0x7f8b6409e230 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: _finish_auth 0 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: _check_auth_tickets 2024-09-16T05:37:06.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b580015a0 con 0x7f8b6409e230 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 5 monclient: authenticate success, global_id 4937 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: dump: 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.390 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b58001800 con 0x7f8b6409e230 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 4 set_mon_vals no callback set 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 1 -- 172.21.15.6:0/3835877622 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b58001bd0 con 0x7f8b6409e230 2024-09-16T05:37:06.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6899b640 10 monclient: dump: 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.392 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.393 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 10 monclient: shutdown 2024-09-16T05:37:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 1 -- 172.21.15.6:0/3835877622 >> v1:172.21.15.6:6790/0 conn(0x7f8b6409e230 legacy=0x7f8b5000cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 1 -- 172.21.15.6:0/3835877622 shutdown_connections 2024-09-16T05:37:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 1 -- 172.21.15.6:0/3835877622 >> 172.21.15.6:0/3835877622 conn(0x7f8b640a1650 msgr2=0x7f8b640a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 1 -- 172.21.15.6:0/3835877622 shutdown_connections 2024-09-16T05:37:06.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 1 -- 172.21.15.6:0/3835877622 wait complete. 2024-09-16T05:37:06.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 10 monclient: build_initial_monmap 2024-09-16T05:37:06.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 10 monclient: monmap: 2024-09-16T05:37:06.396 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:37:06.396 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:06.396 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 1 Processor -- start 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 1 -- start start 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.386+0000 7f8b6bc01640 10 monclient: init 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:06.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient(hunting): picked mon.noname-c con 0x7f8b50010660 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient(hunting): picked mon.noname-a con 0x7f8b6409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b641203d0 con 0x7f8b6409e230 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b64122940 con 0x7f8b6409aa30 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6409c1f0 con 0x7f8b50010660 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient(hunting): authenticate will time out at 1088.196367s 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6919c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b6411fcc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:06.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6a19e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8b6409e230 0x7f8b641d3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33410/0 (socket says 172.21.15.6:33410) 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6a19e640 1 -- 172.21.15.6:0/296241167 learned_addr learned my addr 172.21.15.6:0/296241167 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3457169746 0 0) 0x7f8b6409c1f0 con 0x7f8b50010660 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient(hunting): my global_id is 4940 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b50011370 con 0x7f8b50010660 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2269951122 0 0) 0x7f8b50011370 con 0x7f8b50010660 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b6409c1f0 con 0x7f8b50010660 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b54004300 con 0x7f8b50010660 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:06.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient(hunting): dump: 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3554114963 0 0) 0x7f8b6409c1f0 con 0x7f8b50010660 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 >> v1:172.21.15.63:6789/0 conn(0x7f8b6409aa30 legacy=0x7f8b6411fcc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 >> v1:172.21.15.6:6789/0 conn(0x7f8b6409e230 legacy=0x7f8b641d3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 monclient: found mon.c 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b64122380 con 0x7f8b50010660 2024-09-16T05:37:06.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 5 monclient: authenticate success, global_id 4940 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient: _renew_subs 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b64122660 con 0x7f8b50010660 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient: _renew_subs 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b64070c90 con 0x7f8b50010660 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b540047a0 con 0x7f8b50010660 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b54004b70 con 0x7f8b50010660 2024-09-16T05:37:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 10 monclient: dump: 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b63fff640 4 set_mon_vals no callback set 2024-09-16T05:37:06.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8b54004d90 con 0x7f8b50010660 2024-09-16T05:37:06.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:06.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7f8b54044540 con 0x7f8b50010660 2024-09-16T05:37:06.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:06.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:06.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b44005090 con 0x7f8b50010660 2024-09-16T05:37:06.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.390+0000 7f8b6919c640 10 monclient: get_auth_request con 0x7f8b50037bf0 auth_method 0 2024-09-16T05:37:06.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.394+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8b54015440 con 0x7f8b50010660 2024-09-16T05:37:06.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.394+0000 7f8b627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:06.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.394+0000 7f8b627fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.618+0000 7f8b6bc01640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:37:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.618+0000 7f8b6bc01640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:37:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.618+0000 7f8b6bc01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.618+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8b44005380 con 0x7f8b50010660 2024-09-16T05:37:06.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.618+0000 7f8b627fc640 1 -- 172.21.15.6:0/296241167 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 646139772) 0x7f8b540190f0 con 0x7f8b50010660 2024-09-16T05:37:06.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.618+0000 7f8b627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:37:06.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.618+0000 7f8b627fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:06.622 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":18,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:37:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.622+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8b50037bf0 legacy=0x7f8b5003a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.622+0000 7f8b6bc01640 10 monclient: shutdown 2024-09-16T05:37:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.622+0000 7f8b6bc01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.622+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 >> v1:172.21.15.6:6790/0 conn(0x7f8b50010660 legacy=0x7f8b6411f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.622+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 shutdown_connections 2024-09-16T05:37:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.622+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 >> 172.21.15.6:0/296241167 conn(0x7f8b640a1650 msgr2=0x7f8b6409c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.622+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 shutdown_connections 2024-09-16T05:37:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.622+0000 7f8b6bc01640 1 -- 172.21.15.6:0/296241167 wait complete. 2024-09-16T05:37:06.639 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:37:06.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:37:06.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:37:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient: get_monmap_and_config 2024-09-16T05:37:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient: build_initial_monmap 2024-09-16T05:37:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient: monmap: 2024-09-16T05:37:06.763 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:37:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:37:06.763236+0000 2024-09-16T05:37:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:37:06.763236+0000 2024-09-16T05:37:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.764 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:06.765 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:06.765 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 Processor -- start 2024-09-16T05:37:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- start start 2024-09-16T05:37:06.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient: init 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient(hunting): picked mon.noname-c con 0x7f8ed409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed40a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed40a1370 con 0x7f8ed40a0da0 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed409ead0 con 0x7f8ed409aa30 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed409edb0 con 0x7f8ed409e230 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient(hunting): authenticate will time out at 1088.569589s 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed37fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8ed409aa30 legacy=0x7f8ed409ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:06.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed2ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8ed409e230 0x7f8ed41399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:43070/0 (socket says 172.21.15.6:43070) 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed2ffd640 1 -- 172.21.15.6:0/1830867364 learned_addr learned my addr 172.21.15.6:0/1830867364 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed3fff640 1 --1- 172.21.15.6:0/1830867364 >> v1:172.21.15.6:6789/0 conn(0x7f8ed40a0da0 0x7f8ed413d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33422/0 (socket says 172.21.15.6:33422) 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715090906 0 0) 0x7f8ed409edb0 con 0x7f8ed409e230 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient(hunting): my global_id is 4943 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ebc003530 con 0x7f8ed409e230 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430981244 0 0) 0x7f8ed40a1370 con 0x7f8ed40a0da0 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient(hunting): my global_id is 4956 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ed409edb0 con 0x7f8ed40a0da0 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4291632604 0 0) 0x7f8ebc003530 con 0x7f8ed409e230 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 >> v1:172.21.15.63:6789/0 conn(0x7f8ed409aa30 legacy=0x7f8ed409ae10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 >> v1:172.21.15.6:6789/0 conn(0x7f8ed40a0da0 legacy=0x7f8ed413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 monclient: found mon.noname-b 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed409dd50 con 0x7f8ed409e230 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:06.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ec8001550 con 0x7f8ed409e230 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: dump: 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.776 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 5 monclient: authenticate success, global_id 4943 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ec80017b0 con 0x7f8ed409e230 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 1 -- 172.21.15.6:0/1830867364 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ec8001b00 con 0x7f8ed409e230 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 4 set_mon_vals no callback set 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8ed27fc640 10 monclient: dump: 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.777 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 10 monclient: shutdown 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- 172.21.15.6:0/1830867364 >> v1:172.21.15.6:6790/0 conn(0x7f8ed409e230 legacy=0x7f8ed41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- 172.21.15.6:0/1830867364 shutdown_connections 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- 172.21.15.6:0/1830867364 >> 172.21.15.6:0/1830867364 conn(0x7f8ed40a1650 msgr2=0x7f8ed40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:06.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- 172.21.15.6:0/1830867364 shutdown_connections 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.762+0000 7f8eda03c640 1 -- 172.21.15.6:0/1830867364 wait complete. 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: build_initial_monmap 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: monmap: 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:06.779 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 1 Processor -- start 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 1 -- start start 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: init 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient(hunting): picked mon.noname-c con 0x7f8ed40a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed4138770 con 0x7f8ed409aa30 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed4138a50 con 0x7f8ed409e230 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed406e820 con 0x7f8ed40a0da0 2024-09-16T05:37:06.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient(hunting): authenticate will time out at 1088.573237s 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ed37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ed409aa30 0x7f8ed4136310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33432/0 (socket says 172.21.15.6:33432) 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ed37fe640 1 -- 172.21.15.6:0/4113586448 learned_addr learned my addr 172.21.15.6:0/4113586448 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ed2ffd640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.63:6789/0 conn(0x7f8ed409e230 legacy=0x7f8ed41cde10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3968710683 0 0) 0x7f8ed4138770 con 0x7f8ed409aa30 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): my global_id is 4959 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ec0003530 con 0x7f8ed409aa30 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1196974389 0 0) 0x7f8ed406e820 con 0x7f8ed40a0da0 2024-09-16T05:37:06.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): my global_id is 4946 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ed4138770 con 0x7f8ed40a0da0 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 360324910 0 0) 0x7f8ec0003530 con 0x7f8ed409aa30 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ed406e820 con 0x7f8ed409aa30 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ec40040c0 con 0x7f8ed409aa30 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): dump: 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.782 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 999448117 0 0) 0x7f8ed4138770 con 0x7f8ed40a0da0 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ec0003530 con 0x7f8ed40a0da0 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ecc001d20 con 0x7f8ed40a0da0 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:06.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): dump: 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3405908002 0 0) 0x7f8ed406e820 con 0x7f8ed409aa30 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.6:6790/0 conn(0x7f8ed40a0da0 legacy=0x7f8ed41d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.63:6789/0 conn(0x7f8ed409e230 legacy=0x7f8ed41cde10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 monclient: found mon.a 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed4138490 con 0x7f8ed409aa30 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: _finish_auth 0 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 5 monclient: authenticate success, global_id 4959 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: _renew_subs 2024-09-16T05:37:06.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed4135e80 con 0x7f8ed409aa30 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: _renew_subs 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ed4136a20 con 0x7f8ed409aa30 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ec4004320 con 0x7f8ed409aa30 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ec4004670 con 0x7f8ed409aa30 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ed17fa640 4 set_mon_vals no callback set 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: dump: 2024-09-16T05:37:06.786 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8eb40065d0 con 0x7f8eb4002870 2024-09-16T05:37:06.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8ec4004810 con 0x7f8ed409aa30 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7f8ec40442c0 con 0x7f8ed409aa30 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1104295730 0 0) 0x7f8eb40065d0 con 0x7f8eb4002870 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient(hunting): my global_id is 4962 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ed406e820 con 0x7f8eb4002870 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2568202709 0 0) 0x7f8ed406e820 con 0x7f8eb4002870 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eb40065d0 con 0x7f8eb4002870 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ec80015b0 con 0x7f8eb4002870 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 10 monclient: dump: 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ed3fff640 10 monclient: get_auth_request con 0x7f8ec0036aa0 auth_method 0 2024-09-16T05:37:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2213131560 0 0) 0x7f8eb40065d0 con 0x7f8eb4002870 2024-09-16T05:37:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.766+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8eb400a160 con 0x7f8eb4002870 2024-09-16T05:37:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.770+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8ec80016b0 con 0x7f8eb4002870 2024-09-16T05:37:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.770+0000 7f8ebbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.770+0000 7f8ebbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.770+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.6:6789/0 conn(0x7f8eb4002870 legacy=0x7f8eb4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.770+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.6:6789/0 conn(0x7f8eb4002870 legacy=0x7f8eb4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:37:06.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8eda03c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:37:06.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8eb4009e30 con 0x7f8eb40090f0 2024-09-16T05:37:06.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731426593 0 0) 0x7f8eb4009e30 con 0x7f8eb40090f0 2024-09-16T05:37:06.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient(hunting): my global_id is 4965 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb40065d0 con 0x7f8eb40090f0 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 206446842 0 0) 0x7f8eb40065d0 con 0x7f8eb40090f0 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eb4009e30 con 0x7f8eb40090f0 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ec8004a50 con 0x7f8eb40090f0 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient: dump: 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1338994731 0 0) 0x7f8eb4009e30 con 0x7f8eb40090f0 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8eb400c3a0 con 0x7f8eb40090f0 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4287 (unknown 0 0 2355583340) 0x7f8ec8004cc0 con 0x7f8eb40090f0 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:37:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.6:6789/0 conn(0x7f8eb40090f0 legacy=0x7f8eb4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8ebbfff640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.6:6789/0 conn(0x7f8eb40090f0 legacy=0x7f8eb4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:37:06.794 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:37:06.794 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:37:06.794 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:37:06.794 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:37:06.794 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 18, 2024-09-16T05:37:06.794 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 155830, 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:37:06.795 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:37:06.796 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:37:06.797 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:37:06.798 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:37:06.799 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:37:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.801 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:37:06.802 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:37:06.803 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.804 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:37:06.805 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:37:06.807 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:37:06.807 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:37:06.807 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.807 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.807 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:37:06.807 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:37:06.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8ec0036aa0 legacy=0x7f8ec0038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8eda03c640 10 monclient: shutdown 2024-09-16T05:37:06.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.774+0000 7f8eda03c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.778+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 >> v1:172.21.15.6:6789/0 conn(0x7f8ed409aa30 legacy=0x7f8ed4136310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.778+0000 7f8ed3fff640 1 -- 172.21.15.6:0/4113586448 reap_dead start 2024-09-16T05:37:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.778+0000 7f8ed3fff640 1 -- 172.21.15.6:0/4113586448 reap_dead start 2024-09-16T05:37:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.778+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 shutdown_connections 2024-09-16T05:37:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.778+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 >> 172.21.15.6:0/4113586448 conn(0x7f8ed40a1650 msgr2=0x7f8ed40a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.778+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 shutdown_connections 2024-09-16T05:37:06.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.778+0000 7f8eda03c640 1 -- 172.21.15.6:0/4113586448 wait complete. 2024-09-16T05:37:06.809 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:37:06.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:37:06.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient: get_monmap_and_config 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient: build_initial_monmap 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient: monmap: 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:37:06.919921+0000 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:37:06.919921+0000 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.920 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:06.921 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:06.921 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:06.921 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 Processor -- start 2024-09-16T05:37:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- start start 2024-09-16T05:37:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient: init 2024-09-16T05:37:06.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient(hunting): picked mon.noname-b con 0x7f3d2009bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2009bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient(hunting): picked mon.noname-a con 0x7f3d201397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d200a1350 con 0x7f3d201397f0 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d200a10d0 con 0x7f3d2009bf20 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2009d940 con 0x7f3d2009bb40 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient(hunting): authenticate will time out at 1088.726453s 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d26425640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3d2009bb40 0x7f3d2009e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:43082/0 (socket says 172.21.15.6:43082) 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d26c26640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3d201397f0 0x7f3d2013bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33452/0 (socket says 172.21.15.6:33452) 2024-09-16T05:37:06.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d26425640 1 -- 172.21.15.6:0/2656291967 learned_addr learned my addr 172.21.15.6:0/2656291967 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25c24640 1 -- 172.21.15.6:0/2656291967 >> v1:172.21.15.63:6789/0 conn(0x7f3d2009bf20 legacy=0x7f3d2009c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3969980353 0 0) 0x7f3d2009d940 con 0x7f3d2009bb40 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient(hunting): my global_id is 4949 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d08003530 con 0x7f3d2009bb40 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3272759051 0 0) 0x7f3d200a1350 con 0x7f3d201397f0 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient(hunting): my global_id is 4968 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d2009d940 con 0x7f3d201397f0 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2499733804 0 0) 0x7f3d2009d940 con 0x7f3d201397f0 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 >> v1:172.21.15.6:6790/0 conn(0x7f3d2009bb40 legacy=0x7f3d2009e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 >> v1:172.21.15.63:6789/0 conn(0x7f3d2009bf20 legacy=0x7f3d2009c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: _finish_hunting 0 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 monclient: found mon.noname-a 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2009dc20 con 0x7f3d201397f0 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: _finish_auth 0 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: _check_auth_tickets 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:06.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d18001600 con 0x7f3d201397f0 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: dump: 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.932 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 5 monclient: authenticate success, global_id 4968 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d180018f0 con 0x7f3d201397f0 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 1 -- 172.21.15.6:0/2656291967 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d18001d00 con 0x7f3d201397f0 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 4 set_mon_vals no callback set 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d25423640 10 monclient: dump: 2024-09-16T05:37:06.933 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 10 monclient: shutdown 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:06.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- 172.21.15.6:0/2656291967 >> v1:172.21.15.6:6789/0 conn(0x7f3d201397f0 legacy=0x7f3d2013bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- 172.21.15.6:0/2656291967 shutdown_connections 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- 172.21.15.6:0/2656291967 >> 172.21.15.6:0/2656291967 conn(0x7f3d200a1650 msgr2=0x7f3d200a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- 172.21.15.6:0/2656291967 shutdown_connections 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.918+0000 7f3d28689640 1 -- 172.21.15.6:0/2656291967 wait complete. 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: build_initial_monmap 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: monmap: 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:37:06.935 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 1 Processor -- start 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 1 -- start start 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: init 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient(hunting): picked mon.noname-a con 0x7f3d2009bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:06.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2009bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient(hunting): picked mon.noname-b con 0x7f3d201397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d200a3ac0 con 0x7f3d2009bb40 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2009b1e0 con 0x7f3d201397f0 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2013bff0 con 0x7f3d2009bf20 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient(hunting): authenticate will time out at 1088.729335s 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d26425640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3d2009bb40 0x7f3d20139bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33468/0 (socket says 172.21.15.6:33468) 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d26425640 1 -- 172.21.15.6:0/28046891 learned_addr learned my addr 172.21.15.6:0/28046891 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d26c26640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.63:6789/0 conn(0x7f3d201397f0 legacy=0x7f3d201d14d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174640378 0 0) 0x7f3d10001030 con 0x7f3d2009bb40 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): my global_id is 4971 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d0c003530 con 0x7f3d2009bb40 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289586796 0 0) 0x7f3d2013bff0 con 0x7f3d2009bf20 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): my global_id is 4952 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d10001030 con 0x7f3d2009bf20 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1125508163 0 0) 0x7f3d0c003530 con 0x7f3d2009bb40 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d2013bff0 con 0x7f3d2009bb40 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d10001a80 con 0x7f3d2009bb40 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:06.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): dump: 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3703473528 0 0) 0x7f3d10001030 con 0x7f3d2009bf20 2024-09-16T05:37:06.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d0c003530 con 0x7f3d2009bf20 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d14004480 con 0x7f3d2009bf20 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): dump: 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.940 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3169586343 0 0) 0x7f3d2013bff0 con 0x7f3d2009bb40 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.6:6790/0 conn(0x7f3d2009bf20 legacy=0x7f3d2013a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.63:6789/0 conn(0x7f3d201397f0 legacy=0x7f3d201d14d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 monclient: found mon.a 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2013a9f0 con 0x7f3d2009bb40 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: _finish_auth 0 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:06.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 5 monclient: authenticate success, global_id 4971 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: _renew_subs 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2013c5b0 con 0x7f3d2009bb40 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d10004aa0 con 0x7f3d2009bb40 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d10004df0 con 0x7f3d2009bb40 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: dump: 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.942 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1ffff640 4 set_mon_vals no callback set 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: _renew_subs 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d2013c2d0 con 0x7f3d2009bb40 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d200008e0 con 0x7f3d2009aa30 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3d1000fb90 con 0x7f3d2009bb40 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(62..62 src has 1..62) ==== 4382+0+0 (unknown 4127620019 0 0) 0x7f3d10044690 con 0x7f3d2009bb40 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3191512864 0 0) 0x7f3d200008e0 con 0x7f3d2009aa30 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient(hunting): my global_id is 4955 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d2013bff0 con 0x7f3d2009aa30 2024-09-16T05:37:06.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3578842198 0 0) 0x7f3d2013bff0 con 0x7f3d2009aa30 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d200008e0 con 0x7f3d2009aa30 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d14004e30 con 0x7f3d2009aa30 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d26c26640 10 monclient: get_auth_request con 0x7f3d0c036990 auth_method 0 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 10 monclient: dump: 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.945 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3172035770 0 0) 0x7f3d200008e0 con 0x7f3d2009aa30 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.922+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d20071c60 con 0x7f3d2009aa30 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.926+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3d14007200 con 0x7f3d2009aa30 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.926+0000 7f3d1effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.926+0000 7f3d1effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:06.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.926+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.6:6790/0 conn(0x7f3d2009aa30 legacy=0x7f3d2006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.926+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.6:6790/0 conn(0x7f3d2009aa30 legacy=0x7f3d2006ded0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:37:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:37:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d20071e10 con 0x7f3d2002d660 2024-09-16T05:37:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1615908593 0 0) 0x7f3d20071e10 con 0x7f3d2002d660 2024-09-16T05:37:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient(hunting): my global_id is 4958 2024-09-16T05:37:06.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d200008e0 con 0x7f3d2002d660 2024-09-16T05:37:06.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2359129911 0 0) 0x7f3d200008e0 con 0x7f3d2002d660 2024-09-16T05:37:06.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d20071e10 con 0x7f3d2002d660 2024-09-16T05:37:06.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d14005580 con 0x7f3d2002d660 2024-09-16T05:37:06.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:06.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient: dump: 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3820909216 0 0) 0x7f3d20071e10 con 0x7f3d2002d660 2024-09-16T05:37:06.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d2002da40 con 0x7f3d2002d660 2024-09-16T05:37:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4118 (unknown 0 0 2669404953) 0x7f3d14006890 con 0x7f3d2002d660 2024-09-16T05:37:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:37:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.6:6790/0 conn(0x7f3d2002d660 legacy=0x7f3d2006e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d1effd640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.6:6790/0 conn(0x7f3d2002d660 legacy=0x7f3d2006e5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:37:06.951 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:37:06.951 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:37:06.951 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 18, 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 155986, 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:37:06.952 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:37:06.953 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.954 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:37:06.955 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:37:06.956 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:37:06.956 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:37:06.956 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:37:06.956 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:37:06.956 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:37:06.956 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:37:06.956 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:37:06.956 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.957 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:37:06.959 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:37:06.960 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:37:06.961 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:37:06.961 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:37:06.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.961 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:37:06.962 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.963 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:37:06.964 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:37:06.965 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:37:06.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:37:06.965 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:37:06.965 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:37:06.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3d0c036990 legacy=0x7f3d0c038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 10 monclient: shutdown 2024-09-16T05:37:06.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:06.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 >> v1:172.21.15.6:6789/0 conn(0x7f3d2009bb40 legacy=0x7f3d20139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:06.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d26c26640 1 -- 172.21.15.6:0/28046891 reap_dead start 2024-09-16T05:37:06.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d26c26640 1 -- 172.21.15.6:0/28046891 reap_dead start 2024-09-16T05:37:06.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 shutdown_connections 2024-09-16T05:37:06.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 >> 172.21.15.6:0/28046891 conn(0x7f3d200a1650 msgr2=0x7f3d20139020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:06.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 shutdown_connections 2024-09-16T05:37:06.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:37:06.930+0000 7f3d28689640 1 -- 172.21.15.6:0/28046891 wait complete. 2024-09-16T05:37:06.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:37:06.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:37:07.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.015+0000 7f62a27fc640 10 monclient: tick 2024-09-16T05:37:07.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.015+0000 7f62a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:07.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.015+0000 7f62a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:07.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.387+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 v63) ==== 110+0+0 (unknown 2568431708 0 0) 0x7f62ac014530 con 0x7f62bc09ca20 2024-09-16T05:37:07.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.387+0000 7f62a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T05:37:07.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.387+0000 7f62a37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:07.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.491+0000 7f62c15ea640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T05:37:07.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.491+0000 7f62c15ea640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T05:37:07.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.491+0000 7f62c15ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:07.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.491+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) -- 0x7f629c005be0 con 0x7f62bc09ca20 2024-09-16T05:37:07.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.491+0000 7f62a37fe640 1 -- 172.21.15.63:0/2410880128 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 already renamed to 'bar' v63) ==== 134+0+0 (unknown 527670528 0 0) 0x7f62ac0181e0 con 0x7f62bc09ca20 2024-09-16T05:37:07.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.491+0000 7f62a37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T05:37:07.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.491+0000 7f62a37fe640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar' 2024-09-16T05:37:07.496 INFO:tasks.workunit.client.0.smithi063.stderr:already renamed to 'bar' 2024-09-16T05:37:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.495+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 >> v1:172.21.15.63:6812/3643207211 conn(0x7f62a8036a40 legacy=0x7f62a8038f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:07.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.495+0000 7f62c15ea640 10 monclient: shutdown 2024-09-16T05:37:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.495+0000 7f62c15ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.495+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 >> v1:172.21.15.6:6789/0 conn(0x7f62bc09ca20 legacy=0x7f62bc12dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.495+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 shutdown_connections 2024-09-16T05:37:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.495+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 >> 172.21.15.63:0/2410880128 conn(0x7f62bc0a2420 msgr2=0x7f62bc12bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.495+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 shutdown_connections 2024-09-16T05:37:07.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.495+0000 7f62c15ea640 1 -- 172.21.15.63:0/2410880128 wait complete. 2024-09-16T05:37:07.514 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls 2024-09-16T05:37:07.514 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false grep foo-asdf 2024-09-16T05:37:07.514 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:07.514 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep foo-asdf 2024-09-16T05:37:07.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient: get_monmap_and_config 2024-09-16T05:37:07.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient: build_initial_monmap 2024-09-16T05:37:07.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient: monmap: 2024-09-16T05:37:07.622 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:07.622 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:07.622 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:07.621727+0000 2024-09-16T05:37:07.622 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:07.621727+0000 2024-09-16T05:37:07.623 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:07.623 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:07.623 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:07.623 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:07.623 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:07.623 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 1 Processor -- start 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 1 -- start start 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient: init 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient(hunting): picked mon.noname-c con 0x7fdc8009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient(hunting): picked mon.noname-a con 0x7fdc8006dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient(hunting): picked mon.noname-b con 0x7fdc800a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc8009f800 con 0x7fdc8006dd30 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc800a0ab0 con 0x7fdc8009ac90 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc800a0d90 con 0x7fdc800a0210 2024-09-16T05:37:07.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc85845640 10 monclient(hunting): authenticate will time out at 1084.398704s 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7effd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fdc8009ac90 legacy=0x7fdc8009b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7f7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdc800a0210 0x7fdc8013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60964/0 (socket says 172.21.15.63:60964) 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7f7fe640 1 -- 172.21.15.63:0/1047390137 learned_addr learned my addr 172.21.15.63:0/1047390137 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7e7fc640 1 --1- 172.21.15.63:0/1047390137 >> v1:172.21.15.6:6789/0 conn(0x7fdc8006dd30 0x7fdc801399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44378/0 (socket says 172.21.15.63:44378) 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022207655 0 0) 0x7fdc8009f800 con 0x7fdc8006dd30 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 10 monclient(hunting): my global_id is 4974 2024-09-16T05:37:07.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc6c003530 con 0x7fdc8006dd30 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266914974 0 0) 0x7fdc800a0d90 con 0x7fdc800a0210 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 10 monclient(hunting): my global_id is 4961 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.619+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc8009f800 con 0x7fdc800a0210 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1660880969 0 0) 0x7fdc8009f800 con 0x7fdc800a0210 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 >> v1:172.21.15.63:6789/0 conn(0x7fdc8009ac90 legacy=0x7fdc8009b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 >> v1:172.21.15.6:6789/0 conn(0x7fdc8006dd30 legacy=0x7fdc801399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:07.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 1 monclient: found mon.noname-b 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8009e770 con 0x7fdc800a0210 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 5 monclient: authenticate success, global_id 4961 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdc70001600 con 0x7fdc800a0210 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: dump: 2024-09-16T05:37:07.629 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc70001860 con 0x7fdc800a0210 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 1 -- 172.21.15.63:0/1047390137 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdc70001cb0 con 0x7fdc800a0210 2024-09-16T05:37:07.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7dffb640 10 monclient: dump: 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:07.631 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 4 set_mon_vals no callback set 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient: shutdown 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- 172.21.15.63:0/1047390137 >> v1:172.21.15.6:6790/0 conn(0x7fdc800a0210 legacy=0x7fdc8013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- 172.21.15.63:0/1047390137 shutdown_connections 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- 172.21.15.63:0/1047390137 >> 172.21.15.63:0/1047390137 conn(0x7fdc800a1c90 msgr2=0x7fdc800a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- 172.21.15.63:0/1047390137 shutdown_connections 2024-09-16T05:37:07.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- 172.21.15.63:0/1047390137 wait complete. 2024-09-16T05:37:07.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient: build_initial_monmap 2024-09-16T05:37:07.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient: monmap: 2024-09-16T05:37:07.636 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:07.636 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:07.636 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:07.636 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 Processor -- start 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- start start 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient: init 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:07.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient(hunting): picked mon.noname-c con 0x7fdc8006dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient(hunting): picked mon.noname-b con 0x7fdc8009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient(hunting): picked mon.noname-a con 0x7fdc800a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc8009d7d0 con 0x7fdc800a0210 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc8009dfd0 con 0x7fdc8009ac90 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc80122c40 con 0x7fdc8006dd30 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc85845640 10 monclient(hunting): authenticate will time out at 1084.402389s 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7e7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fdc8009ac90 legacy=0x7fdc801d0240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdc800a0210 0x7fdc801d3970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44384/0 (socket says 172.21.15.63:44384) 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc7f7fe640 1 -- 172.21.15.63:0/903176537 learned_addr learned my addr 172.21.15.63:0/903176537 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1111441170 0 0) 0x7fdc80122c40 con 0x7fdc8006dd30 2024-09-16T05:37:07.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): my global_id is 4964 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc60003530 con 0x7fdc8006dd30 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1352152285 0 0) 0x7fdc8009d7d0 con 0x7fdc800a0210 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): my global_id is 4977 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc80122c40 con 0x7fdc800a0210 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3728049406 0 0) 0x7fdc60003530 con 0x7fdc8006dd30 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc8009d7d0 con 0x7fdc8006dd30 2024-09-16T05:37:07.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdc780040d0 con 0x7fdc8006dd30 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): dump: 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:07.640 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2885378428 0 0) 0x7fdc80122c40 con 0x7fdc800a0210 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc60003530 con 0x7fdc800a0210 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdc70001bb0 con 0x7fdc800a0210 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.623+0000 7fdc77fff640 10 monclient(hunting): dump: 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.641 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3728569756 0 0) 0x7fdc8009d7d0 con 0x7fdc8006dd30 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 >> v1:172.21.15.63:6789/0 conn(0x7fdc8009ac90 legacy=0x7fdc801d0240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 >> v1:172.21.15.6:6789/0 conn(0x7fdc800a0210 legacy=0x7fdc801d3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 monclient: found mon.c 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:07.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc80122f20 con 0x7fdc8006dd30 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 10 monclient: _finish_auth 0 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 5 monclient: authenticate success, global_id 4964 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 10 monclient: _renew_subs 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc80121d50 con 0x7fdc8006dd30 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 10 monclient: _renew_subs 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc801343e0 con 0x7fdc8006dd30 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc78004330 con 0x7fdc8006dd30 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:07.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdc78004680 con 0x7fdc8006dd30 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc7d7fa640 4 set_mon_vals no callback set 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 10 monclient: dump: 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:07.644 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc64005090 con 0x7fdc8006dd30 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdc78004820 con 0x7fdc8006dd30 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(63..63 src has 1..63) ==== 4377+0+0 (unknown 3876452279 0 0) 0x7fdc780442a0 con 0x7fdc8006dd30 2024-09-16T05:37:07.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.627+0000 7fdc7e7fc640 10 monclient: get_auth_request con 0x7fdc60036a70 auth_method 0 2024-09-16T05:37:07.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.639+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fdc780151a0 con 0x7fdc8006dd30 2024-09-16T05:37:07.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.639+0000 7fdc77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:07.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.639+0000 7fdc77fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:07.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.803+0000 7fdc85845640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:07.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.803+0000 7fdc85845640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:07.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.803+0000 7fdc85845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:07.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.803+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fdc64005380 con 0x7fdc8006dd30 2024-09-16T05:37:07.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.803+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v63) ==== 65+0+60 (unknown 3399564340 0 1596542489) 0x7fdc78018e50 con 0x7fdc8006dd30 2024-09-16T05:37:07.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.803+0000 7fdc77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:07.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.803+0000 7fdc77fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fdc64005ac0 con 0x7fdc8006dd30 2024-09-16T05:37:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc77fff640 1 -- 172.21.15.63:0/903176537 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v63) ==== 65+0+60 (unknown 3399564340 0 1596542489) 0x7fdc7801d710 con 0x7fdc8006dd30 2024-09-16T05:37:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc77fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc77fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 >> v1:172.21.15.63:6812/3643207211 conn(0x7fdc60036a70 legacy=0x7fdc60038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 10 monclient: shutdown 2024-09-16T05:37:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 >> v1:172.21.15.6:6790/0 conn(0x7fdc8006dd30 legacy=0x7fdc80133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 shutdown_connections 2024-09-16T05:37:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 >> 172.21.15.63:0/903176537 conn(0x7fdc800a1c90 msgr2=0x7fdc8006e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 shutdown_connections 2024-09-16T05:37:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:07.899+0000 7fdc85845640 1 -- 172.21.15.63:0/903176537 wait complete. 2024-09-16T05:37:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule ls 2024-09-16T05:37:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false grep bar-asdf 2024-09-16T05:37:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep bar-asdf 2024-09-16T05:37:08.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient: get_monmap_and_config 2024-09-16T05:37:08.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient: build_initial_monmap 2024-09-16T05:37:08.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient: monmap: 2024-09-16T05:37:08.035 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:08.035 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.035 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:08.034399+0000 2024-09-16T05:37:08.035 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:08.034399+0000 2024-09-16T05:37:08.035 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:08.035 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.036 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:08.036 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:08.036 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:08.036 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.043 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 1 Processor -- start 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 1 -- start start 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient: init 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient(hunting): picked mon.noname-b con 0x7ff54409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient(hunting): picked mon.noname-a con 0x7ff54406dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient(hunting): picked mon.noname-c con 0x7ff5440a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff54409f800 con 0x7ff54406dd30 2024-09-16T05:37:08.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5440a0ab0 con 0x7ff5440a0210 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5440a0d90 con 0x7ff54409ac90 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff548e93640 10 monclient(hunting): authenticate will time out at 1084.811544s 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff542d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff5440a0210 legacy=0x7ff54413d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff541d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff54406dd30 0x7ff5441399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44400/0 (socket says 172.21.15.63:44400) 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.031+0000 7ff541d9b640 1 -- 172.21.15.63:0/148998325 learned_addr learned my addr 172.21.15.63:0/148998325 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777267048 0 0) 0x7ff54409f800 con 0x7ff54406dd30 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient(hunting): my global_id is 4980 2024-09-16T05:37:08.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff530003530 con 0x7ff54406dd30 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810595278 0 0) 0x7ff5440a0d90 con 0x7ff54409ac90 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient(hunting): my global_id is 4967 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff54409f800 con 0x7ff54409ac90 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1323614709 0 0) 0x7ff54409f800 con 0x7ff54409ac90 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 >> v1:172.21.15.63:6789/0 conn(0x7ff5440a0210 legacy=0x7ff54413d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 >> v1:172.21.15.6:6789/0 conn(0x7ff54406dd30 legacy=0x7ff5441399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: _finish_hunting 0 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 monclient: found mon.noname-b 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54409e770 con 0x7ff54409ac90 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: _finish_auth 0 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: _check_auth_tickets 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff52c001600 con 0x7ff54409ac90 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: dump: 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 5 monclient: authenticate success, global_id 4967 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff52c001960 con 0x7ff54409ac90 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 1 -- 172.21.15.63:0/148998325 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff52c003620 con 0x7ff54409ac90 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:08.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54159a640 10 monclient: dump: 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:08.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 4 set_mon_vals no callback set 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient: shutdown 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- 172.21.15.63:0/148998325 >> v1:172.21.15.6:6790/0 conn(0x7ff54409ac90 legacy=0x7ff54409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- 172.21.15.63:0/148998325 shutdown_connections 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- 172.21.15.63:0/148998325 >> 172.21.15.63:0/148998325 conn(0x7ff5440a1c90 msgr2=0x7ff5440a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- 172.21.15.63:0/148998325 shutdown_connections 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- 172.21.15.63:0/148998325 wait complete. 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient: build_initial_monmap 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient: monmap: 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:08.050 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 Processor -- start 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- start start 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient: init 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:08.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient(hunting): picked mon.noname-a con 0x7ff54406dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient(hunting): picked mon.noname-c con 0x7ff54409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient(hunting): picked mon.noname-b con 0x7ff5440a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff544126540 con 0x7ff54406dd30 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff544123750 con 0x7ff5440a0210 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff544125090 con 0x7ff54409ac90 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff542d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff5440a0210 legacy=0x7ff5441d3810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff548e93640 10 monclient(hunting): authenticate will time out at 1084.815334s 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff54406dd30 0x7ff544122930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44410/0 (socket says 172.21.15.63:44410) 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.035+0000 7ff54259c640 1 -- 172.21.15.63:0/1677191595 learned_addr learned my addr 172.21.15.63:0/1677191595 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:08.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4191401785 0 0) 0x7ff544126540 con 0x7ff54406dd30 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): my global_id is 4983 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff524003530 con 0x7ff54406dd30 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2027998527 0 0) 0x7ff544125090 con 0x7ff54409ac90 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): my global_id is 4970 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff544126540 con 0x7ff54409ac90 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 14495915 0 0) 0x7ff544126540 con 0x7ff54409ac90 2024-09-16T05:37:08.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff544125090 con 0x7ff54409ac90 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff534001ba0 con 0x7ff54409ac90 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): dump: 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.054 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3687649524 0 0) 0x7ff524003530 con 0x7ff54406dd30 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff544126540 con 0x7ff54406dd30 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff52c001420 con 0x7ff54406dd30 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:08.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient(hunting): dump: 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3237946865 0 0) 0x7ff544125090 con 0x7ff54409ac90 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 >> v1:172.21.15.63:6789/0 conn(0x7ff5440a0210 legacy=0x7ff5441d3810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 >> v1:172.21.15.6:6789/0 conn(0x7ff54406dd30 legacy=0x7ff544122930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 monclient: found mon.c 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff544126820 con 0x7ff54409ac90 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient: _finish_auth 0 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:08.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 5 monclient: authenticate success, global_id 4970 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 10 monclient: _renew_subs 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff54406e160 con 0x7ff54409ac90 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 10 monclient: _renew_subs 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff544124e10 con 0x7ff54409ac90 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5340020b0 con 0x7ff54409ac90 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff534004d10 con 0x7ff54409ac90 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:08.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53bfff640 4 set_mon_vals no callback set 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 10 monclient: dump: 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.059 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff528005090 con 0x7ff54409ac90 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff534004eb0 con 0x7ff54409ac90 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(63..63 src has 1..63) ==== 4377+0+0 (unknown 3876452279 0 0) 0x7ff534044560 con 0x7ff54409ac90 2024-09-16T05:37:08.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.039+0000 7ff54259c640 10 monclient: get_auth_request con 0x7ff524036ac0 auth_method 0 2024-09-16T05:37:08.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.051+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff534015460 con 0x7ff54409ac90 2024-09-16T05:37:08.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.051+0000 7ff53affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:08.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.051+0000 7ff53affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:08.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.223+0000 7ff548e93640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:08.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.223+0000 7ff548e93640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:08.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.223+0000 7ff548e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.223+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7ff528005380 con 0x7ff54409ac90 2024-09-16T05:37:08.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.223+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v63) ==== 65+0+60 (unknown 3399564340 0 1596542489) 0x7ff534019110 con 0x7ff54409ac90 2024-09-16T05:37:08.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.223+0000 7ff53affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:08.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.223+0000 7ff53affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:08.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.323+0000 7ff548e93640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:08.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.323+0000 7ff548e93640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:08.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.323+0000 7ff548e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.323+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7ff528005ac0 con 0x7ff54409ac90 2024-09-16T05:37:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.323+0000 7ff53affd640 1 -- 172.21.15.63:0/1677191595 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v63) ==== 65+0+60 (unknown 3399564340 0 1596542489) 0x7ff53401d9d0 con 0x7ff54409ac90 2024-09-16T05:37:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.323+0000 7ff53affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T05:37:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.323+0000 7ff53affd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:08.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.327+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff524036ac0 legacy=0x7ff524038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:08.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.327+0000 7ff548e93640 10 monclient: shutdown 2024-09-16T05:37:08.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.327+0000 7ff548e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:08.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.327+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 >> v1:172.21.15.6:6790/0 conn(0x7ff54409ac90 legacy=0x7ff544123040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:08.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.327+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 shutdown_connections 2024-09-16T05:37:08.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.327+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 >> 172.21.15.63:0/1677191595 conn(0x7ff5440a1c90 msgr2=0x7ff544134930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:08.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.327+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 shutdown_connections 2024-09-16T05:37:08.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.327+0000 7ff548e93640 1 -- 172.21.15.63:0/1677191595 wait complete. 2024-09-16T05:37:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rm foo 2024-09-16T05:37:08.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.463+0000 7fe72e41c640 10 monclient: get_monmap_and_config 2024-09-16T05:37:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.463+0000 7fe72e41c640 10 monclient: build_initial_monmap 2024-09-16T05:37:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.463+0000 7fe72e41c640 10 monclient: monmap: 2024-09-16T05:37:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:08.467341+0000 2024-09-16T05:37:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:08.467341+0000 2024-09-16T05:37:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:08.469 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.469 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:08.469 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:08.469 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:08.469 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 Processor -- start 2024-09-16T05:37:08.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- start start 2024-09-16T05:37:08.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient: init 2024-09-16T05:37:08.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:08.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:08.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient(hunting): picked mon.noname-b con 0x7fe728136960 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:08.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient(hunting): picked mon.noname-c con 0x7fe7280a1c40 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:08.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient(hunting): picked mon.noname-a con 0x7fe728137450 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:08.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7280a09c0 con 0x7fe728137450 2024-09-16T05:37:08.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7280a0f80 con 0x7fe7280a1c40 2024-09-16T05:37:08.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7280a4bb0 con 0x7fe728136960 2024-09-16T05:37:08.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient(hunting): authenticate will time out at 1085.244959s 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe7277fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe7280a1c40 legacy=0x7fe72813bdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72c9b9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe728137450 0x7fe72813f4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44426/0 (socket says 172.21.15.63:44426) 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe727fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe728136960 0x7fe728136d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60998/0 (socket says 172.21.15.63:60998) 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72c9b9640 1 -- 172.21.15.63:0/437866206 learned_addr learned my addr 172.21.15.63:0/437866206 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102717104 0 0) 0x7fe7280a4bb0 con 0x7fe728136960 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:08.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient(hunting): my global_id is 4973 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe710003530 con 0x7fe728136960 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953277567 0 0) 0x7fe7280a09c0 con 0x7fe728137450 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient(hunting): my global_id is 4986 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7280a4bb0 con 0x7fe728137450 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1383793608 0 0) 0x7fe710003530 con 0x7fe728136960 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 >> v1:172.21.15.63:6789/0 conn(0x7fe7280a1c40 legacy=0x7fe72813bdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 >> v1:172.21.15.6:6789/0 conn(0x7fe728137450 legacy=0x7fe72813f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 monclient: found mon.noname-b 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7280a4e60 con 0x7fe728136960 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe718001620 con 0x7fe728136960 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:08.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: dump: 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 5 monclient: authenticate success, global_id 4973 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe718001880 con 0x7fe728136960 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 1 -- 172.21.15.63:0/437866206 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe718001bd0 con 0x7fe728136960 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe726ffd640 10 monclient: dump: 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.486 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 4 set_mon_vals no callback set 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 10 monclient: shutdown 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- 172.21.15.63:0/437866206 >> v1:172.21.15.6:6790/0 conn(0x7fe728136960 legacy=0x7fe728136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- 172.21.15.63:0/437866206 shutdown_connections 2024-09-16T05:37:08.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- 172.21.15.63:0/437866206 >> 172.21.15.63:0/437866206 conn(0x7fe7280a60a0 msgr2=0x7fe7280a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- 172.21.15.63:0/437866206 shutdown_connections 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.467+0000 7fe72e41c640 1 -- 172.21.15.63:0/437866206 wait complete. 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: build_initial_monmap 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: monmap: 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.488 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 1 Processor -- start 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 1 -- start start 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: init 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient(hunting): picked mon.noname-b con 0x7fe7280a1c40 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient(hunting): picked mon.noname-c con 0x7fe728136960 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:08.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient(hunting): picked mon.noname-a con 0x7fe728137450 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7281c8690 con 0x7fe728137450 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe728126990 con 0x7fe7280a1c40 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7280a72a0 con 0x7fe728136960 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient(hunting): authenticate will time out at 1085.248995s 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe727fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe7280a1c40 legacy=0x7fe728135fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe7277fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe728136960 0x7fe7281d0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32770/0 (socket says 172.21.15.63:32770) 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe7277fe640 1 -- 172.21.15.63:0/1900351627 learned_addr learned my addr 172.21.15.63:0/1900351627 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979115643 0 0) 0x7fe7280a4bb0 con 0x7fe728137450 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:08.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): my global_id is 4989 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe714003530 con 0x7fe728137450 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2585056988 0 0) 0x7fe7280a72a0 con 0x7fe728136960 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): my global_id is 4976 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7280a4bb0 con 0x7fe728136960 2024-09-16T05:37:08.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1174845355 0 0) 0x7fe7280a4bb0 con 0x7fe728136960 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7280a72a0 con 0x7fe728136960 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe71c004160 con 0x7fe728136960 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.492 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1524822959 0 0) 0x7fe714003530 con 0x7fe728137450 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7280a4bb0 con 0x7fe728137450 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe720001970 con 0x7fe728137450 2024-09-16T05:37:08.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:08.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:08.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:08.494 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.494 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.494 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.494 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 695415156 0 0) 0x7fe7280a72a0 con 0x7fe728136960 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 >> v1:172.21.15.63:6789/0 conn(0x7fe7280a1c40 legacy=0x7fe728135fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:08.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 >> v1:172.21.15.6:6789/0 conn(0x7fe728137450 legacy=0x7fe7281d3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:08.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:37:08.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 monclient: found mon.c 2024-09-16T05:37:08.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:08.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7280a0500 con 0x7fe728136960 2024-09-16T05:37:08.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient: _finish_auth 0 2024-09-16T05:37:08.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:37:08.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 5 monclient: authenticate success, global_id 4976 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: _renew_subs 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe7281246a0 con 0x7fe728136960 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: _renew_subs 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe728125440 con 0x7fe728136960 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe71c004670 con 0x7fe728136960 2024-09-16T05:37:08.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe71c004a40 con 0x7fe728136960 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe7267fc640 4 set_mon_vals no callback set 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe724ff9640 10 monclient: dump: 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.498 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:08.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.471+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe708005090 con 0x7fe728136960 2024-09-16T05:37:08.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.475+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe71c004c60 con 0x7fe728136960 2024-09-16T05:37:08.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.475+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(63..63 src has 1..63) ==== 4377+0+0 (unknown 3876452279 0 0) 0x7fe71c044460 con 0x7fe728136960 2024-09-16T05:37:08.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.475+0000 7fe727fff640 10 monclient: get_auth_request con 0x7fe714036a70 auth_method 0 2024-09-16T05:37:08.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.487+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe71c015420 con 0x7fe728136960 2024-09-16T05:37:08.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.487+0000 7fe724ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:08.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.487+0000 7fe724ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:08.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.651+0000 7fe72e41c640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:08.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.651+0000 7fe72e41c640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:08.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.651+0000 7fe72e41c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:08.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:08.651+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) -- 0x7fe708005380 con 0x7fe728136960 2024-09-16T05:37:09.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.403+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 v64) ==== 80+0+0 (unknown 3525359998 0 0) 0x7fe71c014d20 con 0x7fe728136960 2024-09-16T05:37:09.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.403+0000 7fe724ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.403+0000 7fe724ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.507+0000 7fe72e41c640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.507+0000 7fe72e41c640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.507+0000 7fe72e41c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:09.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.507+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) -- 0x7fe708005770 con 0x7fe728136960 2024-09-16T05:37:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe724ff9640 1 -- 172.21.15.63:0/1900351627 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v64) ==== 103+0+0 (unknown 206984665 0 0) 0x7fe71c01d8d0 con 0x7fe728136960 2024-09-16T05:37:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe724ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe724ff9640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T05:37:09.513 INFO:tasks.workunit.client.0.smithi063.stderr:rule foo does not exist 2024-09-16T05:37:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe714036a70 legacy=0x7fe714038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe72e41c640 10 monclient: shutdown 2024-09-16T05:37:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe72e41c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:09.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 >> v1:172.21.15.6:6790/0 conn(0x7fe728136960 legacy=0x7fe7281d0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:09.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 shutdown_connections 2024-09-16T05:37:09.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 >> 172.21.15.63:0/1900351627 conn(0x7fe7280a60a0 msgr2=0x7fe728075410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:09.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 shutdown_connections 2024-09-16T05:37:09.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.511+0000 7fe72e41c640 1 -- 172.21.15.63:0/1900351627 wait complete. 2024-09-16T05:37:09.530 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rm foo 2024-09-16T05:37:09.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.647+0000 7f93c5432640 10 monclient: get_monmap_and_config 2024-09-16T05:37:09.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.647+0000 7f93c5432640 10 monclient: build_initial_monmap 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.647+0000 7f93c5432640 10 monclient: monmap: 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:09.652160+0000 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:09.652160+0000 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:09.653 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:09.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 Processor -- start 2024-09-16T05:37:09.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- start start 2024-09-16T05:37:09.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient: init 2024-09-16T05:37:09.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:09.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:09.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient(hunting): picked mon.noname-c con 0x7f93c009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:09.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient(hunting): picked mon.noname-a con 0x7f93c00a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:09.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient(hunting): picked mon.noname-b con 0x7f93c009b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c0074200 con 0x7f93c00a0280 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c009f900 con 0x7f93c009ac90 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c009e870 con 0x7f93c009b780 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient(hunting): authenticate will time out at 1086.429122s 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93beffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f93c009ac90 legacy=0x7f93c009b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93be7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f93c00a0280 0x7f93c01398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44448/0 (socket says 172.21.15.63:44448) 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93be7fc640 1 -- 172.21.15.63:0/3708215879 learned_addr learned my addr 172.21.15.63:0/3708215879 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288210079 0 0) 0x7f93c0074200 con 0x7f93c00a0280 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:09.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient(hunting): my global_id is 4992 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a8003530 con 0x7f93c00a0280 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1160495069 0 0) 0x7f93c009e870 con 0x7f93c009b780 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient(hunting): my global_id is 4979 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93c0074200 con 0x7f93c009b780 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1044024787 0 0) 0x7f93a8003530 con 0x7f93c00a0280 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 >> v1:172.21.15.6:6790/0 conn(0x7f93c009b780 legacy=0x7f93c013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 >> v1:172.21.15.63:6789/0 conn(0x7f93c009ac90 legacy=0x7f93c009b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:09.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 monclient: found mon.noname-a 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c0141730 con 0x7f93c00a0280 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 5 monclient: authenticate success, global_id 4992 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f93b4001620 con 0x7f93c00a0280 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: dump: 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:09.660 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93b4001850 con 0x7f93c00a0280 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:09.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 1 -- 172.21.15.63:0/3708215879 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f93b4001c60 con 0x7f93c00a0280 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 4 set_mon_vals no callback set 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93bdffb640 10 monclient: dump: 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:09.662 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 10 monclient: shutdown 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- 172.21.15.63:0/3708215879 >> v1:172.21.15.6:6789/0 conn(0x7f93c00a0280 legacy=0x7f93c01398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- 172.21.15.63:0/3708215879 shutdown_connections 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- 172.21.15.63:0/3708215879 >> 172.21.15.63:0/3708215879 conn(0x7f93c00a5290 msgr2=0x7f93c00a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- 172.21.15.63:0/3708215879 shutdown_connections 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.651+0000 7f93c5432640 1 -- 172.21.15.63:0/3708215879 wait complete. 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: build_initial_monmap 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: monmap: 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:09.663 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 1 Processor -- start 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 1 -- start start 2024-09-16T05:37:09.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: init 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient(hunting): picked mon.noname-c con 0x7f93c009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient(hunting): picked mon.noname-b con 0x7f93c009b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient(hunting): picked mon.noname-a con 0x7f93c00a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c0134200 con 0x7f93c00a0280 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c0140700 con 0x7f93c009b780 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93c013efd0 con 0x7f93c009ac90 2024-09-16T05:37:09.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient(hunting): authenticate will time out at 1086.433052s 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93be7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f93c009b780 legacy=0x7f93c01cdcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93beffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f93c009ac90 0x7f93c0133af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32794/0 (socket says 172.21.15.63:32794) 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93beffd640 1 -- 172.21.15.63:0/1998981275 learned_addr learned my addr 172.21.15.63:0/1998981275 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 525209016 0 0) 0x7f93c013efd0 con 0x7f93c009ac90 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): my global_id is 4982 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93ac003530 con 0x7f93c009ac90 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904771588 0 0) 0x7f93c0134200 con 0x7f93c00a0280 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:09.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): my global_id is 4995 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93c013efd0 con 0x7f93c00a0280 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 344594391 0 0) 0x7f93ac003530 con 0x7f93c009ac90 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93c0134200 con 0x7f93c009ac90 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f93b00044e0 con 0x7f93c009ac90 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): dump: 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:09.670 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:09.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2670591659 0 0) 0x7f93c013efd0 con 0x7f93c00a0280 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93ac003530 con 0x7f93c00a0280 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f93b80011e0 con 0x7f93c00a0280 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient(hunting): dump: 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:09.672 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2287083069 0 0) 0x7f93c0134200 con 0x7f93c009ac90 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 >> v1:172.21.15.63:6789/0 conn(0x7f93c009b780 legacy=0x7f93c01cdcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 >> v1:172.21.15.6:6789/0 conn(0x7f93c00a0280 legacy=0x7f93c01d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 monclient: found mon.c 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c006dfd0 con 0x7f93c009ac90 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:09.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 5 monclient: authenticate success, global_id 4982 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: _renew_subs 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93c006da80 con 0x7f93c009ac90 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: _renew_subs 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f93c013f470 con 0x7f93c009ac90 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93b0004740 con 0x7f93c009ac90 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f93b0004b10 con 0x7f93c009ac90 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:09.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93a77fe640 10 monclient: dump: 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93bd7fa640 4 set_mon_vals no callback set 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:09.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.655+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93a0005090 con 0x7f93c009ac90 2024-09-16T05:37:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.659+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f93b0004cb0 con 0x7f93c009ac90 2024-09-16T05:37:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.659+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(64..64 src has 1..64) ==== 4322+0+0 (unknown 2551441050 0 0) 0x7f93b00446e0 con 0x7f93c009ac90 2024-09-16T05:37:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.659+0000 7f93be7fc640 10 monclient: get_auth_request con 0x7f93ac036aa0 auth_method 0 2024-09-16T05:37:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.671+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f93b00156d0 con 0x7f93c009ac90 2024-09-16T05:37:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.671+0000 7f93a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:09.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.671+0000 7f93a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:09.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.831+0000 7f93c5432640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.831+0000 7f93c5432640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.831+0000 7f93c5432640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:09.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.831+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) -- 0x7f93a0005380 con 0x7f93c009ac90 2024-09-16T05:37:09.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.831+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v64) ==== 103+0+0 (unknown 206984665 0 0) 0x7f93b0014fd0 con 0x7f93c009ac90 2024-09-16T05:37:09.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.831+0000 7f93a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.831+0000 7f93a77fe640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-16T05:37:09.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.927+0000 7f93c5432640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.927+0000 7f93c5432640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.927+0000 7f93c5432640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:09.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.927+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) -- 0x7f93a0005770 con 0x7f93c009ac90 2024-09-16T05:37:09.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.927+0000 7f93a77fe640 1 -- 172.21.15.63:0/1998981275 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v64) ==== 103+0+0 (unknown 206984665 0 0) 0x7f93b001db80 con 0x7f93c009ac90 2024-09-16T05:37:09.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93a77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T05:37:09.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93a77fe640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T05:37:09.933 INFO:tasks.workunit.client.0.smithi063.stderr:rule foo does not exist 2024-09-16T05:37:09.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 >> v1:172.21.15.63:6812/3643207211 conn(0x7f93ac036aa0 legacy=0x7f93ac038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:09.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93c5432640 10 monclient: shutdown 2024-09-16T05:37:09.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93c5432640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:09.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 >> v1:172.21.15.6:6790/0 conn(0x7f93c009ac90 legacy=0x7f93c0133af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:09.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 shutdown_connections 2024-09-16T05:37:09.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 >> 172.21.15.63:0/1998981275 conn(0x7f93c00a5290 msgr2=0x7f93c00a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:09.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 shutdown_connections 2024-09-16T05:37:09.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:09.931+0000 7f93c5432640 1 -- 172.21.15.63:0/1998981275 wait complete. 2024-09-16T05:37:09.948 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rm bar 2024-09-16T05:37:10.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.051+0000 7f9a70eda640 10 monclient: get_monmap_and_config 2024-09-16T05:37:10.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.051+0000 7f9a70eda640 10 monclient: build_initial_monmap 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.051+0000 7f9a70eda640 10 monclient: monmap: 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:10.056029+0000 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:10.056029+0000 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:10.057 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 Processor -- start 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- start start 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient: init 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient(hunting): picked mon.noname-a con 0x7f9a6c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient(hunting): picked mon.noname-c con 0x7f9a6c0a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient(hunting): picked mon.noname-b con 0x7f9a6c09b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c074200 con 0x7f9a6c09ac90 2024-09-16T05:37:10.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c09f900 con 0x7f9a6c0a0280 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c09e870 con 0x7f9a6c09b780 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient(hunting): authenticate will time out at 1086.832918s 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a63fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9a6c0a0280 legacy=0x7f9a6c1398b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a6a59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9a6c09ac90 0x7f9a6c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44462/0 (socket says 172.21.15.63:44462) 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a6ad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9a6c09b780 0x7f9a6c13cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32796/0 (socket says 172.21.15.63:32796) 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a6a59c640 1 -- 172.21.15.63:0/963994322 learned_addr learned my addr 172.21.15.63:0/963994322 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200640916 0 0) 0x7f9a6c09e870 con 0x7f9a6c09b780 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:10.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient(hunting): my global_id is 4985 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a58003530 con 0x7f9a6c09b780 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911950155 0 0) 0x7f9a6c074200 con 0x7f9a6c09ac90 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient(hunting): my global_id is 4998 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a6c09e870 con 0x7f9a6c09ac90 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2111165437 0 0) 0x7f9a6c09e870 con 0x7f9a6c09ac90 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 >> v1:172.21.15.6:6790/0 conn(0x7f9a6c09b780 legacy=0x7f9a6c13cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 >> v1:172.21.15.63:6789/0 conn(0x7f9a6c0a0280 legacy=0x7f9a6c1398b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 monclient: found mon.noname-a 2024-09-16T05:37:10.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c141730 con 0x7f9a6c09ac90 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: _finish_auth 0 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 5 monclient: authenticate success, global_id 4998 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9a5c001580 con 0x7f9a6c09ac90 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: dump: 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.066 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9a5c0017e0 con 0x7f9a6c09ac90 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:10.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 4 set_mon_vals no callback set 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 1 -- 172.21.15.63:0/963994322 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9a5c001b30 con 0x7f9a6c09ac90 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a69d9b640 10 monclient: dump: 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.068 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 10 monclient: shutdown 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- 172.21.15.63:0/963994322 >> v1:172.21.15.6:6789/0 conn(0x7f9a6c09ac90 legacy=0x7f9a6c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- 172.21.15.63:0/963994322 shutdown_connections 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- 172.21.15.63:0/963994322 >> 172.21.15.63:0/963994322 conn(0x7f9a6c0a5290 msgr2=0x7f9a6c0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- 172.21.15.63:0/963994322 shutdown_connections 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.055+0000 7f9a70eda640 1 -- 172.21.15.63:0/963994322 wait complete. 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: build_initial_monmap 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: monmap: 2024-09-16T05:37:10.069 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 1 Processor -- start 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 1 -- start start 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: init 2024-09-16T05:37:10.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient(hunting): picked mon.noname-c con 0x7f9a6c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient(hunting): picked mon.noname-a con 0x7f9a6c09b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient(hunting): picked mon.noname-b con 0x7f9a6c0a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c13e2b0 con 0x7f9a6c09b780 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c06da80 con 0x7f9a6c0a0280 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c13efc0 con 0x7f9a6c09ac90 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a6ad9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9a6c0a0280 legacy=0x7f9a6c1d36b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient(hunting): authenticate will time out at 1086.837335s 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a63fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9a6c09b780 0x7f9a6c09cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44464/0 (socket says 172.21.15.63:44464) 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a6a59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9a6c09ac90 0x7f9a6c09c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32806/0 (socket says 172.21.15.63:32806) 2024-09-16T05:37:10.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a63fff640 1 -- 172.21.15.63:0/1102490803 learned_addr learned my addr 172.21.15.63:0/1102490803 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1206586651 0 0) 0x7f9a6c13e2b0 con 0x7f9a6c09b780 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): my global_id is 5001 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a4c003530 con 0x7f9a6c09b780 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1970557003 0 0) 0x7f9a6c13efc0 con 0x7f9a6c09ac90 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): my global_id is 4988 2024-09-16T05:37:10.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a6c13e2b0 con 0x7f9a6c09ac90 2024-09-16T05:37:10.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 504600187 0 0) 0x7f9a4c003530 con 0x7f9a6c09b780 2024-09-16T05:37:10.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a6c13efc0 con 0x7f9a6c09b780 2024-09-16T05:37:10.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1637581980 0 0) 0x7f9a6c13e2b0 con 0x7f9a6c09ac90 2024-09-16T05:37:10.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a4c003530 con 0x7f9a6c09ac90 2024-09-16T05:37:10.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9a54004150 con 0x7f9a6c09b780 2024-09-16T05:37:10.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:10.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:10.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9a5c001480 con 0x7f9a6c09ac90 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.077 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 835434076 0 0) 0x7f9a4c003530 con 0x7f9a6c09ac90 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 >> v1:172.21.15.63:6789/0 conn(0x7f9a6c0a0280 legacy=0x7f9a6c1d36b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 >> v1:172.21.15.6:6789/0 conn(0x7f9a6c09b780 legacy=0x7f9a6c09cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 monclient: found mon.c 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c09d250 con 0x7f9a6c09ac90 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:10.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 5 monclient: authenticate success, global_id 4988 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: _renew_subs 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6c13d730 con 0x7f9a6c09ac90 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: _renew_subs 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9a6c13ece0 con 0x7f9a6c09ac90 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9a5c001660 con 0x7f9a6c09ac90 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9a5c004c40 con 0x7f9a6c09ac90 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a6959a640 4 set_mon_vals no callback set 2024-09-16T05:37:10.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.059+0000 7f9a62ffd640 10 monclient: dump: 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.063+0000 7f9a70eda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.063+0000 7f9a70eda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.063+0000 7f9a70eda640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:10.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.063+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a50005090 con 0x7f9a6c09ac90 2024-09-16T05:37:10.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.063+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9a5c004de0 con 0x7f9a6c09ac90 2024-09-16T05:37:10.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.063+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(64..64 src has 1..64) ==== 4322+0+0 (unknown 2551441050 0 0) 0x7f9a5c044270 con 0x7f9a6c09ac90 2024-09-16T05:37:10.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.063+0000 7f9a63fff640 10 monclient: get_auth_request con 0x7f9a4c036a40 auth_method 0 2024-09-16T05:37:10.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.075+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9a5c015260 con 0x7f9a6c09ac90 2024-09-16T05:37:10.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.075+0000 7f9a62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:10.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.075+0000 7f9a62ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:10.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.235+0000 7f9a70eda640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T05:37:10.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.235+0000 7f9a70eda640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T05:37:10.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.235+0000 7f9a70eda640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:10.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.235+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) -- 0x7f9a50005380 con 0x7f9a6c09ac90 2024-09-16T05:37:10.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.411+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 v65) ==== 80+0+0 (unknown 2066902429 0 0) 0x7f9a5c014b60 con 0x7f9a6c09ac90 2024-09-16T05:37:10.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.415+0000 7f9a62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T05:37:10.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.415+0000 7f9a62ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:10.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.515+0000 7f9a70eda640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T05:37:10.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.515+0000 7f9a70eda640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T05:37:10.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.515+0000 7f9a70eda640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:10.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.515+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) -- 0x7f9a50005770 con 0x7f9a6c09ac90 2024-09-16T05:37:10.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.515+0000 7f9a62ffd640 1 -- 172.21.15.63:0/1102490803 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v65) ==== 103+0+0 (unknown 3520952052 0 0) 0x7f9a5c01d710 con 0x7f9a6c09ac90 2024-09-16T05:37:10.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.515+0000 7f9a62ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T05:37:10.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.515+0000 7f9a62ffd640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-16T05:37:10.520 INFO:tasks.workunit.client.0.smithi063.stderr:rule bar does not exist 2024-09-16T05:37:10.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.519+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9a4c036a40 legacy=0x7f9a4c038f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:10.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.519+0000 7f9a70eda640 10 monclient: shutdown 2024-09-16T05:37:10.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.519+0000 7f9a70eda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:10.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.519+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 >> v1:172.21.15.6:6790/0 conn(0x7f9a6c09ac90 legacy=0x7f9a6c09c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:10.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.519+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 shutdown_connections 2024-09-16T05:37:10.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.519+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 >> 172.21.15.63:0/1102490803 conn(0x7f9a6c0a5290 msgr2=0x7f9a6c137830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:10.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.519+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 shutdown_connections 2024-09-16T05:37:10.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.519+0000 7f9a70eda640 1 -- 172.21.15.63:0/1102490803 wait complete. 2024-09-16T05:37:10.537 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create pinning_pool 1 2024-09-16T05:37:10.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.655+0000 7f7b9da18640 10 monclient: get_monmap_and_config 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.655+0000 7f7b9da18640 10 monclient: build_initial_monmap 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.655+0000 7f7b9da18640 10 monclient: monmap: 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:10.660117+0000 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:10.660117+0000 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:10.661 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 Processor -- start 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- start start 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient: init 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9809b540 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient(hunting): picked mon.noname-a con 0x7f7b980a1520 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient(hunting): picked mon.noname-c con 0x7f7b98138d70 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9809f890 con 0x7f7b980a1520 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98072e80 con 0x7f7b98138d70 2024-09-16T05:37:10.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9809ce30 con 0x7f7b9809b540 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient(hunting): authenticate will time out at 1087.437510s 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b977fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7b98138d70 legacy=0x7f7b9813b160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b96ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7b9809b540 0x7f7b9809fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32818/0 (socket says 172.21.15.63:32818) 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b96ffd640 1 -- 172.21.15.63:0/1661554826 learned_addr learned my addr 172.21.15.63:0/1661554826 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3755496827 0 0) 0x7f7b9809f890 con 0x7f7b980a1520 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient(hunting): my global_id is 5004 2024-09-16T05:37:10.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b84003530 con 0x7f7b980a1520 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640618016 0 0) 0x7f7b9809ce30 con 0x7f7b9809b540 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient(hunting): my global_id is 4991 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b9809f890 con 0x7f7b9809b540 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 360030361 0 0) 0x7f7b9809f890 con 0x7f7b9809b540 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 >> v1:172.21.15.63:6789/0 conn(0x7f7b98138d70 legacy=0x7f7b9813b160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 >> v1:172.21.15.6:6789/0 conn(0x7f7b980a1520 legacy=0x7f7b980a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 monclient: found mon.noname-b 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9809d110 con 0x7f7b9809b540 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7b80001580 con 0x7f7b9809b540 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: dump: 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.668 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 5 monclient: authenticate success, global_id 4991 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b80001870 con 0x7f7b9809b540 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 1 -- 172.21.15.63:0/1661554826 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7b80001c80 con 0x7f7b9809b540 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:10.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b95ffb640 10 monclient: dump: 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:10.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 4 set_mon_vals no callback set 2024-09-16T05:37:10.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 10 monclient: shutdown 2024-09-16T05:37:10.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:10.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- 172.21.15.63:0/1661554826 >> v1:172.21.15.6:6790/0 conn(0x7f7b9809b540 legacy=0x7f7b9809fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:10.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- 172.21.15.63:0/1661554826 shutdown_connections 2024-09-16T05:37:10.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- 172.21.15.63:0/1661554826 >> 172.21.15.63:0/1661554826 conn(0x7f7b980a5290 msgr2=0x7f7b980a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:10.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- 172.21.15.63:0/1661554826 shutdown_connections 2024-09-16T05:37:10.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.659+0000 7f7b9da18640 1 -- 172.21.15.63:0/1661554826 wait complete. 2024-09-16T05:37:10.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: build_initial_monmap 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: monmap: 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:10.676 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 1 Processor -- start 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 1 -- start start 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: init 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient(hunting): picked mon.noname-c con 0x7f7b9809b540 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient(hunting): picked mon.noname-b con 0x7f7b980a1520 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient(hunting): picked mon.noname-a con 0x7f7b98138d70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98122010 con 0x7f7b98138d70 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98072a60 con 0x7f7b980a1520 2024-09-16T05:37:10.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b967fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7b980a1520 legacy=0x7f7b98132910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98138950 con 0x7f7b9809b540 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient(hunting): authenticate will time out at 1087.441893s 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b96ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7b9809b540 0x7f7b981cb110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32824/0 (socket says 172.21.15.63:32824) 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b96ffd640 1 -- 172.21.15.63:0/2433596618 learned_addr learned my addr 172.21.15.63:0/2433596618 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b977fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7b98138d70 0x7f7b981d34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44474/0 (socket says 172.21.15.63:44474) 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443271747 0 0) 0x7f7b98122010 con 0x7f7b98138d70 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): my global_id is 5007 2024-09-16T05:37:10.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b78003530 con 0x7f7b98138d70 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2517834537 0 0) 0x7f7b98138950 con 0x7f7b9809b540 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): my global_id is 4994 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b98122010 con 0x7f7b9809b540 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 884882627 0 0) 0x7f7b78003530 con 0x7f7b98138d70 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b98138950 con 0x7f7b98138d70 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7b90004210 con 0x7f7b98138d70 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): dump: 2024-09-16T05:37:10.679 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:10.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3684997881 0 0) 0x7f7b98122010 con 0x7f7b9809b540 2024-09-16T05:37:10.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b78003530 con 0x7f7b9809b540 2024-09-16T05:37:10.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7b80001bc0 con 0x7f7b9809b540 2024-09-16T05:37:10.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:10.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:10.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient(hunting): dump: 2024-09-16T05:37:10.681 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.681 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.681 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.682 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.682 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.682 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.682 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.682 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.682 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.682 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3256108267 0 0) 0x7f7b98138950 con 0x7f7b98138d70 2024-09-16T05:37:10.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 >> v1:172.21.15.6:6790/0 conn(0x7f7b9809b540 legacy=0x7f7b981cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:10.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 >> v1:172.21.15.63:6789/0 conn(0x7f7b980a1520 legacy=0x7f7b98132910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:10.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:10.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 monclient: found mon.a 2024-09-16T05:37:10.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:10.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9809c0f0 con 0x7f7b98138d70 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient: _finish_auth 0 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 5 monclient: authenticate success, global_id 5007 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: _renew_subs 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7b98121d30 con 0x7f7b98138d70 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: _renew_subs 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:10.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7b98133020 con 0x7f7b98138d70 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b900047e0 con 0x7f7b98138d70 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7b90004b30 con 0x7f7b98138d70 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b8ffff640 10 monclient: dump: 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.663+0000 7f7b957fa640 4 set_mon_vals no callback set 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.667+0000 7f7b9da18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.667+0000 7f7b9da18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.667+0000 7f7b9da18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.667+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b7c005090 con 0x7f7b98138d70 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.667+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7b9000f8d0 con 0x7f7b98138d70 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.667+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(65..65 src has 1..65) ==== 4267+0+0 (unknown 3260315123 0 0) 0x7f7b90044440 con 0x7f7b98138d70 2024-09-16T05:37:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.667+0000 7f7b96ffd640 10 monclient: get_auth_request con 0x7f7b78036b10 auth_method 0 2024-09-16T05:37:10.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.679+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7b900153b0 con 0x7f7b98138d70 2024-09-16T05:37:10.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.679+0000 7f7b8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:10.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.679+0000 7f7b8ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:10.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.839+0000 7f7b9da18640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T05:37:10.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.839+0000 7f7b9da18640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T05:37:10.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.839+0000 7f7b9da18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:10.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:10.839+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) -- 0x7f7b7c005b90 con 0x7f7b98138d70 2024-09-16T05:37:11.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.415+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' created v66) ==== 127+0+0 (unknown 2786053607 0 0) 0x7f7b90019060 con 0x7f7b98138d70 2024-09-16T05:37:11.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.415+0000 7f7b8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T05:37:11.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.415+0000 7f7b8ffff640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' created 2024-09-16T05:37:11.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.515+0000 7f7b9da18640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T05:37:11.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.515+0000 7f7b9da18640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T05:37:11.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.515+0000 7f7b9da18640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:11.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.515+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) -- 0x7f7b7c005610 con 0x7f7b98138d70 2024-09-16T05:37:11.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.515+0000 7f7b8ffff640 1 -- 172.21.15.63:0/2433596618 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' already exists v66) ==== 134+0+0 (unknown 1360820213 0 0) 0x7f7b9001d920 con 0x7f7b98138d70 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.515+0000 7f7b8ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.515+0000 7f7b8ffff640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' already exists 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'pinning_pool' already exists 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.519+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7b78036b10 legacy=0x7f7b78038fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.519+0000 7f7b9da18640 10 monclient: shutdown 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.519+0000 7f7b9da18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.519+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 >> v1:172.21.15.6:6789/0 conn(0x7f7b98138d70 legacy=0x7f7b981d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.519+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 shutdown_connections 2024-09-16T05:37:11.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.519+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 >> 172.21.15.63:0/2433596618 conn(0x7f7b980a5290 msgr2=0x7f7b980a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.519+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 shutdown_connections 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.519+0000 7f7b9da18640 1 -- 172.21.15.63:0/2433596618 wait complete. 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rm replicated_rule 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.655+0000 7f0814ce5640 10 monclient: get_monmap_and_config 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.655+0000 7f0814ce5640 10 monclient: build_initial_monmap 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.655+0000 7f0814ce5640 10 monclient: monmap: 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:11.659489+0000 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:11.659489+0000 2024-09-16T05:37:11.702 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 Processor -- start 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- start start 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient: init 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient(hunting): picked mon.noname-a con 0x7f0810075510 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient(hunting): picked mon.noname-b con 0x7f0810136dd0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:11.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient(hunting): picked mon.noname-c con 0x7f081013d150 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0810137490 con 0x7f0810075510 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08101371d0 con 0x7f081013d150 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0810099e70 con 0x7f0810136dd0 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient(hunting): authenticate will time out at 1088.436963s 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080ed9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f081013d150 legacy=0x7f081013f540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0807fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0810136dd0 0x7f081013ba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32838/0 (socket says 172.21.15.63:32838) 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0807fff640 1 -- 172.21.15.63:0/263843880 learned_addr learned my addr 172.21.15.63:0/263843880 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080e59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0810075510 0x7f081009bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44478/0 (socket says 172.21.15.63:44478) 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4045177487 0 0) 0x7f0810137490 con 0x7f0810075510 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:11.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient(hunting): my global_id is 5010 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07fc003530 con 0x7f0810075510 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 803538630 0 0) 0x7f0810099e70 con 0x7f0810136dd0 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient(hunting): my global_id is 4997 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0810137490 con 0x7f0810136dd0 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1540388540 0 0) 0x7f07fc003530 con 0x7f0810075510 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 >> v1:172.21.15.6:6790/0 conn(0x7f0810136dd0 legacy=0x7f081013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 >> v1:172.21.15.63:6789/0 conn(0x7f081013d150 legacy=0x7f081013f540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 monclient: found mon.noname-a 2024-09-16T05:37:11.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08100758f0 con 0x7f0810075510 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: _finish_auth 0 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 5 monclient: authenticate success, global_id 5010 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0800001620 con 0x7f0810075510 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:11.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: dump: 2024-09-16T05:37:11.707 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.707 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.707 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.707 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0800001880 con 0x7f0810075510 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 4 set_mon_vals no callback set 2024-09-16T05:37:11.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 1 -- 172.21.15.63:0/263843880 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0800001c90 con 0x7f0810075510 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f080dd9b640 10 monclient: dump: 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.709 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient: shutdown 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- 172.21.15.63:0/263843880 >> v1:172.21.15.6:6789/0 conn(0x7f0810075510 legacy=0x7f081009bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- 172.21.15.63:0/263843880 shutdown_connections 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- 172.21.15.63:0/263843880 >> 172.21.15.63:0/263843880 conn(0x7f08100a2020 msgr2=0x7f08100a2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- 172.21.15.63:0/263843880 shutdown_connections 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 1 -- 172.21.15.63:0/263843880 wait complete. 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient: build_initial_monmap 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.659+0000 7f0814ce5640 10 monclient: monmap: 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:11.712 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 1 Processor -- start 2024-09-16T05:37:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 1 -- start start 2024-09-16T05:37:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: init 2024-09-16T05:37:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient(hunting): picked mon.noname-b con 0x7f0810075510 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient(hunting): picked mon.noname-a con 0x7f0810136dd0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient(hunting): picked mon.noname-c con 0x7f081013d150 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f081008cd70 con 0x7f0810136dd0 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08101ccb20 con 0x7f0810075510 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f081008e360 con 0x7f081013d150 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient(hunting): authenticate will time out at 1088.440792s 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f080e59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0810075510 legacy=0x7f081008e6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0807fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0810136dd0 0x7f08101cbd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44480/0 (socket says 172.21.15.63:44480) 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f080ed9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f081013d150 0x7f08101cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32844/0 (socket says 172.21.15.63:32844) 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0807fff640 1 -- 172.21.15.63:0/3055774220 learned_addr learned my addr 172.21.15.63:0/3055774220 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 227580181 0 0) 0x7f081008cd70 con 0x7f0810136dd0 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): my global_id is 5013 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f0003560 con 0x7f0810136dd0 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487332192 0 0) 0x7f081008e360 con 0x7f081013d150 2024-09-16T05:37:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): my global_id is 5000 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f081008cd70 con 0x7f081013d150 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3838982932 0 0) 0x7f081008cd70 con 0x7f081013d150 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f081008e360 con 0x7f081013d150 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f08080042f0 con 0x7f081013d150 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4193803081 0 0) 0x7f07f0003560 con 0x7f0810136dd0 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f081008cd70 con 0x7f0810136dd0 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07f8001950 con 0x7f0810136dd0 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1242459943 0 0) 0x7f081008e360 con 0x7f081013d150 2024-09-16T05:37:11.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 >> v1:172.21.15.63:6789/0 conn(0x7f0810075510 legacy=0x7f081008e6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 >> v1:172.21.15.6:6789/0 conn(0x7f0810136dd0 legacy=0x7f08101cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 monclient: found mon.c 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081008d710 con 0x7f081013d150 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 5 monclient: authenticate success, global_id 5000 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: _renew_subs 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:11.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f081008c880 con 0x7f081013d150 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: _renew_subs 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f08101d5140 con 0x7f081013d150 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0808004990 con 0x7f081013d150 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0808004ce0 con 0x7f081013d150 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f080d59a640 4 set_mon_vals no callback set 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0806ffd640 10 monclient: dump: 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.663+0000 7f0814ce5640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f4005090 con 0x7f081013d150 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.667+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f080800fb00 con 0x7f081013d150 2024-09-16T05:37:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.667+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(66..66 src has 1..66) ==== 4618+0+0 (unknown 246164099 0 0) 0x7f0808044770 con 0x7f081013d150 2024-09-16T05:37:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.667+0000 7f080e59c640 10 monclient: get_auth_request con 0x7f07f0036a00 auth_method 0 2024-09-16T05:37:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.679+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0808015640 con 0x7f081013d150 2024-09-16T05:37:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.679+0000 7f0806ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.679+0000 7f0806ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:11.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.847+0000 7f0814ce5640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T05:37:11.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.847+0000 7f0814ce5640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T05:37:11.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.847+0000 7f0814ce5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:11.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.847+0000 7f0814ce5640 1 -- 172.21.15.63:0/3055774220 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "replicated_rule"} v 0) -- 0x7f07f4005b90 con 0x7f081013d150 2024-09-16T05:37:11.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.847+0000 7f0806ffd640 1 -- 172.21.15.63:0/3055774220 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "replicated_rule"}]=-16 crush rule replicated_rule (0) is in use v66) ==== 132+0+0 (unknown 4153333503 0 0) 0x7f0808014f40 con 0x7f081013d150 2024-09-16T05:37:11.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.847+0000 7f0806ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T05:37:11.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.847+0000 7f0806ffd640 10 monclient: _finish_command 2 = mon:16 crush rule replicated_rule (0) is in use 2024-09-16T05:37:11.852 INFO:tasks.workunit.client.0.smithi063.stderr:Error EBUSY: crush rule replicated_rule (0) is in use 2024-09-16T05:37:11.856 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:11.856 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2024-09-16T05:37:11.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient: get_monmap_and_config 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient: build_initial_monmap 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient: monmap: 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:11.977272+0000 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:11.977272+0000 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.978 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:11.979 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:11.979 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:11.979 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 1 Processor -- start 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 1 -- start start 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient: init 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient(hunting): picked mon.noname-a con 0x7f81941339b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient(hunting): picked mon.noname-b con 0x7f819409fc50 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient(hunting): picked mon.noname-c con 0x7f819413d140 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:11.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81940a0710 con 0x7f81941339b0 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81940a09f0 con 0x7f819413d140 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81940a0cd0 con 0x7f819409fc50 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient(hunting): authenticate will time out at 1088.754121s 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81990f9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f819413d140 legacy=0x7f819413f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f8193fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f819409fc50 0x7f819413ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32858/0 (socket says 172.21.15.63:32858) 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f8193fff640 1 -- 172.21.15.63:0/1566189941 learned_addr learned my addr 172.21.15.63:0/1566189941 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775254265 0 0) 0x7f81940a0cd0 con 0x7f819409fc50 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient(hunting): my global_id is 5003 2024-09-16T05:37:11.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f817c003530 con 0x7f819409fc50 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3489025807 0 0) 0x7f81940a0710 con 0x7f81941339b0 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient(hunting): my global_id is 5016 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81940a0cd0 con 0x7f81941339b0 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 513940860 0 0) 0x7f81940a0cd0 con 0x7f81941339b0 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 >> v1:172.21.15.6:6790/0 conn(0x7f819409fc50 legacy=0x7f819413ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:11.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 >> v1:172.21.15.63:6789/0 conn(0x7f819413d140 legacy=0x7f819413f530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 monclient: found mon.noname-a 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81940a15b0 con 0x7f81941339b0 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8184001600 con 0x7f81941339b0 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:11.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: dump: 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 5 monclient: authenticate success, global_id 5016 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81840018e0 con 0x7f81941339b0 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f819ab5c640 4 set_mon_vals no callback set 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 1 -- 172.21.15.63:0/1566189941 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8184001cf0 con 0x7f81941339b0 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.975+0000 7f81937fe640 10 monclient: dump: 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.988 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: shutdown 2024-09-16T05:37:11.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- 172.21.15.63:0/1566189941 >> v1:172.21.15.6:6789/0 conn(0x7f81941339b0 legacy=0x7f8194133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- 172.21.15.63:0/1566189941 shutdown_connections 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- 172.21.15.63:0/1566189941 >> 172.21.15.63:0/1566189941 conn(0x7f81940a1e90 msgr2=0x7f8194075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- 172.21.15.63:0/1566189941 shutdown_connections 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- 172.21.15.63:0/1566189941 wait complete. 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: build_initial_monmap 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: monmap: 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:11.990 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:11.991 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:11.991 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.991 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:11.991 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:11.991 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:11.991 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 Processor -- start 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- start start 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: init 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient(hunting): picked mon.noname-b con 0x7f819409fc50 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient(hunting): picked mon.noname-c con 0x7f81941339b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient(hunting): picked mon.noname-a con 0x7f819413d140 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8194050560 con 0x7f819413d140 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81941328b0 con 0x7f819409fc50 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f819412fa20 con 0x7f81941339b0 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:11.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient(hunting): authenticate will time out at 1088.757914s 2024-09-16T05:37:11.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81988f8640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f819409fc50 legacy=0x7f819412ec00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:11.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81990f9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f819413d140 0x7f81941d3c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44510/0 (socket says 172.21.15.63:44510) 2024-09-16T05:37:11.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f8193fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f81941339b0 0x7f819412f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32872/0 (socket says 172.21.15.63:32872) 2024-09-16T05:37:11.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81990f9640 1 -- 172.21.15.63:0/927437759 learned_addr learned my addr 172.21.15.63:0/927437759 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:11.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586285060 0 0) 0x7f819412fa20 con 0x7f81941339b0 2024-09-16T05:37:11.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): my global_id is 5006 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8180003530 con 0x7f81941339b0 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3135280561 0 0) 0x7f8194050560 con 0x7f819413d140 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): my global_id is 5019 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f819412fa20 con 0x7f819413d140 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1460563967 0 0) 0x7f8180003530 con 0x7f81941339b0 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8194050560 con 0x7f81941339b0 2024-09-16T05:37:11.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8188001820 con 0x7f81941339b0 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): dump: 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.997 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1854577685 0 0) 0x7f819412fa20 con 0x7f819413d140 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8180003530 con 0x7f819413d140 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f818c0043c0 con 0x7f819413d140 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient(hunting): dump: 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.998 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2894918097 0 0) 0x7f8194050560 con 0x7f81941339b0 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 >> v1:172.21.15.63:6789/0 conn(0x7f819409fc50 legacy=0x7f819412ec00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 >> v1:172.21.15.6:6789/0 conn(0x7f819413d140 legacy=0x7f81941d3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 monclient: found mon.c 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:11.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819413c8b0 con 0x7f81941339b0 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f81917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 5 monclient: authenticate success, global_id 5006 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: _renew_subs 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8194140830 con 0x7f81941339b0 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: _renew_subs 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.979+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8194132b90 con 0x7f81941339b0 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81880047f0 con 0x7f81941339b0 2024-09-16T05:37:12.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81917fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:12.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8188004b40 con 0x7f81941339b0 2024-09-16T05:37:12.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:12.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f8192ffd640 4 set_mon_vals no callback set 2024-09-16T05:37:12.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:12.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81917fa640 10 monclient: dump: 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f819ab5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:12.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f819ab5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:12.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f819ab5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8174005090 con 0x7f81941339b0 2024-09-16T05:37:12.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f818800f8e0 con 0x7f81941339b0 2024-09-16T05:37:12.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(66..66 src has 1..66) ==== 4618+0+0 (unknown 246164099 0 0) 0x7f8188044430 con 0x7f81941339b0 2024-09-16T05:37:12.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.983+0000 7f81988f8640 10 monclient: get_auth_request con 0x7f8180036a70 auth_method 0 2024-09-16T05:37:12.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.995+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f81880152a0 con 0x7f81941339b0 2024-09-16T05:37:12.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.995+0000 7f81917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:12.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:11.995+0000 7f81917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:12.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.151+0000 7f819ab5c640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:37:12.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.151+0000 7f819ab5c640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:37:12.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.151+0000 7f819ab5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.151+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8174005380 con 0x7f81941339b0 2024-09-16T05:37:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.435+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' removed v67) ==== 172+0+0 (unknown 3726064440 0 0) 0x7f8188014ba0 con 0x7f81941339b0 2024-09-16T05:37:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.435+0000 7f81917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:37:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.435+0000 7f81917fa640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' removed 2024-09-16T05:37:12.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.531+0000 7f819ab5c640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:37:12.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.531+0000 7f819ab5c640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:37:12.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.531+0000 7f819ab5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.531+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8174000cb0 con 0x7f81941339b0 2024-09-16T05:37:12.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.531+0000 7f81917fa640 1 -- 172.21.15.63:0/927437759 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' does not exist v67) ==== 179+0+0 (unknown 671324971 0 0) 0x7f8188018ef0 con 0x7f81941339b0 2024-09-16T05:37:12.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.531+0000 7f81917fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:37:12.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.531+0000 7f81917fa640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' does not exist 2024-09-16T05:37:12.537 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'pinning_pool' does not exist 2024-09-16T05:37:12.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.535+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8180036a70 legacy=0x7f8180038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:12.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.535+0000 7f819ab5c640 10 monclient: shutdown 2024-09-16T05:37:12.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.535+0000 7f819ab5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:12.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.535+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 >> v1:172.21.15.6:6790/0 conn(0x7f81941339b0 legacy=0x7f819412f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:12.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.535+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 shutdown_connections 2024-09-16T05:37:12.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.535+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 >> 172.21.15.63:0/927437759 conn(0x7f81940a1e90 msgr2=0x7f819409d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:12.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.535+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 shutdown_connections 2024-09-16T05:37:12.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.535+0000 7f819ab5c640 1 -- 172.21.15.63:0/927437759 wait complete. 2024-09-16T05:37:12.552 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush add-bucket foo osd 2024-09-16T05:37:12.553 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:12.553 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket foo osd 2024-09-16T05:37:12.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.667+0000 7f06aec3e640 10 monclient: get_monmap_and_config 2024-09-16T05:37:12.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.667+0000 7f06aec3e640 10 monclient: build_initial_monmap 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.667+0000 7f06aec3e640 10 monclient: monmap: 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:12.672072+0000 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:12.672072+0000 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:12.673 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 Processor -- start 2024-09-16T05:37:12.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- start start 2024-09-16T05:37:12.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient: init 2024-09-16T05:37:12.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:12.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:12.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient(hunting): picked mon.noname-b con 0x7f06a8136a50 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient(hunting): picked mon.noname-c con 0x7f06a809fd70 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient(hunting): picked mon.noname-a con 0x7f06a813d0d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a80a05a0 con 0x7f06a813d0d0 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a80a0880 con 0x7f06a809fd70 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a80a0b60 con 0x7f06a8136a50 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient(hunting): authenticate will time out at 1089.449095s 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f06a809fd70 legacy=0x7f06a813b9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06ad1db640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f06a813d0d0 0x7f06a813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44516/0 (socket says 172.21.15.63:44516) 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06ac9da640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f06a8136a50 0x7f06a8136e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32880/0 (socket says 172.21.15.63:32880) 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06ad1db640 1 -- 172.21.15.63:0/1350641577 learned_addr learned my addr 172.21.15.63:0/1350641577 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2113170632 0 0) 0x7f06a80a0b60 con 0x7f06a8136a50 2024-09-16T05:37:12.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient(hunting): my global_id is 5009 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0698003530 con 0x7f06a8136a50 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001538265 0 0) 0x7f06a80a05a0 con 0x7f06a813d0d0 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient(hunting): my global_id is 5022 2024-09-16T05:37:12.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a80a0b60 con 0x7f06a813d0d0 2024-09-16T05:37:12.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 846970211 0 0) 0x7f0698003530 con 0x7f06a8136a50 2024-09-16T05:37:12.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 >> v1:172.21.15.63:6789/0 conn(0x7f06a809fd70 legacy=0x7f06a813b9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:12.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 >> v1:172.21.15.6:6789/0 conn(0x7f06a813d0d0 legacy=0x7f06a813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:12.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 monclient: found mon.noname-b 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a8137540 con 0x7f06a8136a50 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f069c001620 con 0x7f06a8136a50 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: dump: 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:12.683 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 5 monclient: authenticate success, global_id 5009 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f069c001880 con 0x7f06a8136a50 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:12.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 1 -- 172.21.15.63:0/1350641577 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f069c001c10 con 0x7f06a8136a50 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 4 set_mon_vals no callback set 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06a77fe640 10 monclient: dump: 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.685 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 10 monclient: shutdown 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- 172.21.15.63:0/1350641577 >> v1:172.21.15.6:6790/0 conn(0x7f06a8136a50 legacy=0x7f06a8136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- 172.21.15.63:0/1350641577 shutdown_connections 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- 172.21.15.63:0/1350641577 >> 172.21.15.63:0/1350641577 conn(0x7f06a80a5290 msgr2=0x7f06a80a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:12.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- 172.21.15.63:0/1350641577 shutdown_connections 2024-09-16T05:37:12.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.671+0000 7f06aec3e640 1 -- 172.21.15.63:0/1350641577 wait complete. 2024-09-16T05:37:12.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: build_initial_monmap 2024-09-16T05:37:12.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: monmap: 2024-09-16T05:37:12.687 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:12.687 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:12.687 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:12.687 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:12.687 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:12.688 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.688 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:12.688 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:12.688 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:12.688 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 1 Processor -- start 2024-09-16T05:37:12.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 1 -- start start 2024-09-16T05:37:12.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: init 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient(hunting): picked mon.noname-b con 0x7f06a809fd70 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient(hunting): picked mon.noname-a con 0x7f06a8136a50 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient(hunting): picked mon.noname-c con 0x7f06a813d0d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a8120650 con 0x7f06a8136a50 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a811f200 con 0x7f06a809fd70 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06ac9da640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f06a809fd70 legacy=0x7f06a81cb4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a811f480 con 0x7f06a813d0d0 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient(hunting): authenticate will time out at 1089.453339s 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f06a8136a50 0x7f06a81d0490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44526/0 (socket says 172.21.15.63:44526) 2024-09-16T05:37:12.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a7fff640 1 -- 172.21.15.63:0/913670216 learned_addr learned my addr 172.21.15.63:0/913670216 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1118407642 0 0) 0x7f06a811f480 con 0x7f06a813d0d0 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): my global_id is 5012 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f068c003530 con 0x7f06a813d0d0 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217691070 0 0) 0x7f06a8120650 con 0x7f06a8136a50 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): my global_id is 5025 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a811f480 con 0x7f06a8136a50 2024-09-16T05:37:12.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 563744324 0 0) 0x7f068c003530 con 0x7f06a813d0d0 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06a8120650 con 0x7f06a813d0d0 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f06a0001b80 con 0x7f06a813d0d0 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): dump: 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:12.691 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3026146837 0 0) 0x7f06a811f480 con 0x7f06a8136a50 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f068c003530 con 0x7f06a8136a50 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f06940040e0 con 0x7f06a8136a50 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient(hunting): dump: 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.692 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1138553661 0 0) 0x7f06a8120650 con 0x7f06a813d0d0 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 >> v1:172.21.15.63:6789/0 conn(0x7f06a809fd70 legacy=0x7f06a81cb4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 >> v1:172.21.15.6:6789/0 conn(0x7f06a8136a50 legacy=0x7f06a81d0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 monclient: found mon.c 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a8120930 con 0x7f06a813d0d0 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 5 monclient: authenticate success, global_id 5012 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: _renew_subs 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f06a80a3fa0 con 0x7f06a813d0d0 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: _renew_subs 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06aec3e640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f06a811dd30 con 0x7f06a813d0d0 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06a0004b70 con 0x7f06a813d0d0 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:12.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f06a0004ec0 con 0x7f06a813d0d0 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a6ffd640 4 set_mon_vals no callback set 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.675+0000 7f06a57fa640 10 monclient: dump: 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:12.695 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.679+0000 7f06aec3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:12.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.679+0000 7f06aec3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:12.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.679+0000 7f06aec3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.679+0000 7f06aec3e640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0688005090 con 0x7f06a813d0d0 2024-09-16T05:37:12.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.679+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f06a000fc60 con 0x7f06a813d0d0 2024-09-16T05:37:12.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.679+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(67..67 src has 1..67) ==== 4267+0+0 (unknown 979939819 0 0) 0x7f06a0044630 con 0x7f06a813d0d0 2024-09-16T05:37:12.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.679+0000 7f06ac9da640 10 monclient: get_auth_request con 0x7f068c036a30 auth_method 0 2024-09-16T05:37:12.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.691+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f06a0015660 con 0x7f06a813d0d0 2024-09-16T05:37:12.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.691+0000 7f06a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:12.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.691+0000 7f06a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:12.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.859+0000 7f06aec3e640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T05:37:12.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.859+0000 7f06aec3e640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T05:37:12.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.859+0000 7f06aec3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.859+0000 7f06aec3e640 1 -- 172.21.15.63:0/913670216 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"} v 0) -- 0x7f0688005b90 con 0x7f06a813d0d0 2024-09-16T05:37:12.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.863+0000 7f06a57fa640 1 -- 172.21.15.63:0/913670216 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}]=-22 type 'osd' is for devices, not buckets v67) ==== 136+0+0 (unknown 3056906605 0 0) 0x7f06a0014f60 con 0x7f06a813d0d0 2024-09-16T05:37:12.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.863+0000 7f06a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T05:37:12.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.863+0000 7f06a57fa640 10 monclient: _finish_command 2 = mon:22 type 'osd' is for devices, not buckets 2024-09-16T05:37:12.865 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2024-09-16T05:37:12.870 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:12.870 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket foo root 2024-09-16T05:37:12.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient: get_monmap_and_config 2024-09-16T05:37:12.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient: build_initial_monmap 2024-09-16T05:37:12.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient: monmap: 2024-09-16T05:37:12.986 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:12.985937+0000 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:12.985937+0000 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:12.987 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 1 Processor -- start 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 1 -- start start 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient: init 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient(hunting): picked mon.noname-a con 0x7f962409ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient(hunting): picked mon.noname-b con 0x7f96240a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient(hunting): picked mon.noname-c con 0x7f962409b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9624074200 con 0x7f962409ac90 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f962409f900 con 0x7f962409b780 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f962409e870 con 0x7f96240a0280 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:12.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9629240640 10 monclient(hunting): authenticate will time out at 1089.763015s 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962359e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f962409b780 legacy=0x7f962413cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9622d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f962409ac90 0x7f962409b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44530/0 (socket says 172.21.15.63:44530) 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f9622d9d640 1 -- 172.21.15.63:0/3833938352 learned_addr learned my addr 172.21.15.63:0/3833938352 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f961bfff640 1 --1- 172.21.15.63:0/3833938352 >> v1:172.21.15.6:6790/0 conn(0x7f96240a0280 0x7f96241398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32884/0 (socket says 172.21.15.63:32884) 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3518583389 0 0) 0x7f962409e870 con 0x7f96240a0280 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 10 monclient(hunting): _negotiate 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 10 monclient(hunting): my global_id is 5015 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9610003530 con 0x7f96240a0280 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130021830 0 0) 0x7f9624074200 con 0x7f962409ac90 2024-09-16T05:37:12.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 10 monclient(hunting): _negotiate 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 10 monclient(hunting): my global_id is 5028 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.983+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f962409e870 con 0x7f962409ac90 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3015593223 0 0) 0x7f9610003530 con 0x7f96240a0280 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 >> v1:172.21.15.63:6789/0 conn(0x7f962409b780 legacy=0x7f962413cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 >> v1:172.21.15.6:6789/0 conn(0x7f962409ac90 legacy=0x7f962409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: _finish_hunting 0 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 1 monclient: found mon.noname-b 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9624141730 con 0x7f96240a0280 2024-09-16T05:37:12.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: _finish_auth 0 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: _check_auth_tickets 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f960c001620 con 0x7f96240a0280 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: dump: 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.993 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 5 monclient: authenticate success, global_id 5015 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f960c001880 con 0x7f96240a0280 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 1 -- 172.21.15.63:0/3833938352 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f960c001bd0 con 0x7f96240a0280 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:12.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f962259c640 10 monclient: dump: 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:12.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 4 set_mon_vals no callback set 2024-09-16T05:37:13.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient: shutdown 2024-09-16T05:37:13.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:13.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- 172.21.15.63:0/3833938352 >> v1:172.21.15.6:6790/0 conn(0x7f96240a0280 legacy=0x7f96241398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:13.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- 172.21.15.63:0/3833938352 shutdown_connections 2024-09-16T05:37:13.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- 172.21.15.63:0/3833938352 >> 172.21.15.63:0/3833938352 conn(0x7f96240a5290 msgr2=0x7f96240a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- 172.21.15.63:0/3833938352 shutdown_connections 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- 172.21.15.63:0/3833938352 wait complete. 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient: build_initial_monmap 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient: monmap: 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:13.001 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 Processor -- start 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- start start 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient: init 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient(hunting): picked mon.noname-b con 0x7f962409ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient(hunting): picked mon.noname-c con 0x7f962409b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9622d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f962409ac90 legacy=0x7f9624134050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:13.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient(hunting): picked mon.noname-a con 0x7f96240a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f962413dbf0 con 0x7f96240a0280 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f962406df30 con 0x7f962409ac90 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f962413d910 con 0x7f962409b780 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.987+0000 7f9629240640 10 monclient(hunting): authenticate will time out at 1089.767659s 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f962409b780 0x7f96241cd660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32888/0 (socket says 172.21.15.63:32888) 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f962359e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f96240a0280 0x7f96241d0d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44536/0 (socket says 172.21.15.63:44536) 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961bfff640 1 -- 172.21.15.63:0/3161179140 learned_addr learned my addr 172.21.15.63:0/3161179140 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 787053374 0 0) 0x7f962413d910 con 0x7f962409b780 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:13.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): my global_id is 5018 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9604003530 con 0x7f962409b780 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1025871913 0 0) 0x7f962413dbf0 con 0x7f96240a0280 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): my global_id is 5031 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f962413d910 con 0x7f96240a0280 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3621825557 0 0) 0x7f9604003530 con 0x7f962409b780 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f962413dbf0 con 0x7f962409b780 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f960c0047f0 con 0x7f962409b780 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:13.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): dump: 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:13.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3855151536 0 0) 0x7f962413d910 con 0x7f96240a0280 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9604003530 con 0x7f96240a0280 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f961c004300 con 0x7f96240a0280 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient(hunting): dump: 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:13.006 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2092428112 0 0) 0x7f962413dbf0 con 0x7f962409b780 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 >> v1:172.21.15.63:6789/0 conn(0x7f962409ac90 legacy=0x7f9624134050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 >> v1:172.21.15.6:6789/0 conn(0x7f96240a0280 legacy=0x7f96241d0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 monclient: found mon.c 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962413ded0 con 0x7f962409b780 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:13.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 5 monclient: authenticate success, global_id 5018 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 10 monclient: _renew_subs 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f962413c4c0 con 0x7f962409b780 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 10 monclient: _renew_subs 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f96241d2680 con 0x7f962409b780 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f960c001860 con 0x7f962409b780 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9621d9b640 4 set_mon_vals no callback set 2024-09-16T05:37:13.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f960c004de0 con 0x7f962409b780 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 10 monclient: dump: 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:13.009 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:13.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:13.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:13.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9608005090 con 0x7f962409b780 2024-09-16T05:37:13.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f960c00fb80 con 0x7f962409b780 2024-09-16T05:37:13.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(67..67 src has 1..67) ==== 4267+0+0 (unknown 979939819 0 0) 0x7f960c044440 con 0x7f962409b780 2024-09-16T05:37:13.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:12.991+0000 7f9622d9d640 10 monclient: get_auth_request con 0x7f9604036a50 auth_method 0 2024-09-16T05:37:13.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.003+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f960c019060 con 0x7f962409b780 2024-09-16T05:37:13.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.003+0000 7f961b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:13.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.003+0000 7f961b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:13.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.167+0000 7f9629240640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T05:37:13.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.167+0000 7f9629240640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T05:37:13.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.167+0000 7f9629240640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:13.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.167+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) -- 0x7f9608005b90 con 0x7f962409b780 2024-09-16T05:37:13.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.447+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 added bucket foo type root to crush map v68) ==== 138+0+0 (unknown 1666432368 0 0) 0x7f960c01d920 con 0x7f962409b780 2024-09-16T05:37:13.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.447+0000 7f961b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T05:37:13.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.447+0000 7f961b7fe640 10 monclient: _finish_command 2 = system:0 added bucket foo type root to crush map 2024-09-16T05:37:13.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.543+0000 7f9629240640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T05:37:13.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.543+0000 7f9629240640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T05:37:13.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.543+0000 7f9629240640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:13.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.543+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) -- 0x7f9608005610 con 0x7f962409b780 2024-09-16T05:37:13.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.547+0000 7f961b7fe640 1 -- 172.21.15.63:0/3161179140 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 bucket 'foo' already exists v68) ==== 126+0+0 (unknown 1036427144 0 0) 0x7f960c040280 con 0x7f962409b780 2024-09-16T05:37:13.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.547+0000 7f961b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T05:37:13.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.547+0000 7f961b7fe640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-16T05:37:13.551 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'foo' already exists 2024-09-16T05:37:13.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.547+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9604036a50 legacy=0x7f9604038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:13.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.547+0000 7f9629240640 10 monclient: shutdown 2024-09-16T05:37:13.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.547+0000 7f9629240640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:13.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.547+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 >> v1:172.21.15.6:6790/0 conn(0x7f962409b780 legacy=0x7f96241cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:13.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.551+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 shutdown_connections 2024-09-16T05:37:13.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.551+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 >> 172.21.15.63:0/3161179140 conn(0x7f96240a5290 msgr2=0x7f9624136890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:13.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.551+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 shutdown_connections 2024-09-16T05:37:13.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.551+0000 7f9629240640 1 -- 172.21.15.63:0/3161179140 wait complete. 2024-09-16T05:37:13.567 INFO:tasks.workunit.client.0.smithi063.stderr:++ ceph osd create 2024-09-16T05:37:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient: get_monmap_and_config 2024-09-16T05:37:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient: build_initial_monmap 2024-09-16T05:37:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient: monmap: 2024-09-16T05:37:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.684 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:13.682819+0000 2024-09-16T05:37:13.684 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:13.682819+0000 2024-09-16T05:37:13.684 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:13.684 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.684 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:13.684 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:13.684 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:13.684 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 1 Processor -- start 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 1 -- start start 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient: init 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient(hunting): picked mon.noname-a con 0x7fcc4009ae20 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient(hunting): picked mon.noname-c con 0x7fcc4009b910 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient(hunting): picked mon.noname-b con 0x7fcc40139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc400a0ff0 con 0x7fcc4009ae20 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc3e7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcc4009b910 legacy=0x7fcc401388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc4009da90 con 0x7fcc4009b910 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc400a1430 con 0x7fcc40139fc0 2024-09-16T05:37:13.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.679+0000 7fcc452f7640 10 monclient(hunting): authenticate will time out at 1090.459913s 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcc4009ae20 0x7fcc4009b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44538/0 (socket says 172.21.15.63:44538) 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3effd640 1 -- 172.21.15.63:0/52549862 learned_addr learned my addr 172.21.15.63:0/52549862 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 286020882 0 0) 0x7fcc400a0ff0 con 0x7fcc4009ae20 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient(hunting): my global_id is 5034 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc28003530 con 0x7fcc4009ae20 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915546894 0 0) 0x7fcc400a1430 con 0x7fcc40139fc0 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:13.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient(hunting): my global_id is 5021 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc400a0ff0 con 0x7fcc40139fc0 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1387279519 0 0) 0x7fcc28003530 con 0x7fcc4009ae20 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 >> v1:172.21.15.6:6790/0 conn(0x7fcc40139fc0 legacy=0x7fcc4013c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 >> v1:172.21.15.63:6789/0 conn(0x7fcc4009b910 legacy=0x7fcc401388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 monclient: found mon.noname-a 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4009d1f0 con 0x7fcc4009ae20 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:13.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc30001620 con 0x7fcc4009ae20 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 5 monclient: authenticate success, global_id 5034 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: dump: 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.693 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc30001880 con 0x7fcc4009ae20 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 1 -- 172.21.15.63:0/52549862 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc30001c10 con 0x7fcc4009ae20 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 4 set_mon_vals no callback set 2024-09-16T05:37:13.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3dffb640 10 monclient: dump: 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient: shutdown 2024-09-16T05:37:13.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 -- 172.21.15.63:0/52549862 >> v1:172.21.15.6:6789/0 conn(0x7fcc4009ae20 legacy=0x7fcc4009b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 -- 172.21.15.63:0/52549862 shutdown_connections 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 -- 172.21.15.63:0/52549862 >> 172.21.15.63:0/52549862 conn(0x7fcc400a1b60 msgr2=0x7fcc400a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 -- 172.21.15.63:0/52549862 shutdown_connections 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 -- 172.21.15.63:0/52549862 wait complete. 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient: build_initial_monmap 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient: monmap: 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:13.696 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 Processor -- start 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 -- start start 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient: init 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient(hunting): picked mon.noname-b con 0x7fcc4009ae20 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient(hunting): picked mon.noname-c con 0x7fcc4009b910 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 10 monclient(hunting): picked mon.noname-a con 0x7fcc40139fc0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:13.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc40124b30 con 0x7fcc40139fc0 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc3effd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcc4009ae20 legacy=0x7fcc40136210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.683+0000 7fcc452f7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc40124e10 con 0x7fcc4009ae20 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc400a3b60 con 0x7fcc4009b910 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 monclient(hunting): authenticate will time out at 1090.464034s 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc3e7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcc4009b910 0x7fcc401d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:32902/0 (socket says 172.21.15.63:32902) 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc3e7fc640 1 -- 172.21.15.63:0/741718116 learned_addr learned my addr 172.21.15.63:0/741718116 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc3f7fe640 1 --1- 172.21.15.63:0/741718116 >> v1:172.21.15.6:6789/0 conn(0x7fcc40139fc0 0x7fcc401d3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44550/0 (socket says 172.21.15.63:44550) 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1544656940 0 0) 0x7fcc40124b30 con 0x7fcc40139fc0 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:13.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:13.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): my global_id is 5037 2024-09-16T05:37:13.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc2c003530 con 0x7fcc40139fc0 2024-09-16T05:37:13.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220682163 0 0) 0x7fcc400a3b60 con 0x7fcc4009b910 2024-09-16T05:37:13.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:13.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:13.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:13.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): my global_id is 5024 2024-09-16T05:37:13.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc40124b30 con 0x7fcc4009b910 2024-09-16T05:37:13.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 584262641 0 0) 0x7fcc2c003530 con 0x7fcc40139fc0 2024-09-16T05:37:13.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc400a3b60 con 0x7fcc40139fc0 2024-09-16T05:37:13.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc38001c00 con 0x7fcc40139fc0 2024-09-16T05:37:13.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:13.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:13.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): dump: 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4210032182 0 0) 0x7fcc40124b30 con 0x7fcc4009b910 2024-09-16T05:37:13.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc2c003530 con 0x7fcc4009b910 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc34004160 con 0x7fcc4009b910 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient(hunting): dump: 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:13.704 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1035293939 0 0) 0x7fcc400a3b60 con 0x7fcc40139fc0 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 >> v1:172.21.15.6:6790/0 conn(0x7fcc4009b910 legacy=0x7fcc401d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 >> v1:172.21.15.63:6789/0 conn(0x7fcc4009ae20 legacy=0x7fcc40136210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 monclient: found mon.a 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc40124850 con 0x7fcc40139fc0 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:13.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 5 monclient: authenticate success, global_id 5037 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 monclient: _renew_subs 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc401236e0 con 0x7fcc40139fc0 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 monclient: _renew_subs 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc4006ed70 con 0x7fcc40139fc0 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc38003980 con 0x7fcc40139fc0 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc3d7fa640 4 set_mon_vals no callback set 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc38003cd0 con 0x7fcc40139fc0 2024-09-16T05:37:13.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 10 monclient: dump: 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:13.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc20005090 con 0x7fcc40139fc0 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcc38003e70 con 0x7fcc40139fc0 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(68..68 src has 1..68) ==== 4294+0+0 (unknown 841238779 0 0) 0x7fcc38044500 con 0x7fcc40139fc0 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.687+0000 7fcc3effd640 10 monclient: get_auth_request con 0x7fcc2c036a00 auth_method 0 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.703+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fcc38015670 con 0x7fcc40139fc0 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.703+0000 7fcc277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:13.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.703+0000 7fcc277fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:13.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.875+0000 7fcc452f7640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:37:13.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.875+0000 7fcc452f7640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T05:37:13.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.875+0000 7fcc452f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:13.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:13.875+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7fcc20005380 con 0x7fcc40139fc0 2024-09-16T05:37:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:14.439+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v69) ==== 58+0+2 (unknown 2157058348 0 1113811849) 0x7fcc38014f70 con 0x7fcc40139fc0 2024-09-16T05:37:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:14.439+0000 7fcc277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T05:37:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:14.439+0000 7fcc277fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:14.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:14.551+0000 7fcc452f7640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:37:14.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:14.551+0000 7fcc452f7640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T05:37:14.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:14.551+0000 7fcc452f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:14.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:14.551+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7fcc20005ac0 con 0x7fcc40139fc0 2024-09-16T05:37:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.455+0000 7fcc277fe640 1 -- 172.21.15.63:0/741718116 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v70) ==== 58+0+2 (unknown 1524632769 0 1371665406) 0x7fcc380192c0 con 0x7fcc40139fc0 2024-09-16T05:37:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.455+0000 7fcc277fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T05:37:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.455+0000 7fcc277fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.459+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 >> v1:172.21.15.63:6812/3643207211 conn(0x7fcc2c036a00 legacy=0x7fcc2c038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.459+0000 7fcc452f7640 10 monclient: shutdown 2024-09-16T05:37:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.459+0000 7fcc452f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.459+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 >> v1:172.21.15.6:6789/0 conn(0x7fcc40139fc0 legacy=0x7fcc401d3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.459+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 shutdown_connections 2024-09-16T05:37:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.459+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 >> 172.21.15.63:0/741718116 conn(0x7fcc400a1b60 msgr2=0x7fcc400a42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.459+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 shutdown_connections 2024-09-16T05:37:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.459+0000 7fcc452f7640 1 -- 172.21.15.63:0/741718116 wait complete. 2024-09-16T05:37:15.484 INFO:tasks.workunit.client.0.smithi063.stderr:+ o1=7 2024-09-16T05:37:15.484 INFO:tasks.workunit.client.0.smithi063.stderr:++ ceph osd create 2024-09-16T05:37:15.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.599+0000 7f817cdcf640 10 monclient: get_monmap_and_config 2024-09-16T05:37:15.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.599+0000 7f817cdcf640 10 monclient: build_initial_monmap 2024-09-16T05:37:15.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.599+0000 7f817cdcf640 10 monclient: monmap: 2024-09-16T05:37:15.604 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:15.604 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:15.604 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:15.603625+0000 2024-09-16T05:37:15.604 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:15.603625+0000 2024-09-16T05:37:15.605 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:15.605 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:15.605 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:15.605 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:15.605 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:15.605 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 Processor -- start 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- start start 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient: init 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient(hunting): picked mon.noname-a con 0x7f817809ee70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient(hunting): picked mon.noname-b con 0x7f81780a45c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient(hunting): picked mon.noname-c con 0x7f817809f960 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81780a4ee0 con 0x7f817809ee70 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81781413d0 con 0x7f817809f960 2024-09-16T05:37:15.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81781425b0 con 0x7f81780a45c0 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient(hunting): authenticate will time out at 1092.381129s 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8176d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f817809f960 legacy=0x7f817813eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f817809ee70 0x7f81780a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40380/0 (socket says 172.21.15.63:40380) 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f816dd9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f81780a45c0 0x7f817809f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47844/0 (socket says 172.21.15.63:47844) 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817659c640 1 -- 172.21.15.63:0/681109281 learned_addr learned my addr 172.21.15.63:0/681109281 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1960082657 0 0) 0x7f81781425b0 con 0x7f81780a45c0 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient(hunting): my global_id is 5027 2024-09-16T05:37:15.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8164003530 con 0x7f81780a45c0 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176609410 0 0) 0x7f81780a4ee0 con 0x7f817809ee70 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient(hunting): my global_id is 5040 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81781425b0 con 0x7f817809ee70 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2100590481 0 0) 0x7f8164003530 con 0x7f81780a45c0 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 >> v1:172.21.15.63:6789/0 conn(0x7f817809f960 legacy=0x7f817813eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 >> v1:172.21.15.6:6789/0 conn(0x7f817809ee70 legacy=0x7f81780a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 monclient: found mon.noname-b 2024-09-16T05:37:15.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8178143790 con 0x7f81780a45c0 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: _finish_auth 0 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8160001620 con 0x7f81780a45c0 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: dump: 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:15.612 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 5 monclient: authenticate success, global_id 5027 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8160001880 con 0x7f81780a45c0 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 1 -- 172.21.15.63:0/681109281 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8160001bd0 con 0x7f81780a45c0 2024-09-16T05:37:15.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f8175d9b640 10 monclient: dump: 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:15.614 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 4 set_mon_vals no callback set 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 10 monclient: shutdown 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- 172.21.15.63:0/681109281 >> v1:172.21.15.6:6790/0 conn(0x7f81780a45c0 legacy=0x7f817809f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- 172.21.15.63:0/681109281 shutdown_connections 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- 172.21.15.63:0/681109281 >> 172.21.15.63:0/681109281 conn(0x7f81780a5d10 msgr2=0x7f81780a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- 172.21.15.63:0/681109281 shutdown_connections 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.603+0000 7f817cdcf640 1 -- 172.21.15.63:0/681109281 wait complete. 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: build_initial_monmap 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: monmap: 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:15.615 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:15.616 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:15.616 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:15.616 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:15.616 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:15.616 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:15.616 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:15.616 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:15.616 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:15.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 1 Processor -- start 2024-09-16T05:37:15.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 1 -- start start 2024-09-16T05:37:15.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: init 2024-09-16T05:37:15.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:15.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:15.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient(hunting): picked mon.noname-b con 0x7f817809ee70 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:15.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient(hunting): picked mon.noname-a con 0x7f817809f960 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:15.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient(hunting): picked mon.noname-c con 0x7f81780a45c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81781320e0 con 0x7f817809f960 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817812fe40 con 0x7f817809ee70 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8178143180 con 0x7f81780a45c0 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient(hunting): authenticate will time out at 1092.385073s 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f817809ee70 legacy=0x7f817812f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f817809f960 0x7f817812f730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40396/0 (socket says 172.21.15.63:40396) 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816dd9b640 1 -- 172.21.15.63:0/4218575086 learned_addr learned my addr 172.21.15.63:0/4218575086 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:15.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f8176d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f81780a45c0 0x7f81781d3970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47860/0 (socket says 172.21.15.63:47860) 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1681246592 0 0) 0x7f81781320e0 con 0x7f817809f960 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): my global_id is 5043 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8158003530 con 0x7f817809f960 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2334702174 0 0) 0x7f8178143180 con 0x7f81780a45c0 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): my global_id is 5030 2024-09-16T05:37:15.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81781320e0 con 0x7f81780a45c0 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3408544058 0 0) 0x7f81781320e0 con 0x7f81780a45c0 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8178143180 con 0x7f81780a45c0 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1740407356 0 0) 0x7f8158003530 con 0x7f817809f960 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81781320e0 con 0x7f817809f960 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8170004300 con 0x7f81780a45c0 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): dump: 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.623 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f81600047f0 con 0x7f817809f960 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient(hunting): dump: 2024-09-16T05:37:15.624 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:15.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 63402381 0 0) 0x7f8178143180 con 0x7f81780a45c0 2024-09-16T05:37:15.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 >> v1:172.21.15.63:6789/0 conn(0x7f817809ee70 legacy=0x7f817812f020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:15.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 >> v1:172.21.15.6:6789/0 conn(0x7f817809f960 legacy=0x7f817812f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:15.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:15.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 monclient: found mon.c 2024-09-16T05:37:15.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:15.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81780a0500 con 0x7f81780a45c0 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 5 monclient: authenticate success, global_id 5030 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: _renew_subs 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8178131e00 con 0x7f81780a45c0 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: _renew_subs 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f81780726e0 con 0x7f81780a45c0 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8170004560 con 0x7f81780a45c0 2024-09-16T05:37:15.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8170004930 con 0x7f81780a45c0 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f816f7fe640 10 monclient: dump: 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:15.630 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.607+0000 7f817559a640 4 set_mon_vals no callback set 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.611+0000 7f817cdcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.611+0000 7f817cdcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.611+0000 7f817cdcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.611+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f815c005090 con 0x7f81780a45c0 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.611+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8170004b50 con 0x7f81780a45c0 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.611+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(70..70 src has 1..70) ==== 4524+0+0 (unknown 1365113337 0 0) 0x7f8170044690 con 0x7f81780a45c0 2024-09-16T05:37:15.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.611+0000 7f817659c640 10 monclient: get_auth_request con 0x7f8158036a10 auth_method 0 2024-09-16T05:37:15.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.623+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f81700155c0 con 0x7f81780a45c0 2024-09-16T05:37:15.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.623+0000 7f816f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:15.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.623+0000 7f816f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:15.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.799+0000 7f817cdcf640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:37:15.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.799+0000 7f817cdcf640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T05:37:15.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.799+0000 7f817cdcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:15.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.799+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f815c005380 con 0x7f81780a45c0 2024-09-16T05:37:15.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.991+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v71) ==== 58+0+2 (unknown 4019573301 0 3065575683) 0x7f8170014ec0 con 0x7f81780a45c0 2024-09-16T05:37:15.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.991+0000 7f816f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T05:37:15.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:15.991+0000 7f816f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:16.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:16.095+0000 7f817cdcf640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:37:16.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:16.095+0000 7f817cdcf640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T05:37:16.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:16.095+0000 7f817cdcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:16.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:16.095+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f815c005ac0 con 0x7f81780a45c0 2024-09-16T05:37:17.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:16.999+0000 7f816f7fe640 1 -- 172.21.15.63:0/4218575086 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v72) ==== 58+0+2 (unknown 1377504127 0 2769969524) 0x7f817001da70 con 0x7f81780a45c0 2024-09-16T05:37:17.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:16.999+0000 7f816f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T05:37:17.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:16.999+0000 7f816f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:17.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.003+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8158036a10 legacy=0x7f8158038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:17.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.003+0000 7f817cdcf640 10 monclient: shutdown 2024-09-16T05:37:17.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.003+0000 7f817cdcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:17.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.003+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 >> v1:172.21.15.6:6790/0 conn(0x7f81780a45c0 legacy=0x7f81781d3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:17.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.003+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 shutdown_connections 2024-09-16T05:37:17.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.003+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 >> 172.21.15.63:0/4218575086 conn(0x7f81780a5d10 msgr2=0x7f81780a18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:17.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.003+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 shutdown_connections 2024-09-16T05:37:17.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.003+0000 7f817cdcf640 1 -- 172.21.15.63:0/4218575086 wait complete. 2024-09-16T05:37:17.029 INFO:tasks.workunit.client.0.smithi063.stderr:+ o2=9 2024-09-16T05:37:17.030 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-16T05:37:17.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient: get_monmap_and_config 2024-09-16T05:37:17.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient: build_initial_monmap 2024-09-16T05:37:17.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient: monmap: 2024-09-16T05:37:17.157 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:17.156853+0000 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:17.156853+0000 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:17.158 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 1 Processor -- start 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 1 -- start start 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient: init 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient(hunting): picked mon.noname-b con 0x7fa864134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8640a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient(hunting): picked mon.noname-a con 0x7fa86406dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa864135420 con 0x7fa86406dfa0 2024-09-16T05:37:17.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86409e830 con 0x7fa8640a0810 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8640a0dc0 con 0x7fa864134930 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient(hunting): authenticate will time out at 1093.933819s 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa862ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa8640a0810 legacy=0x7fa864139820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8637fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa864134930 0x7fa864134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47868/0 (socket says 172.21.15.63:47868) 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8637fe640 1 -- 172.21.15.63:0/2739927202 learned_addr learned my addr 172.21.15.63:0/2739927202 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa863fff640 1 --1- 172.21.15.63:0/2739927202 >> v1:172.21.15.6:6789/0 conn(0x7fa86406dfa0 0x7fa86413cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40402/0 (socket says 172.21.15.63:40402) 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724691521 0 0) 0x7fa864135420 con 0x7fa86406dfa0 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient(hunting): my global_id is 5046 2024-09-16T05:37:17.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa84c003530 con 0x7fa86406dfa0 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 80786398 0 0) 0x7fa8640a0dc0 con 0x7fa864134930 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient(hunting): my global_id is 5033 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa864135420 con 0x7fa864134930 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3398148347 0 0) 0x7fa864135420 con 0x7fa864134930 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 >> v1:172.21.15.63:6789/0 conn(0x7fa8640a0810 legacy=0x7fa864139820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 >> v1:172.21.15.6:6789/0 conn(0x7fa86406dfa0 legacy=0x7fa86413cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 monclient: found mon.noname-b 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:17.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8640a2500 con 0x7fa864134930 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa854001600 con 0x7fa864134930 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: dump: 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.164 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 5 monclient: authenticate success, global_id 5033 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa854001860 con 0x7fa864134930 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 1 -- 172.21.15.63:0/2739927202 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa854001cb0 con 0x7fa864134930 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa8627fc640 10 monclient: dump: 2024-09-16T05:37:17.165 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:17.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.155+0000 7fa869f8a640 4 set_mon_vals no callback set 2024-09-16T05:37:17.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient: shutdown 2024-09-16T05:37:17.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:17.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- 172.21.15.63:0/2739927202 >> v1:172.21.15.6:6790/0 conn(0x7fa864134930 legacy=0x7fa864134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:17.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- 172.21.15.63:0/2739927202 shutdown_connections 2024-09-16T05:37:17.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- 172.21.15.63:0/2739927202 >> 172.21.15.63:0/2739927202 conn(0x7fa8640a1e90 msgr2=0x7fa864075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- 172.21.15.63:0/2739927202 shutdown_connections 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- 172.21.15.63:0/2739927202 wait complete. 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient: build_initial_monmap 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient: monmap: 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:17.171 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 Processor -- start 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- start start 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient: init 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient(hunting): picked mon.noname-b con 0x7fa86406dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8640a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient(hunting): picked mon.noname-a con 0x7fa864134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86409c0f0 con 0x7fa864134930 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa8637fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa86406dfa0 legacy=0x7fa86413d520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa864136700 con 0x7fa86406dfa0 2024-09-16T05:37:17.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8641c96f0 con 0x7fa8640a0810 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa869f8a640 10 monclient(hunting): authenticate will time out at 1093.938452s 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa862ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa8640a0810 0x7fa8641cdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47884/0 (socket says 172.21.15.63:47884) 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa862ffd640 1 -- 172.21.15.63:0/2557399670 learned_addr learned my addr 172.21.15.63:0/2557399670 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2992379316 0 0) 0x7fa86409c0f0 con 0x7fa864134930 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): my global_id is 5049 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa850003530 con 0x7fa864134930 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435771998 0 0) 0x7fa8641c96f0 con 0x7fa8640a0810 2024-09-16T05:37:17.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): my global_id is 5036 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa86409c0f0 con 0x7fa8640a0810 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 435892366 0 0) 0x7fa850003530 con 0x7fa864134930 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8641c96f0 con 0x7fa864134930 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa85c001dc0 con 0x7fa864134930 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): dump: 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.174 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2003632648 0 0) 0x7fa86409c0f0 con 0x7fa8640a0810 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa850003530 con 0x7fa8640a0810 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa8580040e0 con 0x7fa8640a0810 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:17.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.159+0000 7fa84bfff640 10 monclient(hunting): dump: 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 563677260 0 0) 0x7fa8641c96f0 con 0x7fa864134930 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 >> v1:172.21.15.6:6790/0 conn(0x7fa8640a0810 legacy=0x7fa8641cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:17.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 >> v1:172.21.15.63:6789/0 conn(0x7fa86406dfa0 legacy=0x7fa86413d520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 monclient: found mon.a 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86413dc30 con 0x7fa864134930 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 10 monclient: _finish_auth 0 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 5 monclient: authenticate success, global_id 5049 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 10 monclient: _renew_subs 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa864135810 con 0x7fa864134930 2024-09-16T05:37:17.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 10 monclient: _renew_subs 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8641c8c40 con 0x7fa864134930 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa85c002020 con 0x7fa864134930 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa85c004fc0 con 0x7fa864134930 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa861ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 10 monclient: dump: 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.178 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa844005090 con 0x7fa864134930 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa85c00fd60 con 0x7fa864134930 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(72..72 src has 1..72) ==== 4754+0+0 (unknown 2329724798 0 0) 0x7fa85c044900 con 0x7fa864134930 2024-09-16T05:37:17.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.163+0000 7fa8637fe640 10 monclient: get_auth_request con 0x7fa850036a70 auth_method 0 2024-09-16T05:37:17.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.175+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa85c015740 con 0x7fa864134930 2024-09-16T05:37:17.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.175+0000 7fa84bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:17.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.175+0000 7fa84bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:17.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.343+0000 7fa869f8a640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:17.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.343+0000 7fa869f8a640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:17.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.343+0000 7fa869f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:17.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.343+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) -- 0x7fa844005380 con 0x7fa864134930 2024-09-16T05:37:18.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.999+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map v73) ==== 203+0+0 (unknown 991475101 0 0) 0x7fa85c015040 con 0x7fa864134930 2024-09-16T05:37:18.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.999+0000 7fa84bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:17.999+0000 7fa84bfff640 10 monclient: _finish_command 2 = system:0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map 2024-09-16T05:37:18.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.103+0000 7fa869f8a640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.103+0000 7fa869f8a640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.103+0000 7fa869f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:18.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) -- 0x7fa844005ac0 con 0x7fa864134930 2024-09-16T05:37:18.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa84bfff640 1 -- 172.21.15.63:0/2557399670 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v73) ==== 201+0+0 (unknown 4263494915 0 0) 0x7fa85c01dbf0 con 0x7fa864134930 2024-09-16T05:37:18.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa84bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa84bfff640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T05:37:18.110 INFO:tasks.workunit.client.0.smithi063.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T05:37:18.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa850036a70 legacy=0x7fa850038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa869f8a640 10 monclient: shutdown 2024-09-16T05:37:18.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa869f8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:18.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 >> v1:172.21.15.6:6789/0 conn(0x7fa864134930 legacy=0x7fa8641d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 shutdown_connections 2024-09-16T05:37:18.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.107+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 >> 172.21.15.63:0/2557399670 conn(0x7fa8640a1e90 msgr2=0x7fa86409f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:18.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.111+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 shutdown_connections 2024-09-16T05:37:18.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.111+0000 7fa869f8a640 1 -- 172.21.15.63:0/2557399670 wait complete. 2024-09-16T05:37:18.127 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient: get_monmap_and_config 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient: build_initial_monmap 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient: monmap: 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:18.250513+0000 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:18.250513+0000 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:18.251 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:18.252 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:18.252 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 1 Processor -- start 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 1 -- start start 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient: init 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec4134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec40a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec406dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec4135420 con 0x7f8ec4134930 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec409e830 con 0x7f8ec406dfa0 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec40a0dc0 con 0x7f8ec40a0810 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:18.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8eca178640 10 monclient(hunting): authenticate will time out at 1095.027483s 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8ec3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8ec406dfa0 legacy=0x7f8ec413cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8ec37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ec4134930 0x7f8ec4134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40424/0 (socket says 172.21.15.63:40424) 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.247+0000 7f8ec37fe640 1 -- 172.21.15.63:0/2057329333 learned_addr learned my addr 172.21.15.63:0/2057329333 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2619735458 0 0) 0x7f8ec40a0dc0 con 0x7f8ec40a0810 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient(hunting): my global_id is 5039 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eac003530 con 0x7f8ec40a0810 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3377720799 0 0) 0x7f8ec4135420 con 0x7f8ec4134930 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:18.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient(hunting): my global_id is 5052 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ec40a0dc0 con 0x7f8ec4134930 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 442202728 0 0) 0x7f8eac003530 con 0x7f8ec40a0810 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 >> v1:172.21.15.63:6789/0 conn(0x7f8ec406dfa0 legacy=0x7f8ec413cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 >> v1:172.21.15.6:6789/0 conn(0x7f8ec4134930 legacy=0x7f8ec4134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 monclient: found mon.noname-b 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec40a2500 con 0x7f8ec40a0810 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:18.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8eb8001620 con 0x7f8ec40a0810 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: dump: 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.259 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 5 monclient: authenticate success, global_id 5039 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8eb8001940 con 0x7f8ec40a0810 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 1 -- 172.21.15.63:0/2057329333 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8eb8001d10 con 0x7f8ec40a0810 2024-09-16T05:37:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec27fc640 10 monclient: dump: 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 4 set_mon_vals no callback set 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient: shutdown 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- 172.21.15.63:0/2057329333 >> v1:172.21.15.6:6790/0 conn(0x7f8ec40a0810 legacy=0x7f8ec4139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- 172.21.15.63:0/2057329333 shutdown_connections 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- 172.21.15.63:0/2057329333 >> 172.21.15.63:0/2057329333 conn(0x7f8ec40a1e90 msgr2=0x7f8ec4075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- 172.21.15.63:0/2057329333 shutdown_connections 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- 172.21.15.63:0/2057329333 wait complete. 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient: build_initial_monmap 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient: monmap: 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:18.262 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 Processor -- start 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- start start 2024-09-16T05:37:18.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient: init 2024-09-16T05:37:18.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec406dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec40a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec4134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec41352d0 con 0x7f8ec406dfa0 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec40a46d0 con 0x7f8ec40a0810 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec4072210 con 0x7f8ec4134930 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8ec2ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8ec40a0810 legacy=0x7f8ec41315a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.251+0000 7f8eca178640 10 monclient(hunting): authenticate will time out at 1095.031910s 2024-09-16T05:37:18.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8ec37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ec406dfa0 0x7f8ec4130e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40436/0 (socket says 172.21.15.63:40436) 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8ec3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8ec4134930 0x7f8ec4131cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47900/0 (socket says 172.21.15.63:47900) 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8ec37fe640 1 -- 172.21.15.63:0/1115237070 learned_addr learned my addr 172.21.15.63:0/1115237070 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4274054610 0 0) 0x7f8ec4072210 con 0x7f8ec4134930 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): my global_id is 5042 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb0003530 con 0x7f8ec4134930 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160558457 0 0) 0x7f8ec41352d0 con 0x7f8ec406dfa0 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:18.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): my global_id is 5055 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ec4072210 con 0x7f8ec406dfa0 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2598669821 0 0) 0x7f8eb0003530 con 0x7f8ec4134930 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ec41352d0 con 0x7f8ec4134930 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ebc003f40 con 0x7f8ec4134930 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): dump: 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.270 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 160373975 0 0) 0x7f8ec4072210 con 0x7f8ec406dfa0 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eb0003530 con 0x7f8ec406dfa0 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8eb4001c40 con 0x7f8ec406dfa0 2024-09-16T05:37:18.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient(hunting): dump: 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.272 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 147847230 0 0) 0x7f8ec41352d0 con 0x7f8ec4134930 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 >> v1:172.21.15.63:6789/0 conn(0x7f8ec40a0810 legacy=0x7f8ec41315a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 >> v1:172.21.15.6:6789/0 conn(0x7f8ec406dfa0 legacy=0x7f8ec4130e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 monclient: found mon.c 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec409c0f0 con 0x7f8ec4134930 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient: _finish_auth 0 2024-09-16T05:37:18.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 5 monclient: authenticate success, global_id 5042 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 10 monclient: _renew_subs 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec4134ff0 con 0x7f8ec4134930 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 10 monclient: _renew_subs 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ec41d27d0 con 0x7f8ec4134930 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ebc004450 con 0x7f8ec4134930 2024-09-16T05:37:18.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:18.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ebc0047a0 con 0x7f8ec4134930 2024-09-16T05:37:18.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:18.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:18.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 10 monclient: dump: 2024-09-16T05:37:18.275 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.275 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.275 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8ec1ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:18.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:18.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:18.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea4005090 con 0x7f8ec4134930 2024-09-16T05:37:18.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8ebc004940 con 0x7f8ec4134930 2024-09-16T05:37:18.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 5036+0+0 (unknown 1264245377 0 0) 0x7f8ebc044430 con 0x7f8ec4134930 2024-09-16T05:37:18.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.255+0000 7f8ec37fe640 10 monclient: get_auth_request con 0x7f8eb0036a80 auth_method 0 2024-09-16T05:37:18.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.267+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8ebc015200 con 0x7f8ec4134930 2024-09-16T05:37:18.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.267+0000 7f8eabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:18.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.267+0000 7f8eabfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:18.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.431+0000 7f8eca178640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.431+0000 7f8eca178640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.431+0000 7f8eca178640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.431+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) -- 0x7f8ea4005380 con 0x7f8ec4134930 2024-09-16T05:37:18.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.431+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v73) ==== 201+0+0 (unknown 4263494915 0 0) 0x7f8ebc014b00 con 0x7f8ec4134930 2024-09-16T05:37:18.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.431+0000 7f8eabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.431+0000 7f8eabfff640 10 monclient: _finish_command 2 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T05:37:18.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.531+0000 7f8eca178640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.531+0000 7f8eca178640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.531+0000 7f8eca178640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.531+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) -- 0x7f8ea4005be0 con 0x7f8ec4134930 2024-09-16T05:37:18.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eabfff640 1 -- 172.21.15.63:0/1115237070 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v73) ==== 201+0+0 (unknown 4263494915 0 0) 0x7f8ebc018e50 con 0x7f8ec4134930 2024-09-16T05:37:18.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eabfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T05:37:18.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eabfff640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T05:37:18.538 INFO:tasks.workunit.client.0.smithi063.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T05:37:18.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8eb0036a80 legacy=0x7f8eb0038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eca178640 10 monclient: shutdown 2024-09-16T05:37:18.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eca178640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:18.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 >> v1:172.21.15.6:6790/0 conn(0x7f8ec4134930 legacy=0x7f8ec4131cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 shutdown_connections 2024-09-16T05:37:18.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 >> 172.21.15.63:0/1115237070 conn(0x7f8ec40a1e90 msgr2=0x7f8ec40a2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:18.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 shutdown_connections 2024-09-16T05:37:18.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.535+0000 7f8eca178640 1 -- 172.21.15.63:0/1115237070 wait complete. 2024-09-16T05:37:18.556 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-16T05:37:18.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient: get_monmap_and_config 2024-09-16T05:37:18.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient: build_initial_monmap 2024-09-16T05:37:18.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient: monmap: 2024-09-16T05:37:18.682 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:18.682049+0000 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:18.682049+0000 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:18.683 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 1 Processor -- start 2024-09-16T05:37:18.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 1 -- start start 2024-09-16T05:37:18.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient: init 2024-09-16T05:37:18.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f80a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f80a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f80a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f80a53b0 con 0x7fa1f80a4600 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f80a0290 con 0x7fa1f80a3b10 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f80a0dc0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fe382640 10 monclient(hunting): authenticate will time out at 1095.459084s 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1f7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa1f80a3b10 legacy=0x7fa1f80a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:18.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1f77fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa1f80a07a0 0x7fa1f813be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47904/0 (socket says 172.21.15.63:47904) 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1f77fe640 1 -- 172.21.15.63:0/4221227477 learned_addr learned my addr 172.21.15.63:0/4221227477 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.679+0000 7fa1fc91f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa1f80a4600 0x7fa1f813f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40450/0 (socket says 172.21.15.63:40450) 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205956526 0 0) 0x7fa1f80a0dc0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient(hunting): my global_id is 5045 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1e0003530 con 0x7fa1f80a07a0 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264052063 0 0) 0x7fa1f80a53b0 con 0x7fa1f80a4600 2024-09-16T05:37:18.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient(hunting): my global_id is 5058 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1f80a0dc0 con 0x7fa1f80a4600 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3493166340 0 0) 0x7fa1e0003530 con 0x7fa1f80a07a0 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 >> v1:172.21.15.63:6789/0 conn(0x7fa1f80a3b10 legacy=0x7fa1f80a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 >> v1:172.21.15.6:6789/0 conn(0x7fa1f80a4600 legacy=0x7fa1f813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 monclient: found mon.noname-b 2024-09-16T05:37:18.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f80a10a0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa1ec001620 con 0x7fa1f80a07a0 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 5 monclient: authenticate success, global_id 5045 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: dump: 2024-09-16T05:37:18.690 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa1ec001880 con 0x7fa1f80a07a0 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:18.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 1 -- 172.21.15.63:0/4221227477 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa1ec001bd0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 4 set_mon_vals no callback set 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f6ffd640 10 monclient: dump: 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.692 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.693 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient: shutdown 2024-09-16T05:37:18.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:18.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- 172.21.15.63:0/4221227477 >> v1:172.21.15.6:6790/0 conn(0x7fa1f80a07a0 legacy=0x7fa1f813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- 172.21.15.63:0/4221227477 shutdown_connections 2024-09-16T05:37:18.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- 172.21.15.63:0/4221227477 >> 172.21.15.63:0/4221227477 conn(0x7fa1f8075470 msgr2=0x7fa1f8075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:18.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- 172.21.15.63:0/4221227477 shutdown_connections 2024-09-16T05:37:18.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- 172.21.15.63:0/4221227477 wait complete. 2024-09-16T05:37:18.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient: build_initial_monmap 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient: monmap: 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 Processor -- start 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- start start 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient: init 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f80a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f80a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f80a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f81236e0 con 0x7fa1f80a3b10 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f8121f50 con 0x7fa1f80a4600 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f80a87c0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fe382640 10 monclient(hunting): authenticate will time out at 1095.462920s 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1fc91f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa1f80a4600 legacy=0x7fa1f81d3b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:18.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa1f80a3b10 0x7fa1f81d0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40452/0 (socket says 172.21.15.63:40452) 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f77fe640 1 -- 172.21.15.63:0/3232937147 learned_addr learned my addr 172.21.15.63:0/3232937147 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa1f80a07a0 0x7fa1f8135d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47906/0 (socket says 172.21.15.63:47906) 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850690268 0 0) 0x7fa1f80a87c0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 10 monclient(hunting): my global_id is 5048 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1e4003530 con 0x7fa1f80a07a0 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1112360715 0 0) 0x7fa1f81236e0 con 0x7fa1f80a3b10 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:18.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 10 monclient(hunting): my global_id is 5061 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.683+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1f80a87c0 con 0x7fa1f80a3b10 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2823221787 0 0) 0x7fa1e4003530 con 0x7fa1f80a07a0 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1f81236e0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa1e8004000 con 0x7fa1f80a07a0 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.701 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2791009831 0 0) 0x7fa1f80a87c0 con 0x7fa1f80a3b10 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1e4003530 con 0x7fa1f80a3b10 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa1ec001560 con 0x7fa1f80a3b10 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:18.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1987780450 0 0) 0x7fa1f81236e0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 >> v1:172.21.15.63:6789/0 conn(0x7fa1f80a4600 legacy=0x7fa1f81d3b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 >> v1:172.21.15.6:6789/0 conn(0x7fa1f80a3b10 legacy=0x7fa1f81d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:18.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 monclient: found mon.c 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f8123400 con 0x7fa1f80a07a0 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 5 monclient: authenticate success, global_id 5048 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 10 monclient: _renew_subs 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f8121cd0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 10 monclient: _renew_subs 2024-09-16T05:37:18.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa1f8136600 con 0x7fa1f80a07a0 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa1e8004260 con 0x7fa1f80a07a0 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa1e80045b0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f67fc640 4 set_mon_vals no callback set 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 10 monclient: dump: 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:18.705 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d8005090 con 0x7fa1f80a07a0 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa1e8004750 con 0x7fa1f80a07a0 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 5036+0+0 (unknown 1264245377 0 0) 0x7fa1e8044460 con 0x7fa1f80a07a0 2024-09-16T05:37:18.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.687+0000 7fa1f77fe640 10 monclient: get_auth_request con 0x7fa1e4036a70 auth_method 0 2024-09-16T05:37:18.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.699+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa1e80150d0 con 0x7fa1f80a07a0 2024-09-16T05:37:18.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.699+0000 7fa1f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:18.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.699+0000 7fa1f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:18.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.863+0000 7fa1fe382640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:18.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.863+0000 7fa1fe382640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:18.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.863+0000 7fa1fe382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:18.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:18.863+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) -- 0x7fa1d8005380 con 0x7fa1f80a07a0 2024-09-16T05:37:19.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.007+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map v74) ==== 203+0+0 (unknown 823284084 0 0) 0x7fa1e8018d80 con 0x7fa1f80a07a0 2024-09-16T05:37:19.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.007+0000 7fa1f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.007+0000 7fa1f4ff9640 10 monclient: _finish_command 2 = system:0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map 2024-09-16T05:37:19.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.107+0000 7fa1fe382640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.107+0000 7fa1fe382640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.107+0000 7fa1fe382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:19.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.107+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) -- 0x7fa1d8005ac0 con 0x7fa1f80a07a0 2024-09-16T05:37:19.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.107+0000 7fa1f4ff9640 1 -- 172.21.15.63:0/3232937147 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v74) ==== 201+0+0 (unknown 1937127817 0 0) 0x7fa1e801d640 con 0x7fa1f80a07a0 2024-09-16T05:37:19.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.107+0000 7fa1f4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.107+0000 7fa1f4ff9640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T05:37:19.112 INFO:tasks.workunit.client.0.smithi063.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T05:37:19.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.111+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa1e4036a70 legacy=0x7fa1e4038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.111+0000 7fa1fe382640 10 monclient: shutdown 2024-09-16T05:37:19.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.111+0000 7fa1fe382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:19.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.111+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 >> v1:172.21.15.6:6790/0 conn(0x7fa1f80a07a0 legacy=0x7fa1f8135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.111+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 shutdown_connections 2024-09-16T05:37:19.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.111+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 >> 172.21.15.63:0/3232937147 conn(0x7fa1f8075470 msgr2=0x7fa1f81beb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:19.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.111+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 shutdown_connections 2024-09-16T05:37:19.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.111+0000 7fa1fe382640 1 -- 172.21.15.63:0/3232937147 wait complete. 2024-09-16T05:37:19.128 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-16T05:37:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient: get_monmap_and_config 2024-09-16T05:37:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient: build_initial_monmap 2024-09-16T05:37:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient: monmap: 2024-09-16T05:37:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:19.250857+0000 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:19.250857+0000 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:19.252 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 1 Processor -- start 2024-09-16T05:37:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 1 -- start start 2024-09-16T05:37:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient: init 2024-09-16T05:37:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient(hunting): picked mon.noname-a con 0x7f47d0134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient(hunting): picked mon.noname-c con 0x7f47d00a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient(hunting): picked mon.noname-b con 0x7f47d006dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d0135420 con 0x7f47d0134930 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d009e830 con 0x7f47d00a0810 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d00a0dc0 con 0x7f47d006dfa0 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d526f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f47d00a0810 legacy=0x7f47d0139820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.247+0000 7f47d7cd4640 10 monclient(hunting): authenticate will time out at 1096.027970s 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d6271640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f47d006dfa0 0x7f47d013cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47918/0 (socket says 172.21.15.63:47918) 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d6271640 1 -- 172.21.15.63:0/2135219055 learned_addr learned my addr 172.21.15.63:0/2135219055 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d5a70640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f47d0134930 0x7f47d0134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40454/0 (socket says 172.21.15.63:40454) 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395873328 0 0) 0x7f47d0135420 con 0x7f47d0134930 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient(hunting): _negotiate 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient(hunting): my global_id is 5064 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47bc003530 con 0x7f47d0134930 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1820144108 0 0) 0x7f47d00a0dc0 con 0x7f47d006dfa0 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient(hunting): _negotiate 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient(hunting): my global_id is 5051 2024-09-16T05:37:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47d0135420 con 0x7f47d006dfa0 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1542126884 0 0) 0x7f47bc003530 con 0x7f47d0134930 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 >> v1:172.21.15.6:6790/0 conn(0x7f47d006dfa0 legacy=0x7f47d013cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 >> v1:172.21.15.63:6789/0 conn(0x7f47d00a0810 legacy=0x7f47d0139820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: _finish_hunting 0 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 monclient: found mon.noname-a 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d00a2500 con 0x7f47d0134930 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: _finish_auth 0 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: _check_auth_tickets 2024-09-16T05:37:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f47c0001620 con 0x7f47d0134930 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: dump: 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.259 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 5 monclient: authenticate success, global_id 5064 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47c0001880 con 0x7f47d0134930 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 1 -- 172.21.15.63:0/2135219055 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f47c0001c50 con 0x7f47d0134930 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 4 set_mon_vals no callback set 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d4a6e640 10 monclient: dump: 2024-09-16T05:37:19.260 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 10 monclient: shutdown 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 1 -- 172.21.15.63:0/2135219055 >> v1:172.21.15.6:6789/0 conn(0x7f47d0134930 legacy=0x7f47d0134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 1 -- 172.21.15.63:0/2135219055 shutdown_connections 2024-09-16T05:37:19.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 1 -- 172.21.15.63:0/2135219055 >> 172.21.15.63:0/2135219055 conn(0x7f47d00a1e90 msgr2=0x7f47d0075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:19.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 1 -- 172.21.15.63:0/2135219055 shutdown_connections 2024-09-16T05:37:19.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 1 -- 172.21.15.63:0/2135219055 wait complete. 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 10 monclient: build_initial_monmap 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 10 monclient: monmap: 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:19.266 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 1 Processor -- start 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 1 -- start start 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.251+0000 7f47d7cd4640 10 monclient: init 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient(hunting): picked mon.noname-b con 0x7f47d006dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient(hunting): picked mon.noname-c con 0x7f47d00a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient(hunting): picked mon.noname-a con 0x7f47d0134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d0136980 con 0x7f47d0134930 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d5a70640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f47d006dfa0 legacy=0x7f47d01c9770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:19.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d0136c60 con 0x7f47d006dfa0 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d01ca590 con 0x7f47d00a0810 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient(hunting): authenticate will time out at 1096.032226s 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d6271640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f47d0134930 0x7f47d01d19d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40462/0 (socket says 172.21.15.63:40462) 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d6271640 1 -- 172.21.15.63:0/3219161164 learned_addr learned my addr 172.21.15.63:0/3219161164 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3429595547 0 0) 0x7f47d0136980 con 0x7f47d0134930 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): my global_id is 5067 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47b4003530 con 0x7f47d0134930 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225231203 0 0) 0x7f47d01ca590 con 0x7f47d00a0810 2024-09-16T05:37:19.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): my global_id is 5054 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47d0136980 con 0x7f47d00a0810 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 792253339 0 0) 0x7f47b4003530 con 0x7f47d0134930 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47d01ca590 con 0x7f47d0134930 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f47c8001c40 con 0x7f47d0134930 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): dump: 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.269 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1492425404 0 0) 0x7f47d0136980 con 0x7f47d00a0810 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47b4003530 con 0x7f47d00a0810 2024-09-16T05:37:19.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f47cc0040d0 con 0x7f47d00a0810 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient(hunting): dump: 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1629136170 0 0) 0x7f47d01ca590 con 0x7f47d0134930 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 >> v1:172.21.15.6:6790/0 conn(0x7f47d00a0810 legacy=0x7f47d01c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 >> v1:172.21.15.63:6789/0 conn(0x7f47d006dfa0 legacy=0x7f47d01c9770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 monclient: found mon.a 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d01355f0 con 0x7f47d0134930 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:19.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 5 monclient: authenticate success, global_id 5067 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient: _renew_subs 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47d00a3f80 con 0x7f47d0134930 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient: _renew_subs 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f47d0133960 con 0x7f47d0134930 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47c8004b70 con 0x7f47d0134930 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f47c8004ec0 con 0x7f47d0134930 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:19.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c7fff640 4 set_mon_vals no callback set 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 10 monclient: dump: 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b8005090 con 0x7f47d0134930 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f47c800fc60 con 0x7f47d0134930 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(74..74 src has 1..74) ==== 5280+0+0 (unknown 1273044008 0 0) 0x7f47c8044b90 con 0x7f47d0134930 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.255+0000 7f47d5a70640 10 monclient: get_auth_request con 0x7f47b4036a50 auth_method 0 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.271+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f47c80157d0 con 0x7f47d0134930 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.271+0000 7f47c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.271+0000 7f47c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:19.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.443+0000 7f47d7cd4640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.443+0000 7f47d7cd4640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.443+0000 7f47d7cd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.443+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) -- 0x7f47b8005380 con 0x7f47d0134930 2024-09-16T05:37:19.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.443+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v74) ==== 201+0+0 (unknown 1937127817 0 0) 0x7f47c80150d0 con 0x7f47d0134930 2024-09-16T05:37:19.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.443+0000 7f47c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.443+0000 7f47c67fc640 10 monclient: _finish_command 2 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T05:37:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.543+0000 7f47d7cd4640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.543+0000 7f47d7cd4640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.543+0000 7f47d7cd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.543+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) -- 0x7f47b8005ac0 con 0x7f47d0134930 2024-09-16T05:37:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.543+0000 7f47c67fc640 1 -- 172.21.15.63:0/3219161164 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v74) ==== 201+0+0 (unknown 1937127817 0 0) 0x7f47c801dc80 con 0x7f47d0134930 2024-09-16T05:37:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.543+0000 7f47c67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T05:37:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.543+0000 7f47c67fc640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T05:37:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T05:37:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.547+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 >> v1:172.21.15.63:6812/3643207211 conn(0x7f47b4036a50 legacy=0x7f47b4038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.547+0000 7f47d7cd4640 10 monclient: shutdown 2024-09-16T05:37:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.547+0000 7f47d7cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.547+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 >> v1:172.21.15.6:6789/0 conn(0x7f47d0134930 legacy=0x7f47d01d19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.547+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 shutdown_connections 2024-09-16T05:37:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.547+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 >> 172.21.15.63:0/3219161164 conn(0x7f47d00a1e90 msgr2=0x7f47d012d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.547+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 shutdown_connections 2024-09-16T05:37:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.547+0000 7f47d7cd4640 1 -- 172.21.15.63:0/3219161164 wait complete. 2024-09-16T05:37:19.566 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket bar root 2024-09-16T05:37:19.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.683+0000 7f8bcdb0c640 10 monclient: get_monmap_and_config 2024-09-16T05:37:19.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.683+0000 7f8bcdb0c640 10 monclient: build_initial_monmap 2024-09-16T05:37:19.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.683+0000 7f8bcdb0c640 10 monclient: monmap: 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:19.687885+0000 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:19.687885+0000 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:19.689 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 Processor -- start 2024-09-16T05:37:19.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- start start 2024-09-16T05:37:19.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient: init 2024-09-16T05:37:19.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:19.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:19.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient(hunting): picked mon.noname-c con 0x7f8bc809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:19.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient(hunting): picked mon.noname-a con 0x7f8bc80a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient(hunting): picked mon.noname-b con 0x7f8bc809b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bc8074200 con 0x7f8bc80a0280 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bc809f900 con 0x7f8bc809ac90 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bc809e870 con 0x7f8bc809b780 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient(hunting): authenticate will time out at 1096.464817s 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc77fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8bc809ac90 legacy=0x7f8bc809b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc6ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8bc80a0280 0x7f8bc81398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40468/0 (socket says 172.21.15.63:40468) 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc6ffd640 1 -- 172.21.15.63:0/2650978599 learned_addr learned my addr 172.21.15.63:0/2650978599 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc7fff640 1 --1- 172.21.15.63:0/2650978599 >> v1:172.21.15.6:6790/0 conn(0x7f8bc809b780 0x7f8bc813cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47924/0 (socket says 172.21.15.63:47924) 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3116150380 0 0) 0x7f8bc8074200 con 0x7f8bc80a0280 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:19.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient(hunting): my global_id is 5070 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bb0003530 con 0x7f8bc80a0280 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884868854 0 0) 0x7f8bc809e870 con 0x7f8bc809b780 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient(hunting): my global_id is 5057 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bc8074200 con 0x7f8bc809b780 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3007685440 0 0) 0x7f8bb0003530 con 0x7f8bc80a0280 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 >> v1:172.21.15.6:6790/0 conn(0x7f8bc809b780 legacy=0x7f8bc813cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 >> v1:172.21.15.63:6789/0 conn(0x7f8bc809ac90 legacy=0x7f8bc809b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:19.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 monclient: found mon.noname-a 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc8141730 con 0x7f8bc80a0280 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8bbc001620 con 0x7f8bc80a0280 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: dump: 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.695 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 5 monclient: authenticate success, global_id 5070 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bbc001880 con 0x7f8bc80a0280 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 1 -- 172.21.15.63:0/2650978599 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8bbc001bd0 con 0x7f8bc80a0280 2024-09-16T05:37:19.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bc67fc640 10 monclient: dump: 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:19.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 4 set_mon_vals no callback set 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient: shutdown 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/2650978599 >> v1:172.21.15.6:6789/0 conn(0x7f8bc80a0280 legacy=0x7f8bc81398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/2650978599 shutdown_connections 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/2650978599 >> 172.21.15.63:0/2650978599 conn(0x7f8bc80a5290 msgr2=0x7f8bc80a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/2650978599 shutdown_connections 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/2650978599 wait complete. 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient: build_initial_monmap 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 10 monclient: monmap: 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:19.698 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:19.699 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:19.699 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.699 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:19.699 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:19.699 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:19.699 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.687+0000 7f8bcdb0c640 1 Processor -- start 2024-09-16T05:37:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 1 -- start start 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: init 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient(hunting): picked mon.noname-a con 0x7f8bc809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient(hunting): picked mon.noname-c con 0x7f8bc809b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient(hunting): picked mon.noname-b con 0x7f8bc80a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bc809d250 con 0x7f8bc809ac90 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bc809e0d0 con 0x7f8bc80a0280 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bc8140c40 con 0x7f8bc809b780 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient(hunting): authenticate will time out at 1096.468566s 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bc7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8bc80a0280 legacy=0x7f8bc81d36b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bc6ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8bc809b780 0x7f8bc809cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47930/0 (socket says 172.21.15.63:47930) 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bc77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8bc809ac90 0x7f8bc809c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40482/0 (socket says 172.21.15.63:40482) 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bc6ffd640 1 -- 172.21.15.63:0/1648350768 learned_addr learned my addr 172.21.15.63:0/1648350768 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035985129 0 0) 0x7f8bc809d250 con 0x7f8bc809ac90 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): my global_id is 5073 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bb4003530 con 0x7f8bc809ac90 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294171889 0 0) 0x7f8bc8140c40 con 0x7f8bc809b780 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): my global_id is 5060 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bc809d250 con 0x7f8bc809b780 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2036695374 0 0) 0x7f8bb4003530 con 0x7f8bc809ac90 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bc8140c40 con 0x7f8bc809ac90 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8bb80040d0 con 0x7f8bc809ac90 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): dump: 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 652831887 0 0) 0x7f8bc809d250 con 0x7f8bc809b780 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bb4003530 con 0x7f8bc809b780 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8bbc0047f0 con 0x7f8bc809b780 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient(hunting): dump: 2024-09-16T05:37:19.707 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3660568812 0 0) 0x7f8bc8140c40 con 0x7f8bc809ac90 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 >> v1:172.21.15.6:6790/0 conn(0x7f8bc809b780 legacy=0x7f8bc809cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 >> v1:172.21.15.63:6789/0 conn(0x7f8bc80a0280 legacy=0x7f8bc81d36b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:19.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 monclient: found mon.a 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc813f7c0 con 0x7f8bc809ac90 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient: _finish_auth 0 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 5 monclient: authenticate success, global_id 5073 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: _renew_subs 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc813f4b0 con 0x7f8bc809ac90 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: _renew_subs 2024-09-16T05:37:19.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8bc81d4dd0 con 0x7f8bc809ac90 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bb8004330 con 0x7f8bc809ac90 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8bb8004680 con 0x7f8bc809ac90 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 10 monclient: dump: 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:19.710 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:19.711 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bc57fa640 4 set_mon_vals no callback set 2024-09-16T05:37:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba8005090 con 0x7f8bc809ac90 2024-09-16T05:37:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8bb80048a0 con 0x7f8bc809ac90 2024-09-16T05:37:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(74..74 src has 1..74) ==== 5280+0+0 (unknown 1273044008 0 0) 0x7f8bb8044710 con 0x7f8bc809ac90 2024-09-16T05:37:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.691+0000 7f8bc6ffd640 10 monclient: get_auth_request con 0x7f8bb4036a50 auth_method 0 2024-09-16T05:37:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.703+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8bb80152f0 con 0x7f8bc809ac90 2024-09-16T05:37:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.703+0000 7f8baffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.703+0000 7f8baffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:19.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.871+0000 7f8bcdb0c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:19.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.871+0000 7f8bcdb0c640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:19.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.871+0000 7f8bcdb0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:19.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:19.871+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7f8ba8005b90 con 0x7f8bc809ac90 2024-09-16T05:37:20.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.019+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 added bucket bar type root to crush map v75) ==== 138+0+0 (unknown 230693895 0 0) 0x7f8bb8014bf0 con 0x7f8bc809ac90 2024-09-16T05:37:20.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.019+0000 7f8baffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.019+0000 7f8baffff640 10 monclient: _finish_command 2 = system:0 added bucket bar type root to crush map 2024-09-16T05:37:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.119+0000 7f8bcdb0c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.119+0000 7f8bcdb0c640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.119+0000 7f8bcdb0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.119+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7f8ba8005610 con 0x7f8bc809ac90 2024-09-16T05:37:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.119+0000 7f8baffff640 1 -- 172.21.15.63:0/1648350768 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v75) ==== 126+0+0 (unknown 3183435221 0 0) 0x7f8bb8018f40 con 0x7f8bc809ac90 2024-09-16T05:37:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.119+0000 7f8baffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.119+0000 7f8baffff640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-16T05:37:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'bar' already exists 2024-09-16T05:37:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.123+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8bb4036a50 legacy=0x7f8bb4038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.123+0000 7f8bcdb0c640 10 monclient: shutdown 2024-09-16T05:37:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.123+0000 7f8bcdb0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.123+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 >> v1:172.21.15.6:6789/0 conn(0x7f8bc809ac90 legacy=0x7f8bc809c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.123+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 shutdown_connections 2024-09-16T05:37:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.123+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 >> 172.21.15.63:0/1648350768 conn(0x7f8bc80a5290 msgr2=0x7f8bc81348a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.123+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 shutdown_connections 2024-09-16T05:37:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.123+0000 7f8bcdb0c640 1 -- 172.21.15.63:0/1648350768 wait complete. 2024-09-16T05:37:20.141 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket bar root 2024-09-16T05:37:20.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.259+0000 7f52eb950640 10 monclient: get_monmap_and_config 2024-09-16T05:37:20.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.259+0000 7f52eb950640 10 monclient: build_initial_monmap 2024-09-16T05:37:20.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.259+0000 7f52eb950640 10 monclient: monmap: 2024-09-16T05:37:20.265 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:20.265 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.265 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:20.264147+0000 2024-09-16T05:37:20.265 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:20.264147+0000 2024-09-16T05:37:20.265 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:20.266 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.266 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:20.266 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:20.266 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:20.266 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 Processor -- start 2024-09-16T05:37:20.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- start start 2024-09-16T05:37:20.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient: init 2024-09-16T05:37:20.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:20.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:20.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient(hunting): picked mon.noname-b con 0x7f52e409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:20.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient(hunting): picked mon.noname-c con 0x7f52e409b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:20.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient(hunting): picked mon.noname-a con 0x7f52e4139fc0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e40a14f0 con 0x7f52e4139fc0 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e409c680 con 0x7f52e409b780 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e409e930 con 0x7f52e409ac90 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient(hunting): authenticate will time out at 1097.041207s 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52e8eeb640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f52e409b780 legacy=0x7f52e41388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52e96ec640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f52e409ac90 0x7f52e409b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47946/0 (socket says 172.21.15.63:47946) 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52e96ec640 1 -- 172.21.15.63:0/2004431055 learned_addr learned my addr 172.21.15.63:0/2004431055 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52e9eed640 1 --1- 172.21.15.63:0/2004431055 >> v1:172.21.15.6:6789/0 conn(0x7f52e4139fc0 0x7f52e413c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40494/0 (socket says 172.21.15.63:40494) 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 489607026 0 0) 0x7f52e409e930 con 0x7f52e409ac90 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:20.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient(hunting): my global_id is 5063 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52d0005740 con 0x7f52e409ac90 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013213051 0 0) 0x7f52e40a14f0 con 0x7f52e4139fc0 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient(hunting): my global_id is 5076 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52e409e930 con 0x7f52e4139fc0 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3233924859 0 0) 0x7f52d0005740 con 0x7f52e409ac90 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 >> v1:172.21.15.63:6789/0 conn(0x7f52e409b780 legacy=0x7f52e41388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 >> v1:172.21.15.6:6789/0 conn(0x7f52e4139fc0 legacy=0x7f52e413c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:20.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 monclient: found mon.noname-b 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e409c400 con 0x7f52e409ac90 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: _finish_auth 0 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52d4001550 con 0x7f52e409ac90 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: dump: 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.278 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 5 monclient: authenticate success, global_id 5063 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52d4001800 con 0x7f52e409ac90 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:20.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 1 -- 172.21.15.63:0/2004431055 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52d4001bd0 con 0x7f52e409ac90 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 4 set_mon_vals no callback set 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52dbfff640 10 monclient: dump: 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.280 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 10 monclient: shutdown 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- 172.21.15.63:0/2004431055 >> v1:172.21.15.6:6790/0 conn(0x7f52e409ac90 legacy=0x7f52e409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- 172.21.15.63:0/2004431055 shutdown_connections 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- 172.21.15.63:0/2004431055 >> 172.21.15.63:0/2004431055 conn(0x7f52e40a5290 msgr2=0x7f52e40a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- 172.21.15.63:0/2004431055 shutdown_connections 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.263+0000 7f52eb950640 1 -- 172.21.15.63:0/2004431055 wait complete. 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: build_initial_monmap 2024-09-16T05:37:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: monmap: 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 1 Processor -- start 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 1 -- start start 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: init 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient(hunting): picked mon.noname-c con 0x7f52e409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient(hunting): picked mon.noname-b con 0x7f52e409b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient(hunting): picked mon.noname-a con 0x7f52e4139fc0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e4121c00 con 0x7f52e4139fc0 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e40a0720 con 0x7f52e409b780 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e4121920 con 0x7f52e409ac90 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient(hunting): authenticate will time out at 1097.044957s 2024-09-16T05:37:20.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52e8eeb640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f52e409b780 legacy=0x7f52e41d0630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52e96ec640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f52e409ac90 0x7f52e4133290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47954/0 (socket says 172.21.15.63:47954) 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52e9eed640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f52e4139fc0 0x7f52e41d3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40500/0 (socket says 172.21.15.63:40500) 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52e96ec640 1 -- 172.21.15.63:0/1186550994 learned_addr learned my addr 172.21.15.63:0/1186550994 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 39975577 0 0) 0x7f52e4121c00 con 0x7f52e4139fc0 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): my global_id is 5079 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52c8003530 con 0x7f52e4139fc0 2024-09-16T05:37:20.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3840916609 0 0) 0x7f52e4121920 con 0x7f52e409ac90 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): my global_id is 5066 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52e4121c00 con 0x7f52e409ac90 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3643769145 0 0) 0x7f52e4121c00 con 0x7f52e409ac90 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52e4121920 con 0x7f52e409ac90 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52d4001550 con 0x7f52e409ac90 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:20.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.286 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4085655977 0 0) 0x7f52c8003530 con 0x7f52e4139fc0 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52e4121c00 con 0x7f52e4139fc0 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52dc001bc0 con 0x7f52e4139fc0 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.287 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2989267828 0 0) 0x7f52e4121920 con 0x7f52e409ac90 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 >> v1:172.21.15.63:6789/0 conn(0x7f52e409b780 legacy=0x7f52e41d0630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:20.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 >> v1:172.21.15.6:6789/0 conn(0x7f52e4139fc0 legacy=0x7f52e41d3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 monclient: found mon.c 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e41204d0 con 0x7f52e409ac90 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 5 monclient: authenticate success, global_id 5066 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: _renew_subs 2024-09-16T05:37:20.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52e4121640 con 0x7f52e409ac90 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: _renew_subs 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f52e411f180 con 0x7f52e409ac90 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52d4001800 con 0x7f52e409ac90 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52d4004cd0 con 0x7f52e409ac90 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52db7fe640 4 set_mon_vals no callback set 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52d9ffb640 10 monclient: dump: 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.290 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.267+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c4005090 con 0x7f52e409ac90 2024-09-16T05:37:20.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.271+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f52d4004ef0 con 0x7f52e409ac90 2024-09-16T05:37:20.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.271+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(75..75 src has 1..75) ==== 5307+0+0 (unknown 4254335399 0 0) 0x7f52d40448e0 con 0x7f52e409ac90 2024-09-16T05:37:20.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.271+0000 7f52e8eeb640 10 monclient: get_auth_request con 0x7f52c8036a20 auth_method 0 2024-09-16T05:37:20.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.283+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f52d4019150 con 0x7f52e409ac90 2024-09-16T05:37:20.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.283+0000 7f52d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:20.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.283+0000 7f52d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:20.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.443+0000 7f52eb950640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.443+0000 7f52eb950640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.443+0000 7f52eb950640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.443+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7f52c4005b90 con 0x7f52e409ac90 2024-09-16T05:37:20.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.443+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v75) ==== 126+0+0 (unknown 3183435221 0 0) 0x7f52d401da10 con 0x7f52e409ac90 2024-09-16T05:37:20.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.443+0000 7f52d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.443+0000 7f52d9ffb640 10 monclient: _finish_command 2 = system:0 bucket 'bar' already exists 2024-09-16T05:37:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.543+0000 7f52eb950640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.543+0000 7f52eb950640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.543+0000 7f52eb950640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.543+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7f52c4005610 con 0x7f52e409ac90 2024-09-16T05:37:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.543+0000 7f52d9ffb640 1 -- 172.21.15.63:0/1186550994 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v75) ==== 126+0+0 (unknown 3183435221 0 0) 0x7f52d4040590 con 0x7f52e409ac90 2024-09-16T05:37:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.543+0000 7f52d9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T05:37:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.543+0000 7f52d9ffb640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-16T05:37:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'bar' already exists 2024-09-16T05:37:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.547+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 >> v1:172.21.15.63:6812/3643207211 conn(0x7f52c8036a20 legacy=0x7f52c8038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.547+0000 7f52eb950640 10 monclient: shutdown 2024-09-16T05:37:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.547+0000 7f52eb950640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.547+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 >> v1:172.21.15.6:6790/0 conn(0x7f52e409ac90 legacy=0x7f52e4133290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.547+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 shutdown_connections 2024-09-16T05:37:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.547+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 >> 172.21.15.63:0/1186550994 conn(0x7f52e40a5290 msgr2=0x7f52e4071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.547+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 shutdown_connections 2024-09-16T05:37:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.547+0000 7f52eb950640 1 -- 172.21.15.63:0/1186550994 wait complete. 2024-09-16T05:37:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush link host1 root=bar 2024-09-16T05:37:20.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient: get_monmap_and_config 2024-09-16T05:37:20.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient: build_initial_monmap 2024-09-16T05:37:20.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient: monmap: 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:20.685945+0000 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:20.685945+0000 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:20.687 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 1 Processor -- start 2024-09-16T05:37:20.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 1 -- start start 2024-09-16T05:37:20.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient: init 2024-09-16T05:37:20.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient(hunting): picked mon.noname-b con 0x7fe5f009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient(hunting): picked mon.noname-a con 0x7fe5f00a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient(hunting): picked mon.noname-c con 0x7fe5f009b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f0074200 con 0x7fe5f00a0280 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f009f900 con 0x7fe5f009b780 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f009e870 con 0x7fe5f009ac90 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f7fd3640 10 monclient(hunting): authenticate will time out at 1097.463202s 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f6570640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe5f009b780 legacy=0x7fe5f013cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f556e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe5f00a0280 0x7fe5f01398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40510/0 (socket says 172.21.15.63:40510) 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f556e640 1 -- 172.21.15.63:0/3325637986 learned_addr learned my addr 172.21.15.63:0/3325637986 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:20.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.683+0000 7fe5f5d6f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe5f009ac90 0x7fe5f009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47964/0 (socket says 172.21.15.63:47964) 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685790040 0 0) 0x7fe5f0074200 con 0x7fe5f00a0280 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient(hunting): _negotiate 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient(hunting): my global_id is 5082 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5dc003530 con 0x7fe5f00a0280 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3917215996 0 0) 0x7fe5f009e870 con 0x7fe5f009ac90 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient(hunting): _negotiate 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient(hunting): my global_id is 5069 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5f0074200 con 0x7fe5f009ac90 2024-09-16T05:37:20.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 353109432 0 0) 0x7fe5dc003530 con 0x7fe5f00a0280 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 >> v1:172.21.15.6:6790/0 conn(0x7fe5f009ac90 legacy=0x7fe5f009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 >> v1:172.21.15.63:6789/0 conn(0x7fe5f009b780 legacy=0x7fe5f013cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: _finish_hunting 0 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 monclient: found mon.noname-a 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f0141730 con 0x7fe5f00a0280 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: _finish_auth 0 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: _check_auth_tickets 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5ec0019a0 con 0x7fe5f00a0280 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:20.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: dump: 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 5 monclient: authenticate success, global_id 5082 2024-09-16T05:37:20.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5ec001c50 con 0x7fe5f00a0280 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 1 -- 172.21.15.63:0/3325637986 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5ec002060 con 0x7fe5f00a0280 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 4 set_mon_vals no callback set 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f4d6d640 10 monclient: dump: 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.695 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient: shutdown 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3325637986 >> v1:172.21.15.6:6789/0 conn(0x7fe5f00a0280 legacy=0x7fe5f01398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3325637986 shutdown_connections 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3325637986 >> 172.21.15.63:0/3325637986 conn(0x7fe5f00a5290 msgr2=0x7fe5f00a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3325637986 shutdown_connections 2024-09-16T05:37:20.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3325637986 wait complete. 2024-09-16T05:37:20.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient: build_initial_monmap 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient: monmap: 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 Processor -- start 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- start start 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient: init 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient(hunting): picked mon.noname-c con 0x7fe5f009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient(hunting): picked mon.noname-a con 0x7fe5f009b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient(hunting): picked mon.noname-b con 0x7fe5f00a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f013f220 con 0x7fe5f009b780 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f013f500 con 0x7fe5f00a0280 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f013e110 con 0x7fe5f009ac90 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f7fd3640 10 monclient(hunting): authenticate will time out at 1097.467341s 2024-09-16T05:37:20.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f6570640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe5f00a0280 legacy=0x7fe5f009d250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f556e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe5f009b780 0x7fe5f009cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40516/0 (socket says 172.21.15.63:40516) 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.687+0000 7fe5f556e640 1 -- 172.21.15.63:0/3942540723 learned_addr learned my addr 172.21.15.63:0/3942540723 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2913596924 0 0) 0x7fe5f013e110 con 0x7fe5f009ac90 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): my global_id is 5072 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5d4003530 con 0x7fe5f009ac90 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480873991 0 0) 0x7fe5f013f220 con 0x7fe5f009b780 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:20.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): my global_id is 5085 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5f013e110 con 0x7fe5f009b780 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3290526940 0 0) 0x7fe5d4003530 con 0x7fe5f009ac90 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5f013f220 con 0x7fe5f009ac90 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5e4001880 con 0x7fe5f009ac90 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): dump: 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.703 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 665964653 0 0) 0x7fe5f013e110 con 0x7fe5f009b780 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5d4003530 con 0x7fe5f009b780 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5ec001fa0 con 0x7fe5f009b780 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient(hunting): dump: 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.704 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3532162984 0 0) 0x7fe5f013f220 con 0x7fe5f009ac90 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 >> v1:172.21.15.63:6789/0 conn(0x7fe5f00a0280 legacy=0x7fe5f009d250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 >> v1:172.21.15.6:6789/0 conn(0x7fe5f009b780 legacy=0x7fe5f009cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 monclient: found mon.c 2024-09-16T05:37:20.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:20.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f006e040 con 0x7fe5f009ac90 2024-09-16T05:37:20.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 5 monclient: authenticate success, global_id 5072 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 10 monclient: _renew_subs 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f009e0d0 con 0x7fe5f009ac90 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 10 monclient: _renew_subs 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5f01d46c0 con 0x7fe5f009ac90 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5e40047f0 con 0x7fe5f009ac90 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:20.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5e4004b40 con 0x7fe5f009ac90 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 10 monclient: dump: 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:20.708 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:20.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e37fe640 4 set_mon_vals no callback set 2024-09-16T05:37:20.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:20.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:20.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5d8005090 con 0x7fe5f009ac90 2024-09-16T05:37:20.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe5e400f8e0 con 0x7fe5f009ac90 2024-09-16T05:37:20.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(75..75 src has 1..75) ==== 5307+0+0 (unknown 4254335399 0 0) 0x7fe5e4044660 con 0x7fe5f009ac90 2024-09-16T05:37:20.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.691+0000 7fe5f556e640 10 monclient: get_auth_request con 0x7fe5d4036a30 auth_method 0 2024-09-16T05:37:20.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.703+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe5e4015220 con 0x7fe5f009ac90 2024-09-16T05:37:20.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.703+0000 7fe5e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:20.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.703+0000 7fe5e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:20.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.879+0000 7fe5f7fd3640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:20.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.879+0000 7fe5f7fd3640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:20.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.879+0000 7fe5f7fd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:20.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:20.879+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) -- 0x7fe5d8005b90 con 0x7fe5f009ac90 2024-09-16T05:37:21.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.027+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 linked item id -11 name 'host1' to location {root=bar} in crush map v76) ==== 168+0+0 (unknown 2744826401 0 0) 0x7fe5e4014b20 con 0x7fe5f009ac90 2024-09-16T05:37:21.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.027+0000 7fe5e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.027+0000 7fe5e27fc640 10 monclient: _finish_command 2 = system:0 linked item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T05:37:21.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.131+0000 7fe5f7fd3640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.131+0000 7fe5f7fd3640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.131+0000 7fe5f7fd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:21.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.131+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) -- 0x7fe5d8005610 con 0x7fe5f009ac90 2024-09-16T05:37:21.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.131+0000 7fe5e27fc640 1 -- 172.21.15.63:0/3942540723 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v76) ==== 177+0+0 (unknown 4098677782 0 0) 0x7fe5e4018e70 con 0x7fe5f009ac90 2024-09-16T05:37:21.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.131+0000 7fe5e27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.131+0000 7fe5e27fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T05:37:21.135 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T05:37:21.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.135+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe5d4036a30 legacy=0x7fe5d4038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.135+0000 7fe5f7fd3640 10 monclient: shutdown 2024-09-16T05:37:21.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.135+0000 7fe5f7fd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:21.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.135+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 >> v1:172.21.15.6:6790/0 conn(0x7fe5f009ac90 legacy=0x7fe5f009c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.135+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 shutdown_connections 2024-09-16T05:37:21.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.135+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 >> 172.21.15.63:0/3942540723 conn(0x7fe5f00a5290 msgr2=0x7fe5f006e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:21.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.135+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 shutdown_connections 2024-09-16T05:37:21.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.135+0000 7fe5f7fd3640 1 -- 172.21.15.63:0/3942540723 wait complete. 2024-09-16T05:37:21.152 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush link host1 root=bar 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.271+0000 7f672726d640 10 monclient: get_monmap_and_config 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.271+0000 7f672726d640 10 monclient: build_initial_monmap 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.271+0000 7f672726d640 10 monclient: monmap: 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:21.275325+0000 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:21.275325+0000 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:21.277 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.271+0000 7f672726d640 1 Processor -- start 2024-09-16T05:37:21.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.271+0000 7f672726d640 1 -- start start 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.271+0000 7f672726d640 10 monclient: init 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient(hunting): picked mon.noname-b con 0x7f672009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient(hunting): picked mon.noname-a con 0x7f67200a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient(hunting): picked mon.noname-c con 0x7f672009b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6720074200 con 0x7f67200a0280 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672009f900 con 0x7f672009b780 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672009e870 con 0x7f672009ac90 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient(hunting): authenticate will time out at 1098.052338s 2024-09-16T05:37:21.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672580a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f672009b780 legacy=0x7f672013cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6724808640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f67200a0280 0x7f67201398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40532/0 (socket says 172.21.15.63:40532) 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6725009640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f672009ac90 0x7f672009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47978/0 (socket says 172.21.15.63:47978) 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6724808640 1 -- 172.21.15.63:0/3880988342 learned_addr learned my addr 172.21.15.63:0/3880988342 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2653495118 0 0) 0x7f6720074200 con 0x7f67200a0280 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient(hunting): my global_id is 5088 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f670c003530 con 0x7f67200a0280 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2871881451 0 0) 0x7f672009e870 con 0x7f672009ac90 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:21.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient(hunting): my global_id is 5075 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6720074200 con 0x7f672009ac90 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3146910657 0 0) 0x7f670c003530 con 0x7f67200a0280 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 >> v1:172.21.15.6:6790/0 conn(0x7f672009ac90 legacy=0x7f672009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 >> v1:172.21.15.63:6789/0 conn(0x7f672009b780 legacy=0x7f672013cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 monclient: found mon.noname-a 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6720141730 con 0x7f67200a0280 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: _finish_auth 0 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:21.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6710001620 con 0x7f67200a0280 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: dump: 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.283 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 5 monclient: authenticate success, global_id 5088 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f67100018d0 con 0x7f67200a0280 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 1 -- 172.21.15.63:0/3880988342 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6710001ca0 con 0x7f67200a0280 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 4 set_mon_vals no callback set 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f6717fff640 10 monclient: dump: 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.284 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient: shutdown 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- 172.21.15.63:0/3880988342 >> v1:172.21.15.6:6789/0 conn(0x7f67200a0280 legacy=0x7f67201398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- 172.21.15.63:0/3880988342 shutdown_connections 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- 172.21.15.63:0/3880988342 >> 172.21.15.63:0/3880988342 conn(0x7f67200a5290 msgr2=0x7f67200a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- 172.21.15.63:0/3880988342 shutdown_connections 2024-09-16T05:37:21.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- 172.21.15.63:0/3880988342 wait complete. 2024-09-16T05:37:21.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient: build_initial_monmap 2024-09-16T05:37:21.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient: monmap: 2024-09-16T05:37:21.289 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 Processor -- start 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 1 -- start start 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.275+0000 7f672726d640 10 monclient: init 2024-09-16T05:37:21.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient(hunting): picked mon.noname-b con 0x7f672009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient(hunting): picked mon.noname-c con 0x7f672009b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient(hunting): picked mon.noname-a con 0x7f67200a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6725009640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f672009ac90 legacy=0x7f67201cb9a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6720140190 con 0x7f67200a0280 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672009c0f0 con 0x7f672009ac90 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672009e0d0 con 0x7f672009b780 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient(hunting): authenticate will time out at 1098.056357s 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6724808640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f672009b780 0x7f67201d09f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47994/0 (socket says 172.21.15.63:47994) 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6724808640 1 -- 172.21.15.63:0/2294591914 learned_addr learned my addr 172.21.15.63:0/2294591914 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:21.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2932455686 0 0) 0x7f672009e0d0 con 0x7f672009b780 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): my global_id is 5078 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6704003530 con 0x7f672009b780 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1093277970 0 0) 0x7f6720140190 con 0x7f67200a0280 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): my global_id is 5091 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f672009e0d0 con 0x7f67200a0280 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3565604088 0 0) 0x7f672009e0d0 con 0x7f67200a0280 2024-09-16T05:37:21.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6720140190 con 0x7f67200a0280 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6718004340 con 0x7f67200a0280 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.293 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 884483233 0 0) 0x7f6704003530 con 0x7f672009b780 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f672009e0d0 con 0x7f672009b780 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6710001c20 con 0x7f672009b780 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.294 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2470246491 0 0) 0x7f6720140190 con 0x7f67200a0280 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 >> v1:172.21.15.6:6790/0 conn(0x7f672009b780 legacy=0x7f67201d09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 >> v1:172.21.15.63:6789/0 conn(0x7f672009ac90 legacy=0x7f67201cb9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 monclient: found mon.a 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6720140470 con 0x7f67200a0280 2024-09-16T05:37:21.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 5 monclient: authenticate success, global_id 5091 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient: _renew_subs 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6720140890 con 0x7f67200a0280 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient: _renew_subs 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f67201345a0 con 0x7f67200a0280 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f67180045a0 con 0x7f67200a0280 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f67180048f0 con 0x7f67200a0280 2024-09-16T05:37:21.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 10 monclient: dump: 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f67177fe640 4 set_mon_vals no callback set 2024-09-16T05:37:21.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:21.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:21.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6700005090 con 0x7f67200a0280 2024-09-16T05:37:21.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6718004b10 con 0x7f67200a0280 2024-09-16T05:37:21.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.279+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 5433+0+0 (unknown 1172967212 0 0) 0x7f6718044af0 con 0x7f67200a0280 2024-09-16T05:37:21.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.283+0000 7f6725009640 10 monclient: get_auth_request con 0x7f6704036a20 auth_method 0 2024-09-16T05:37:21.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.295+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6718015630 con 0x7f67200a0280 2024-09-16T05:37:21.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.295+0000 7f6715ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:21.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.295+0000 7f6715ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:21.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.463+0000 7f672726d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.463+0000 7f672726d640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.463+0000 7f672726d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.463+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) -- 0x7f6700005b90 con 0x7f67200a0280 2024-09-16T05:37:21.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.463+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v76) ==== 177+0+0 (unknown 4098677782 0 0) 0x7f6718014f30 con 0x7f67200a0280 2024-09-16T05:37:21.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.463+0000 7f6715ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.463+0000 7f6715ffb640 10 monclient: _finish_command 2 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T05:37:21.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.567+0000 7f672726d640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.567+0000 7f672726d640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.567+0000 7f672726d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.567+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) -- 0x7f67000010a0 con 0x7f67200a0280 2024-09-16T05:37:21.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.567+0000 7f6715ffb640 1 -- 172.21.15.63:0/2294591914 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v76) ==== 177+0+0 (unknown 4098677782 0 0) 0x7f6718019280 con 0x7f67200a0280 2024-09-16T05:37:21.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.567+0000 7f6715ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T05:37:21.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.567+0000 7f6715ffb640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T05:37:21.571 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T05:37:21.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.571+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6704036a20 legacy=0x7f6704038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.571+0000 7f672726d640 10 monclient: shutdown 2024-09-16T05:37:21.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.571+0000 7f672726d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:21.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.571+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 >> v1:172.21.15.6:6789/0 conn(0x7f67200a0280 legacy=0x7f67201d4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.571+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 shutdown_connections 2024-09-16T05:37:21.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.571+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 >> 172.21.15.63:0/2294591914 conn(0x7f67200a5290 msgr2=0x7f67200a44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:21.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.571+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 shutdown_connections 2024-09-16T05:37:21.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.571+0000 7f672726d640 1 -- 172.21.15.63:0/2294591914 wait complete. 2024-09-16T05:37:21.589 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush link host2 root=bar 2024-09-16T05:37:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.707+0000 7fb32dad0640 10 monclient: get_monmap_and_config 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.707+0000 7fb32dad0640 10 monclient: build_initial_monmap 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.707+0000 7fb32dad0640 10 monclient: monmap: 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:21.712156+0000 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:21.712156+0000 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:21.713 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 Processor -- start 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- start start 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient: init 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient(hunting): picked mon.noname-a con 0x7fb32809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient(hunting): picked mon.noname-b con 0x7fb3280a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient(hunting): picked mon.noname-c con 0x7fb32809b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb328074200 con 0x7fb32809ac90 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32809f900 con 0x7fb32809b780 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32809e870 con 0x7fb3280a0280 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient(hunting): authenticate will time out at 1098.489387s 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb327fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fb32809b780 legacy=0x7fb32813cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3277fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb32809ac90 0x7fb32809b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40546/0 (socket says 172.21.15.63:40546) 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3277fe640 1 -- 172.21.15.63:0/3552312298 learned_addr learned my addr 172.21.15.63:0/3552312298 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb326ffd640 1 --1- 172.21.15.63:0/3552312298 >> v1:172.21.15.6:6790/0 conn(0x7fb3280a0280 0x7fb3281398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47998/0 (socket says 172.21.15.63:47998) 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768739142 0 0) 0x7fb328074200 con 0x7fb32809ac90 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient(hunting): my global_id is 5094 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb314003530 con 0x7fb32809ac90 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278617911 0 0) 0x7fb32809e870 con 0x7fb3280a0280 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient(hunting): my global_id is 5081 2024-09-16T05:37:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb328074200 con 0x7fb3280a0280 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1482753632 0 0) 0x7fb314003530 con 0x7fb32809ac90 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 >> v1:172.21.15.6:6790/0 conn(0x7fb3280a0280 legacy=0x7fb3281398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 >> v1:172.21.15.63:6789/0 conn(0x7fb32809b780 legacy=0x7fb32813cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 monclient: found mon.noname-a 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb328141730 con 0x7fb32809ac90 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:21.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb3180019a0 con 0x7fb32809ac90 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: dump: 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.724 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 5 monclient: authenticate success, global_id 5094 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb318001c00 con 0x7fb32809ac90 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 1 -- 172.21.15.63:0/3552312298 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb318001f50 con 0x7fb32809ac90 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb3267fc640 10 monclient: dump: 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.725 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 4 set_mon_vals no callback set 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 10 monclient: shutdown 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:21.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- 172.21.15.63:0/3552312298 >> v1:172.21.15.6:6789/0 conn(0x7fb32809ac90 legacy=0x7fb32809b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- 172.21.15.63:0/3552312298 shutdown_connections 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- 172.21.15.63:0/3552312298 >> 172.21.15.63:0/3552312298 conn(0x7fb3280a5290 msgr2=0x7fb3280a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- 172.21.15.63:0/3552312298 shutdown_connections 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.711+0000 7fb32dad0640 1 -- 172.21.15.63:0/3552312298 wait complete. 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: build_initial_monmap 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: monmap: 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:21.727 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 1 Processor -- start 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 1 -- start start 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: init 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient(hunting): picked mon.noname-a con 0x7fb32809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient(hunting): picked mon.noname-c con 0x7fb32809b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient(hunting): picked mon.noname-b con 0x7fb3280a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32813c770 con 0x7fb32809ac90 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3281ca460 con 0x7fb3280a0280 2024-09-16T05:37:21.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32806e0c0 con 0x7fb32809b780 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient(hunting): authenticate will time out at 1098.493395s 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb327fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fb3280a0280 legacy=0x7fb3281d18a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb326ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb32809b780 0x7fb3281c9d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48014/0 (socket says 172.21.15.63:48014) 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb3277fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb32809ac90 0x7fb3281c9640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40554/0 (socket says 172.21.15.63:40554) 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb326ffd640 1 -- 172.21.15.63:0/4253511010 learned_addr learned my addr 172.21.15.63:0/4253511010 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3799664286 0 0) 0x7fb32813c770 con 0x7fb32809ac90 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): my global_id is 5097 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb308003530 con 0x7fb32809ac90 2024-09-16T05:37:21.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1415132590 0 0) 0x7fb32806e0c0 con 0x7fb32809b780 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): my global_id is 5084 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb32813c770 con 0x7fb32809b780 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4104434863 0 0) 0x7fb308003530 con 0x7fb32809ac90 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb32806e0c0 con 0x7fb32809ac90 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb318001960 con 0x7fb32809ac90 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): dump: 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.730 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 849202020 0 0) 0x7fb32813c770 con 0x7fb32809b780 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb308003530 con 0x7fb32809b780 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb310001830 con 0x7fb32809b780 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:21.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient(hunting): dump: 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 371349777 0 0) 0x7fb32806e0c0 con 0x7fb32809ac90 2024-09-16T05:37:21.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 >> v1:172.21.15.6:6790/0 conn(0x7fb32809b780 legacy=0x7fb3281c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 >> v1:172.21.15.63:6789/0 conn(0x7fb3280a0280 legacy=0x7fb3281d18a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 monclient: found mon.a 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32809c0f0 con 0x7fb32809ac90 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient: _finish_auth 0 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 5 monclient: authenticate success, global_id 5097 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: _renew_subs 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb32813be90 con 0x7fb32809ac90 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: _renew_subs 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb32813e180 con 0x7fb32809ac90 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb318001b00 con 0x7fb32809ac90 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb318004fc0 con 0x7fb32809ac90 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb3257fa640 4 set_mon_vals no callback set 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb31ffff640 10 monclient: dump: 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.734 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:21.735 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:21.735 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:21.735 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:21.735 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:21.735 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:21.735 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:21.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:21.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:21.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.715+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb30c005090 con 0x7fb32809ac90 2024-09-16T05:37:21.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.719+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb318005160 con 0x7fb32809ac90 2024-09-16T05:37:21.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.719+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 5433+0+0 (unknown 1172967212 0 0) 0x7fb318044bb0 con 0x7fb32809ac90 2024-09-16T05:37:21.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.719+0000 7fb326ffd640 10 monclient: get_auth_request con 0x7fb3080369e0 auth_method 0 2024-09-16T05:37:21.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.731+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fb318044e20 con 0x7fb32809ac90 2024-09-16T05:37:21.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.731+0000 7fb31ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:21.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.731+0000 7fb31ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:21.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.899+0000 7fb32dad0640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:21.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.899+0000 7fb32dad0640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:21.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.899+0000 7fb32dad0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:21.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:21.899+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) -- 0x7fb30c005b90 con 0x7fb32809ac90 2024-09-16T05:37:22.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.035+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 linked item id -16 name 'host2' to location {root=bar} in crush map v77) ==== 168+0+0 (unknown 1442900798 0 0) 0x7fb3180157b0 con 0x7fb32809ac90 2024-09-16T05:37:22.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.035+0000 7fb31ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.035+0000 7fb31ffff640 10 monclient: _finish_command 2 = system:0 linked item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T05:37:22.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.139+0000 7fb32dad0640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.139+0000 7fb32dad0640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.139+0000 7fb32dad0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.139+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) -- 0x7fb30c005610 con 0x7fb32809ac90 2024-09-16T05:37:22.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.139+0000 7fb31ffff640 1 -- 172.21.15.63:0/4253511010 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v77) ==== 177+0+0 (unknown 647421750 0 0) 0x7fb3180150b0 con 0x7fb32809ac90 2024-09-16T05:37:22.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.139+0000 7fb31ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.139+0000 7fb31ffff640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T05:37:22.144 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T05:37:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.143+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb3080369e0 legacy=0x7fb308038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.143+0000 7fb32dad0640 10 monclient: shutdown 2024-09-16T05:37:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.143+0000 7fb32dad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.143+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 >> v1:172.21.15.6:6789/0 conn(0x7fb32809ac90 legacy=0x7fb3281c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.143+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 shutdown_connections 2024-09-16T05:37:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.143+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 >> 172.21.15.63:0/4253511010 conn(0x7fb3280a5290 msgr2=0x7fb3281368b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.143+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 shutdown_connections 2024-09-16T05:37:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.143+0000 7fb32dad0640 1 -- 172.21.15.63:0/4253511010 wait complete. 2024-09-16T05:37:22.161 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush link host2 root=bar 2024-09-16T05:37:22.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.279+0000 7f4c598a3640 10 monclient: get_monmap_and_config 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.279+0000 7f4c598a3640 10 monclient: build_initial_monmap 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.279+0000 7f4c598a3640 10 monclient: monmap: 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:22.284230+0000 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:22.284230+0000 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:22.285 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:22.286 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 Processor -- start 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- start start 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient: init 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient(hunting): picked mon.noname-c con 0x7f4c54136a50 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient(hunting): picked mon.noname-a con 0x7f4c5409fd70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient(hunting): picked mon.noname-b con 0x7f4c5413d0d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c540a05a0 con 0x7f4c5409fd70 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c540a0880 con 0x7f4c54136a50 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c540a0b60 con 0x7f4c5413d0d0 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient(hunting): authenticate will time out at 1099.061184s 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c52ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4c54136a50 legacy=0x7f4c54136e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c537fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4c5413d0d0 0x7f4c5413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48020/0 (socket says 172.21.15.63:48020) 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c537fe640 1 -- 172.21.15.63:0/1990757400 learned_addr learned my addr 172.21.15.63:0/1990757400 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c527fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4c5409fd70 0x7f4c5413b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40566/0 (socket says 172.21.15.63:40566) 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155132467 0 0) 0x7f4c540a05a0 con 0x7f4c5409fd70 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient(hunting): my global_id is 5100 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c3c003530 con 0x7f4c5409fd70 2024-09-16T05:37:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033863775 0 0) 0x7f4c540a0b60 con 0x7f4c5413d0d0 2024-09-16T05:37:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient(hunting): my global_id is 5087 2024-09-16T05:37:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c540a05a0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 615285266 0 0) 0x7f4c3c003530 con 0x7f4c5409fd70 2024-09-16T05:37:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 >> v1:172.21.15.6:6790/0 conn(0x7f4c5413d0d0 legacy=0x7f4c5413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 >> v1:172.21.15.63:6789/0 conn(0x7f4c54136a50 legacy=0x7f4c54136e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 monclient: found mon.noname-a 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c54137540 con 0x7f4c5409fd70 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4c48001550 con 0x7f4c5409fd70 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: dump: 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 5 monclient: authenticate success, global_id 5100 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c48001800 con 0x7f4c5409fd70 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 1 -- 172.21.15.63:0/1990757400 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4c48001bd0 con 0x7f4c5409fd70 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c51ffb640 10 monclient: dump: 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 4 set_mon_vals no callback set 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 10 monclient: shutdown 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- 172.21.15.63:0/1990757400 >> v1:172.21.15.6:6789/0 conn(0x7f4c5409fd70 legacy=0x7f4c5413b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- 172.21.15.63:0/1990757400 shutdown_connections 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- 172.21.15.63:0/1990757400 >> 172.21.15.63:0/1990757400 conn(0x7f4c540a5290 msgr2=0x7f4c540a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:22.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- 172.21.15.63:0/1990757400 shutdown_connections 2024-09-16T05:37:22.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.283+0000 7f4c598a3640 1 -- 172.21.15.63:0/1990757400 wait complete. 2024-09-16T05:37:22.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: build_initial_monmap 2024-09-16T05:37:22.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: monmap: 2024-09-16T05:37:22.301 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:22.301 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:22.301 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:22.301 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:22.301 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 1 Processor -- start 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 1 -- start start 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: init 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient(hunting): picked mon.noname-c con 0x7f4c5409fd70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:22.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient(hunting): picked mon.noname-b con 0x7f4c54136a50 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient(hunting): picked mon.noname-a con 0x7f4c5413d0d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c541206d0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c527fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4c54136a50 legacy=0x7f4c541d06a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c541203f0 con 0x7f4c54136a50 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c54120c90 con 0x7f4c5409fd70 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient(hunting): authenticate will time out at 1099.065859s 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c52ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4c5409fd70 0x7f4c54135110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48024/0 (socket says 172.21.15.63:48024) 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c52ffd640 1 -- 172.21.15.63:0/2489425971 learned_addr learned my addr 172.21.15.63:0/2489425971 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c537fe640 1 --1- 172.21.15.63:0/2489425971 >> v1:172.21.15.6:6789/0 conn(0x7f4c5413d0d0 0x7f4c541d3dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40568/0 (socket says 172.21.15.63:40568) 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068550952 0 0) 0x7f4c541206d0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:22.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): my global_id is 5103 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c40003530 con 0x7f4c5413d0d0 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696029911 0 0) 0x7f4c54120c90 con 0x7f4c5409fd70 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): my global_id is 5090 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c541206d0 con 0x7f4c5409fd70 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2470643612 0 0) 0x7f4c40003530 con 0x7f4c5413d0d0 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c54120c90 con 0x7f4c5413d0d0 2024-09-16T05:37:22.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4c4c001bc0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): dump: 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.305 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:22.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2356633505 0 0) 0x7f4c541206d0 con 0x7f4c5409fd70 2024-09-16T05:37:22.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c40003530 con 0x7f4c5409fd70 2024-09-16T05:37:22.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4c44004090 con 0x7f4c5409fd70 2024-09-16T05:37:22.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:22.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:22.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient(hunting): dump: 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2958796951 0 0) 0x7f4c54120c90 con 0x7f4c5413d0d0 2024-09-16T05:37:22.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 >> v1:172.21.15.6:6790/0 conn(0x7f4c5409fd70 legacy=0x7f4c54135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 >> v1:172.21.15.63:6789/0 conn(0x7f4c54136a50 legacy=0x7f4c541d06a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 monclient: found mon.a 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c540a43a0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient: _finish_auth 0 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:22.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 5 monclient: authenticate success, global_id 5103 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: _renew_subs 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5409c0f0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: _renew_subs 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4c541209b0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c4c004b70 con 0x7f4c5413d0d0 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4c4c004d20 con 0x7f4c5413d0d0 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:22.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c3bfff640 10 monclient: dump: 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.310 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.287+0000 7f4c50ff9640 4 set_mon_vals no callback set 2024-09-16T05:37:22.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.291+0000 7f4c598a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:22.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.291+0000 7f4c598a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:22.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.291+0000 7f4c598a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.291+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c34005090 con 0x7f4c5413d0d0 2024-09-16T05:37:22.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.291+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4c4c004ec0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.291+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(77..77 src has 1..77) ==== 5457+0+0 (unknown 1984182297 0 0) 0x7f4c4c044ae0 con 0x7f4c5413d0d0 2024-09-16T05:37:22.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.291+0000 7f4c52ffd640 10 monclient: get_auth_request con 0x7f4c40036aa0 auth_method 0 2024-09-16T05:37:22.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.303+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4c4c015600 con 0x7f4c5413d0d0 2024-09-16T05:37:22.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.303+0000 7f4c3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:22.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.303+0000 7f4c3bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:22.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.471+0000 7f4c598a3640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.471+0000 7f4c598a3640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.471+0000 7f4c598a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.471+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) -- 0x7f4c34005b90 con 0x7f4c5413d0d0 2024-09-16T05:37:22.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.471+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v77) ==== 177+0+0 (unknown 647421750 0 0) 0x7f4c4c014f00 con 0x7f4c5413d0d0 2024-09-16T05:37:22.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.471+0000 7f4c3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.471+0000 7f4c3bfff640 10 monclient: _finish_command 2 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T05:37:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) -- 0x7f4c34005610 con 0x7f4c5413d0d0 2024-09-16T05:37:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c3bfff640 1 -- 172.21.15.63:0/2489425971 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v77) ==== 177+0+0 (unknown 647421750 0 0) 0x7f4c4c019250 con 0x7f4c5413d0d0 2024-09-16T05:37:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c3bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T05:37:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c3bfff640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T05:37:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T05:37:22.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4c40036aa0 legacy=0x7f4c40038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 10 monclient: shutdown 2024-09-16T05:37:22.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:22.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 >> v1:172.21.15.6:6789/0 conn(0x7f4c5413d0d0 legacy=0x7f4c541d3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 shutdown_connections 2024-09-16T05:37:22.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 >> 172.21.15.63:0/2489425971 conn(0x7f4c540a5290 msgr2=0x7f4c540a34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:22.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 shutdown_connections 2024-09-16T05:37:22.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.571+0000 7f4c598a3640 1 -- 172.21.15.63:0/2489425971 wait complete. 2024-09-16T05:37:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 2 2024-09-16T05:37:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c osd.7 2024-09-16T05:37:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.703+0000 7fe22de69640 10 monclient: get_monmap_and_config 2024-09-16T05:37:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.703+0000 7fe22de69640 10 monclient: build_initial_monmap 2024-09-16T05:37:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.703+0000 7fe22de69640 10 monclient: monmap: 2024-09-16T05:37:22.708 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:22.707168+0000 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:22.707168+0000 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:22.709 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.703+0000 7fe22de69640 1 Processor -- start 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.703+0000 7fe22de69640 1 -- start start 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.703+0000 7fe22de69640 10 monclient: init 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.703+0000 7fe22de69640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.703+0000 7fe22de69640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient(hunting): picked mon.noname-a con 0x7fe22809acb0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient(hunting): picked mon.noname-c con 0x7fe2280a13f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient(hunting): picked mon.noname-b con 0x7fe22809b7a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2280a0fd0 con 0x7fe22809acb0 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe22809f110 con 0x7fe2280a13f0 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe22809ec90 con 0x7fe22809b7a0 2024-09-16T05:37:22.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient(hunting): authenticate will time out at 1099.484108s 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe226ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe2280a13f0 legacy=0x7fe228139ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2277fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe22809acb0 0x7fe22809b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40582/0 (socket says 172.21.15.63:40582) 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe227fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe22809b7a0 0x7fe22813d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48034/0 (socket says 172.21.15.63:48034) 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2277fe640 1 -- 172.21.15.63:0/1336638632 learned_addr learned my addr 172.21.15.63:0/1336638632 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793269729 0 0) 0x7fe2280a0fd0 con 0x7fe22809acb0 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient(hunting): my global_id is 5106 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe210003530 con 0x7fe22809acb0 2024-09-16T05:37:22.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1733922813 0 0) 0x7fe22809ec90 con 0x7fe22809b7a0 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient(hunting): my global_id is 5093 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2280a0fd0 con 0x7fe22809b7a0 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1821904519 0 0) 0x7fe2280a0fd0 con 0x7fe22809b7a0 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 >> v1:172.21.15.63:6789/0 conn(0x7fe2280a13f0 legacy=0x7fe228139ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 >> v1:172.21.15.6:6789/0 conn(0x7fe22809acb0 legacy=0x7fe22809b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 monclient: found mon.noname-b 2024-09-16T05:37:22.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2280a0730 con 0x7fe22809b7a0 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe220001980 con 0x7fe22809b7a0 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 5 monclient: authenticate success, global_id 5093 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: dump: 2024-09-16T05:37:22.717 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe220001c70 con 0x7fe22809b7a0 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:22.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 4 set_mon_vals no callback set 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 1 -- 172.21.15.63:0/1336638632 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe220002080 con 0x7fe22809b7a0 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe2267fc640 10 monclient: dump: 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.719 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient: shutdown 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- 172.21.15.63:0/1336638632 >> v1:172.21.15.6:6790/0 conn(0x7fe22809b7a0 legacy=0x7fe22813d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- 172.21.15.63:0/1336638632 shutdown_connections 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- 172.21.15.63:0/1336638632 >> 172.21.15.63:0/1336638632 conn(0x7fe2280a1b60 msgr2=0x7fe2280a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- 172.21.15.63:0/1336638632 shutdown_connections 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- 172.21.15.63:0/1336638632 wait complete. 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient: build_initial_monmap 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient: monmap: 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:22.720 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 Processor -- start 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 1 -- start start 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient: init 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:22.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient(hunting): picked mon.noname-c con 0x7fe22809acb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient(hunting): picked mon.noname-a con 0x7fe22809b7a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.707+0000 7fe22de69640 10 monclient(hunting): picked mon.noname-b con 0x7fe2280a13f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe228125420 con 0x7fe22809b7a0 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe228124150 con 0x7fe2280a13f0 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe228123130 con 0x7fe22809acb0 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 monclient(hunting): authenticate will time out at 1099.488038s 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe227fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe2280a13f0 legacy=0x7fe2281d0f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe226ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe22809b7a0 0x7fe228134a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40592/0 (socket says 172.21.15.63:40592) 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe226ffd640 1 -- 172.21.15.63:0/1557322719 learned_addr learned my addr 172.21.15.63:0/1557322719 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:22.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe2277fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe22809acb0 0x7fe2281c8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48046/0 (socket says 172.21.15.63:48046) 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1848368299 0 0) 0x7fe228125420 con 0x7fe22809b7a0 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): my global_id is 5109 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe214003560 con 0x7fe22809b7a0 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1393233948 0 0) 0x7fe228123130 con 0x7fe22809acb0 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): my global_id is 5096 2024-09-16T05:37:22.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe228125420 con 0x7fe22809acb0 2024-09-16T05:37:22.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1867600410 0 0) 0x7fe228125420 con 0x7fe22809acb0 2024-09-16T05:37:22.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe228123130 con 0x7fe22809acb0 2024-09-16T05:37:22.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3774557539 0 0) 0x7fe214003560 con 0x7fe22809b7a0 2024-09-16T05:37:22.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe228125420 con 0x7fe22809b7a0 2024-09-16T05:37:22.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe21c004060 con 0x7fe22809b7a0 2024-09-16T05:37:22.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:22.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:22.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): dump: 2024-09-16T05:37:22.727 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.727 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.727 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.727 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.727 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.727 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.727 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe218001140 con 0x7fe22809acb0 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient(hunting): dump: 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.728 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 666613277 0 0) 0x7fe228125420 con 0x7fe22809b7a0 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 >> v1:172.21.15.6:6790/0 conn(0x7fe22809acb0 legacy=0x7fe2281c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 >> v1:172.21.15.63:6789/0 conn(0x7fe2280a13f0 legacy=0x7fe2281d0f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 monclient: found mon.a 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe228125140 con 0x7fe22809b7a0 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient: _finish_auth 0 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 5 monclient: authenticate success, global_id 5109 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 monclient: _renew_subs 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2281353a0 con 0x7fe22809b7a0 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 monclient: _renew_subs 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2281d2640 con 0x7fe22809b7a0 2024-09-16T05:37:22.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe21c004310 con 0x7fe22809b7a0 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe21c0046e0 con 0x7fe22809b7a0 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe225ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 10 monclient: dump: 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:22.731 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe208005090 con 0x7fe22809b7a0 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe21c004900 con 0x7fe22809b7a0 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(77..77 src has 1..77) ==== 5457+0+0 (unknown 1984182297 0 0) 0x7fe21c044850 con 0x7fe22809b7a0 2024-09-16T05:37:22.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.711+0000 7fe2277fe640 10 monclient: get_auth_request con 0x7fe2140369d0 auth_method 0 2024-09-16T05:37:22.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.727+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe21c015310 con 0x7fe22809b7a0 2024-09-16T05:37:22.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.727+0000 7fe20ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:22.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.727+0000 7fe20ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:22.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.891+0000 7fe22de69640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:22.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.891+0000 7fe22de69640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:22.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.891+0000 7fe22de69640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.891+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fe208005b90 con 0x7fe22809b7a0 2024-09-16T05:37:22.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.891+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1474 (unknown 2226722781 0 2636823495) 0x7fe21c018fc0 con 0x7fe22809b7a0 2024-09-16T05:37:22.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.891+0000 7fe20ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:22.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.891+0000 7fe20ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.987+0000 7fe22de69640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.987+0000 7fe22de69640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.987+0000 7fe22de69640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:22.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.987+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fe2080008e0 con 0x7fe22809b7a0 2024-09-16T05:37:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.987+0000 7fe20ffff640 1 -- 172.21.15.63:0/1557322719 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1474 (unknown 2226722781 0 2636823495) 0x7fe21c01d880 con 0x7fe22809b7a0 2024-09-16T05:37:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.987+0000 7fe20ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:22.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.987+0000 7fe20ffff640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.991+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe2140369d0 legacy=0x7fe214038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.991+0000 7fe22de69640 10 monclient: shutdown 2024-09-16T05:37:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.991+0000 7fe22de69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:22.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.991+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 >> v1:172.21.15.6:6789/0 conn(0x7fe22809b7a0 legacy=0x7fe228134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.991+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 shutdown_connections 2024-09-16T05:37:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.991+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 >> 172.21.15.63:0/1557322719 conn(0x7fe2280a1b60 msgr2=0x7fe2280a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.991+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 shutdown_connections 2024-09-16T05:37:22.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:22.991+0000 7fe22de69640 1 -- 172.21.15.63:0/1557322719 wait complete. 2024-09-16T05:37:23.008 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:23.008 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c host1 2024-09-16T05:37:23.008 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 2 2024-09-16T05:37:23.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.119+0000 7f2dd6076640 10 monclient: get_monmap_and_config 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.119+0000 7f2dd6076640 10 monclient: build_initial_monmap 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.119+0000 7f2dd6076640 10 monclient: monmap: 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:23.123060+0000 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:23.123060+0000 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:23.124 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.119+0000 7f2dd6076640 1 Processor -- start 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.119+0000 7f2dd6076640 1 -- start start 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.119+0000 7f2dd6076640 10 monclient: init 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd009ae20 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd009b910 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd0139fc0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:23.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd00a0ff0 con 0x7f2dd0139fc0 2024-09-16T05:37:23.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd009da90 con 0x7f2dd009ae20 2024-09-16T05:37:23.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd00a1430 con 0x7f2dd009b910 2024-09-16T05:37:23.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:23.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient(hunting): authenticate will time out at 1099.900583s 2024-09-16T05:37:23.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dcf7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2dd009ae20 legacy=0x7f2dd009b200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:23.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dcffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2dd0139fc0 0x7f2dd013c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40608/0 (socket says 172.21.15.63:40608) 2024-09-16T05:37:23.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dceffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2dd009b910 0x7f2dd01388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48052/0 (socket says 172.21.15.63:48052) 2024-09-16T05:37:23.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dcffff640 1 -- 172.21.15.63:0/1653967717 learned_addr learned my addr 172.21.15.63:0/1653967717 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:23.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1232963810 0 0) 0x7f2dd00a1430 con 0x7f2dd009b910 2024-09-16T05:37:23.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient(hunting): my global_id is 5099 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2db8005740 con 0x7f2dd009b910 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734491320 0 0) 0x7f2dd00a0ff0 con 0x7f2dd0139fc0 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient(hunting): my global_id is 5112 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd00a1430 con 0x7f2dd0139fc0 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1882175839 0 0) 0x7f2dd00a1430 con 0x7f2dd0139fc0 2024-09-16T05:37:23.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 >> v1:172.21.15.6:6790/0 conn(0x7f2dd009b910 legacy=0x7f2dd01388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 >> v1:172.21.15.63:6789/0 conn(0x7f2dd009ae20 legacy=0x7f2dd009b200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 monclient: found mon.noname-a 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd009d1f0 con 0x7f2dd0139fc0 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2dc8001830 con 0x7f2dd0139fc0 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:23.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: dump: 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 5 monclient: authenticate success, global_id 5112 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2dc8001a90 con 0x7f2dd0139fc0 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 1 -- 172.21.15.63:0/1653967717 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2dc8001e60 con 0x7f2dd0139fc0 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 4 set_mon_vals no callback set 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dce7fc640 10 monclient: dump: 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.134 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient: shutdown 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 1 -- 172.21.15.63:0/1653967717 >> v1:172.21.15.6:6789/0 conn(0x7f2dd0139fc0 legacy=0x7f2dd013c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 1 -- 172.21.15.63:0/1653967717 shutdown_connections 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 1 -- 172.21.15.63:0/1653967717 >> 172.21.15.63:0/1653967717 conn(0x7f2dd00a1b60 msgr2=0x7f2dd00a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 1 -- 172.21.15.63:0/1653967717 shutdown_connections 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 1 -- 172.21.15.63:0/1653967717 wait complete. 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient: build_initial_monmap 2024-09-16T05:37:23.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.123+0000 7f2dd6076640 10 monclient: monmap: 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 1 Processor -- start 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 1 -- start start 2024-09-16T05:37:23.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: init 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd009ae20 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd009b910 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd0139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd0126b00 con 0x7f2dd009b910 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd0126de0 con 0x7f2dd009ae20 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dcf7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2dd009ae20 legacy=0x7f2dd0138350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd0072270 con 0x7f2dd0139fc0 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient(hunting): authenticate will time out at 1099.904720s 2024-09-16T05:37:23.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dceffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2dd009b910 0x7f2dd01d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40618/0 (socket says 172.21.15.63:40618) 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dcffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2dd0139fc0 0x7f2dd01d3cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48062/0 (socket says 172.21.15.63:48062) 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dceffd640 1 -- 172.21.15.63:0/269496375 learned_addr learned my addr 172.21.15.63:0/269496375 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 43285361 0 0) 0x7f2dd0072270 con 0x7f2dd0139fc0 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): my global_id is 5102 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dbc003530 con 0x7f2dd0139fc0 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156436129 0 0) 0x7f2dd0126b00 con 0x7f2dd009b910 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): my global_id is 5115 2024-09-16T05:37:23.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd0072270 con 0x7f2dd009b910 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3861318524 0 0) 0x7f2dd0072270 con 0x7f2dd009b910 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dd0126b00 con 0x7f2dd009b910 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2dc4001a40 con 0x7f2dd009b910 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): dump: 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.141 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3489444233 0 0) 0x7f2dbc003530 con 0x7f2dd0139fc0 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dd0072270 con 0x7f2dd0139fc0 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2dc8001730 con 0x7f2dd0139fc0 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:23.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient(hunting): dump: 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.143 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2538838527 0 0) 0x7f2dd0072270 con 0x7f2dd0139fc0 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 >> v1:172.21.15.63:6789/0 conn(0x7f2dd009ae20 legacy=0x7f2dd0138350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 >> v1:172.21.15.6:6789/0 conn(0x7f2dd009b910 legacy=0x7f2dd01d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 monclient: found mon.c 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd01270c0 con 0x7f2dd0139fc0 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient: _finish_auth 0 2024-09-16T05:37:23.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 5 monclient: authenticate success, global_id 5102 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: _renew_subs 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd0126820 con 0x7f2dd0139fc0 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: _renew_subs 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2dd0124140 con 0x7f2dd0139fc0 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2dc8001990 con 0x7f2dd0139fc0 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2dc8004fc0 con 0x7f2dd0139fc0 2024-09-16T05:37:23.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dcdffb640 4 set_mon_vals no callback set 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2db7fff640 10 monclient: dump: 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.146 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.127+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db0005090 con 0x7f2dd0139fc0 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.131+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2dc800fd60 con 0x7f2dd0139fc0 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.131+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(77..77 src has 1..77) ==== 5457+0+0 (unknown 1984182297 0 0) 0x7f2dc80449e0 con 0x7f2dd0139fc0 2024-09-16T05:37:23.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.131+0000 7f2dcf7fe640 10 monclient: get_auth_request con 0x7f2dbc0369d0 auth_method 0 2024-09-16T05:37:23.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.143+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2dc8015500 con 0x7f2dd0139fc0 2024-09-16T05:37:23.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.143+0000 7f2db7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:23.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.143+0000 7f2db7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:23.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.311+0000 7f2dd6076640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:23.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.311+0000 7f2dd6076640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:23.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.311+0000 7f2dd6076640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.311+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f2db0005b90 con 0x7f2dd0139fc0 2024-09-16T05:37:23.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.311+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1474 (unknown 2226722781 0 2636823495) 0x7f2dc8019150 con 0x7f2dd0139fc0 2024-09-16T05:37:23.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.311+0000 7f2db7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:23.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.311+0000 7f2db7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:23.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:23.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:23.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f2db00008e0 con 0x7f2dd0139fc0 2024-09-16T05:37:23.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2db7fff640 1 -- 172.21.15.63:0/269496375 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1474 (unknown 2226722781 0 2636823495) 0x7f2dc801da10 con 0x7f2dd0139fc0 2024-09-16T05:37:23.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2db7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:23.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2db7fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:23.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2dbc0369d0 legacy=0x7f2dbc038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 10 monclient: shutdown 2024-09-16T05:37:23.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 >> v1:172.21.15.6:6790/0 conn(0x7f2dd0139fc0 legacy=0x7f2dd01d3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 shutdown_connections 2024-09-16T05:37:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 >> 172.21.15.63:0/269496375 conn(0x7f2dd00a1b60 msgr2=0x7f2dd011d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 shutdown_connections 2024-09-16T05:37:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.411+0000 7f2dd6076640 1 -- 172.21.15.63:0/269496375 wait complete. 2024-09-16T05:37:23.432 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:23.432 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c osd.9 2024-09-16T05:37:23.432 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 2 2024-09-16T05:37:23.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.547+0000 7f4eeb932640 10 monclient: get_monmap_and_config 2024-09-16T05:37:23.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.547+0000 7f4eeb932640 10 monclient: build_initial_monmap 2024-09-16T05:37:23.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.547+0000 7f4eeb932640 10 monclient: monmap: 2024-09-16T05:37:23.551 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:23.550832+0000 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:23.550832+0000 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:23.552 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.547+0000 7f4eeb932640 1 Processor -- start 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.547+0000 7f4eeb932640 1 -- start start 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.547+0000 7f4eeb932640 10 monclient: init 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient(hunting): picked mon.noname-b con 0x7f4ee409ee70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient(hunting): picked mon.noname-c con 0x7f4ee409f960 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient(hunting): picked mon.noname-a con 0x7f4ee40a4c50 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee40a4330 con 0x7f4ee40a4c50 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee40a5540 con 0x7f4ee409f960 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee40a4720 con 0x7f4ee409ee70 2024-09-16T05:37:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:23.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient(hunting): authenticate will time out at 1100.328633s 2024-09-16T05:37:23.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4ee8ecd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4ee409f960 legacy=0x7f4ee413bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:23.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4ee96ce640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4ee409ee70 0x7f4ee409f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48076/0 (socket says 172.21.15.63:48076) 2024-09-16T05:37:23.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4ee96ce640 1 -- 172.21.15.63:0/1448466966 learned_addr learned my addr 172.21.15.63:0/1448466966 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:23.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4ee9ecf640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4ee40a4c50 0x7f4ee413f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40630/0 (socket says 172.21.15.63:40630) 2024-09-16T05:37:23.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1406974124 0 0) 0x7f4ee40a4330 con 0x7f4ee40a4c50 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient(hunting): my global_id is 5118 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed0003530 con 0x7f4ee40a4c50 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2683231923 0 0) 0x7f4ee40a4720 con 0x7f4ee409ee70 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient(hunting): my global_id is 5105 2024-09-16T05:37:23.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ee40a4330 con 0x7f4ee409ee70 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 772769884 0 0) 0x7f4ed0003530 con 0x7f4ee40a4c50 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 >> v1:172.21.15.6:6790/0 conn(0x7f4ee409ee70 legacy=0x7f4ee409f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 >> v1:172.21.15.63:6789/0 conn(0x7f4ee409f960 legacy=0x7f4ee413bd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 monclient: found mon.noname-a 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee40a1280 con 0x7f4ee40a4c50 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: _finish_auth 0 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:23.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4edc0018d0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: dump: 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.561 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 5 monclient: authenticate success, global_id 5118 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4edc001bf0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 4 set_mon_vals no callback set 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 1 -- 172.21.15.63:0/1448466966 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4edc001fc0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4edbfff640 10 monclient: dump: 2024-09-16T05:37:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient: shutdown 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 1 -- 172.21.15.63:0/1448466966 >> v1:172.21.15.6:6789/0 conn(0x7f4ee40a4c50 legacy=0x7f4ee413f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 1 -- 172.21.15.63:0/1448466966 shutdown_connections 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 1 -- 172.21.15.63:0/1448466966 >> 172.21.15.63:0/1448466966 conn(0x7f4ee40a5da0 msgr2=0x7f4ee40a61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 1 -- 172.21.15.63:0/1448466966 shutdown_connections 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 1 -- 172.21.15.63:0/1448466966 wait complete. 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient: build_initial_monmap 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.551+0000 7f4eeb932640 10 monclient: monmap: 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 1 Processor -- start 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 1 -- start start 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient: init 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient(hunting): picked mon.noname-a con 0x7f4ee409ee70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient(hunting): picked mon.noname-b con 0x7f4ee409f960 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient(hunting): picked mon.noname-c con 0x7f4ee40a4c50 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee4132f00 con 0x7f4ee409ee70 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee4135b50 con 0x7f4ee409f960 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee40a82d0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient(hunting): authenticate will time out at 1100.332984s 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ee8ecd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4ee409f960 legacy=0x7f4ee41327f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ee96ce640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4ee409ee70 0x7f4ee41320e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40634/0 (socket says 172.21.15.63:40634) 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ee96ce640 1 -- 172.21.15.63:0/2194903525 learned_addr learned my addr 172.21.15.63:0/2194903525 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366502565 0 0) 0x7f4ee4132f00 con 0x7f4ee409ee70 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): my global_id is 5121 2024-09-16T05:37:23.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec8003530 con 0x7f4ee409ee70 2024-09-16T05:37:23.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 137301319 0 0) 0x7f4ee40a82d0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): my global_id is 5108 2024-09-16T05:37:23.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ee4132f00 con 0x7f4ee40a4c50 2024-09-16T05:37:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 273261756 0 0) 0x7f4ec8003530 con 0x7f4ee409ee70 2024-09-16T05:37:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ee40a82d0 con 0x7f4ee409ee70 2024-09-16T05:37:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4ee0001840 con 0x7f4ee409ee70 2024-09-16T05:37:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1551901741 0 0) 0x7f4ee4132f00 con 0x7f4ee40a4c50 2024-09-16T05:37:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ec8003530 con 0x7f4ee40a4c50 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4edc001890 con 0x7f4ee40a4c50 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1210964609 0 0) 0x7f4ec8003530 con 0x7f4ee40a4c50 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 >> v1:172.21.15.63:6789/0 conn(0x7f4ee409f960 legacy=0x7f4ee41327f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 >> v1:172.21.15.6:6789/0 conn(0x7f4ee409ee70 legacy=0x7f4ee41320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 monclient: found mon.c 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee40a0500 con 0x7f4ee40a4c50 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 5 monclient: authenticate success, global_id 5108 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient: _renew_subs 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee4072750 con 0x7f4ee40a4c50 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient: _renew_subs 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ee4135870 con 0x7f4ee40a4c50 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4edc001bf0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4edc005060 con 0x7f4ee40a4c50 2024-09-16T05:37:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4ed9ffb640 10 monclient: dump: 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.555+0000 7f4edb7fe640 4 set_mon_vals no callback set 2024-09-16T05:37:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.559+0000 7f4eeb932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.559+0000 7f4eeb932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.559+0000 7f4eeb932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.559+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec4005090 con 0x7f4ee40a4c50 2024-09-16T05:37:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.559+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4edc00fe00 con 0x7f4ee40a4c50 2024-09-16T05:37:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.559+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(77..77 src has 1..77) ==== 5457+0+0 (unknown 1984182297 0 0) 0x7f4edc044b50 con 0x7f4ee40a4c50 2024-09-16T05:37:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.559+0000 7f4ee96ce640 10 monclient: get_auth_request con 0x7f4ec8036930 auth_method 0 2024-09-16T05:37:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.571+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4edc0157f0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.571+0000 7f4ed9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.571+0000 7f4ed9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:23.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.735+0000 7f4eeb932640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:23.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.735+0000 7f4eeb932640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:23.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.735+0000 7f4eeb932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.735+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f4ec4005b90 con 0x7f4ee40a4c50 2024-09-16T05:37:23.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.735+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1474 (unknown 2226722781 0 2636823495) 0x7f4edc0150f0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.735+0000 7f4ed9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:23.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.735+0000 7f4ed9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:23.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.839+0000 7f4eeb932640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:23.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.839+0000 7f4eeb932640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:23.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.839+0000 7f4eeb932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:23.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.839+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f4ec40008e0 con 0x7f4ee40a4c50 2024-09-16T05:37:23.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.839+0000 7f4ed9ffb640 1 -- 172.21.15.63:0/2194903525 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1474 (unknown 2226722781 0 2636823495) 0x7f4edc040700 con 0x7f4ee40a4c50 2024-09-16T05:37:23.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.839+0000 7f4ed9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:23.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.839+0000 7f4ed9ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:23.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.843+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4ec8036930 legacy=0x7f4ec8038df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.843+0000 7f4eeb932640 10 monclient: shutdown 2024-09-16T05:37:23.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.843+0000 7f4eeb932640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:23.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.843+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 >> v1:172.21.15.6:6790/0 conn(0x7f4ee40a4c50 legacy=0x7f4ee41d3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.843+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 shutdown_connections 2024-09-16T05:37:23.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.843+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 >> 172.21.15.63:0/2194903525 conn(0x7f4ee40a5da0 msgr2=0x7f4ee40a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:23.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.843+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 shutdown_connections 2024-09-16T05:37:23.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.843+0000 7f4eeb932640 1 -- 172.21.15.63:0/2194903525 wait complete. 2024-09-16T05:37:23.861 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:23.861 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c host2 2024-09-16T05:37:23.861 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 2 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient: get_monmap_and_config 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient: build_initial_monmap 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient: monmap: 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:23.981470+0000 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:23.981470+0000 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:23.982 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:23.983 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:23.983 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 1 Processor -- start 2024-09-16T05:37:23.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 1 -- start start 2024-09-16T05:37:23.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient: init 2024-09-16T05:37:23.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:23.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:23.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient(hunting): picked mon.noname-b con 0x7f81e00a3ba0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:23.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient(hunting): picked mon.noname-c con 0x7f81e00a1290 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:23.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient(hunting): picked mon.noname-a con 0x7f81e00a5170 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:23.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e00761f0 con 0x7f81e00a5170 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e009ff60 con 0x7f81e00a1290 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e00a0130 con 0x7f81e00a3ba0 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e782e640 10 monclient(hunting): authenticate will time out at 1100.758672s 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e4dc9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f81e00a1290 legacy=0x7f81e013be90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e5dcb640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81e00a5170 0x7f81e013f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40642/0 (socket says 172.21.15.63:40642) 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e55ca640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f81e00a3ba0 0x7f81e00a3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48090/0 (socket says 172.21.15.63:48090) 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81e5dcb640 1 -- 172.21.15.63:0/4212541899 learned_addr learned my addr 172.21.15.63:0/4212541899 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1684235462 0 0) 0x7f81e00a0130 con 0x7f81e00a3ba0 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 10 monclient(hunting): my global_id is 5111 2024-09-16T05:37:23.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81cc003530 con 0x7f81e00a3ba0 2024-09-16T05:37:23.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2145574006 0 0) 0x7f81e00761f0 con 0x7f81e00a5170 2024-09-16T05:37:23.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:23.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:23.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:23.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 10 monclient(hunting): my global_id is 5124 2024-09-16T05:37:23.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.979+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81e00a0130 con 0x7f81e00a5170 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2627798946 0 0) 0x7f81e00a0130 con 0x7f81e00a5170 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 >> v1:172.21.15.6:6790/0 conn(0x7f81e00a3ba0 legacy=0x7f81e00a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 >> v1:172.21.15.63:6789/0 conn(0x7f81e00a1290 legacy=0x7f81e013be90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 1 monclient: found mon.noname-a 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e00a4690 con 0x7f81e00a5170 2024-09-16T05:37:23.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: _finish_auth 0 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 5 monclient: authenticate success, global_id 5124 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f81d8001830 con 0x7f81e00a5170 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: dump: 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.995 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81d8001a90 con 0x7f81e00a5170 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 1 -- 172.21.15.63:0/4212541899 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f81d8001ee0 con 0x7f81e00a5170 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d7fff640 10 monclient: dump: 2024-09-16T05:37:23.996 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 4 set_mon_vals no callback set 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient: shutdown 2024-09-16T05:37:23.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- 172.21.15.63:0/4212541899 >> v1:172.21.15.6:6789/0 conn(0x7f81e00a5170 legacy=0x7f81e013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- 172.21.15.63:0/4212541899 shutdown_connections 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- 172.21.15.63:0/4212541899 >> 172.21.15.63:0/4212541899 conn(0x7f81e0075470 msgr2=0x7f81e0075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- 172.21.15.63:0/4212541899 shutdown_connections 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- 172.21.15.63:0/4212541899 wait complete. 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient: build_initial_monmap 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient: monmap: 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:23.998 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 Processor -- start 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- start start 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient: init 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient(hunting): picked mon.noname-a con 0x7f81e00a1290 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient(hunting): picked mon.noname-c con 0x7f81e00a3ba0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient(hunting): picked mon.noname-b con 0x7f81e00a5170 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e01271c0 con 0x7f81e00a1290 2024-09-16T05:37:23.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e00a0500 con 0x7f81e00a5170 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e0126ee0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e782e640 10 monclient(hunting): authenticate will time out at 1100.762975s 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e5dcb640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f81e00a5170 legacy=0x7f81e01d3790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e55ca640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81e00a1290 0x7f81e0136ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40652/0 (socket says 172.21.15.63:40652) 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e4dc9640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f81e00a3ba0 0x7f81e01d0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48106/0 (socket says 172.21.15.63:48106) 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81e55ca640 1 -- 172.21.15.63:0/1785741812 learned_addr learned my addr 172.21.15.63:0/1785741812 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816989471 0 0) 0x7f81e01271c0 con 0x7f81e00a1290 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 10 monclient(hunting): my global_id is 5127 2024-09-16T05:37:24.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81c4003530 con 0x7f81e00a1290 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3641980374 0 0) 0x7f81e0126ee0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 10 monclient(hunting): my global_id is 5114 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.983+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81e01271c0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3500582685 0 0) 0x7f81e01271c0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81e0126ee0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1163004817 0 0) 0x7f81c4003530 con 0x7f81e00a1290 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81e01271c0 con 0x7f81e00a1290 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f81d00040d0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:24.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:24.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f81dc001a40 con 0x7f81e00a1290 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.003 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1670375739 0 0) 0x7f81e0126ee0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 >> v1:172.21.15.63:6789/0 conn(0x7f81e00a5170 legacy=0x7f81e01d3790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 >> v1:172.21.15.6:6789/0 conn(0x7f81e00a1290 legacy=0x7f81e0136ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 monclient: found mon.c 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e01371c0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 5 monclient: authenticate success, global_id 5114 2024-09-16T05:37:24.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 10 monclient: _renew_subs 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f81e01cbbc0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 10 monclient: _renew_subs 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f81e0125ee0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81d00045e0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f81d0004930 con 0x7f81e00a3ba0 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d77fe640 4 set_mon_vals no callback set 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 10 monclient: dump: 2024-09-16T05:37:24.005 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c0005090 con 0x7f81e00a3ba0 2024-09-16T05:37:24.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f81d0004b90 con 0x7f81e00a3ba0 2024-09-16T05:37:24.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(77..77 src has 1..77) ==== 5457+0+0 (unknown 1984182297 0 0) 0x7f81d0044790 con 0x7f81e00a3ba0 2024-09-16T05:37:24.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.987+0000 7f81e55ca640 10 monclient: get_auth_request con 0x7f81c4036aa0 auth_method 0 2024-09-16T05:37:24.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.999+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f81d0015250 con 0x7f81e00a3ba0 2024-09-16T05:37:24.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.999+0000 7f81d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:23.999+0000 7f81d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:24.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.163+0000 7f81e782e640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:24.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.163+0000 7f81e782e640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:24.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.163+0000 7f81e782e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.163+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f81c0005b90 con 0x7f81e00a3ba0 2024-09-16T05:37:24.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.163+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1474 (unknown 2226722781 0 2636823495) 0x7f81d0018f00 con 0x7f81e00a3ba0 2024-09-16T05:37:24.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.163+0000 7f81d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:24.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.163+0000 7f81d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:24.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.259+0000 7f81e782e640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:24.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.259+0000 7f81e782e640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:24.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.259+0000 7f81e782e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.259+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f81c00008e0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81d5ffb640 1 -- 172.21.15.63:0/1785741812 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1474 (unknown 2226722781 0 2636823495) 0x7f81d001d7c0 con 0x7f81e00a3ba0 2024-09-16T05:37:24.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81d5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:24.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81d5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:24.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 >> v1:172.21.15.63:6812/3643207211 conn(0x7f81c4036aa0 legacy=0x7f81c4038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81e782e640 10 monclient: shutdown 2024-09-16T05:37:24.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81e782e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:24.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 >> v1:172.21.15.6:6790/0 conn(0x7f81e00a3ba0 legacy=0x7f81e01d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 shutdown_connections 2024-09-16T05:37:24.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 >> 172.21.15.63:0/1785741812 conn(0x7f81e0075470 msgr2=0x7f81e00a78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:24.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 shutdown_connections 2024-09-16T05:37:24.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.263+0000 7f81e782e640 1 -- 172.21.15.63:0/1785741812 wait complete. 2024-09-16T05:37:24.283 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush rm host1 foo 2024-09-16T05:37:24.284 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:24.284 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm host1 foo 2024-09-16T05:37:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.399+0000 7fd36d6b9640 10 monclient: get_monmap_and_config 2024-09-16T05:37:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.399+0000 7fd36d6b9640 10 monclient: build_initial_monmap 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.399+0000 7fd36d6b9640 10 monclient: monmap: 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:24.404136+0000 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:24.404136+0000 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:24.405 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 Processor -- start 2024-09-16T05:37:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- start start 2024-09-16T05:37:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient: init 2024-09-16T05:37:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient(hunting): picked mon.noname-a con 0x7fd36809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient(hunting): picked mon.noname-b con 0x7fd3680a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient(hunting): picked mon.noname-c con 0x7fd36809b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd368074200 con 0x7fd36809ac90 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd36809f900 con 0x7fd36809b780 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd36809e870 con 0x7fd3680a0280 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient(hunting): authenticate will time out at 1101.181055s 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3677fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd36809b780 legacy=0x7fd36813cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd35ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd3680a0280 0x7fd3681398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55342/0 (socket says 172.21.15.63:55342) 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd366ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd36809ac90 0x7fd36809b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42808/0 (socket says 172.21.15.63:42808) 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd35ffff640 1 -- 172.21.15.63:0/511199519 learned_addr learned my addr 172.21.15.63:0/511199519 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896209199 0 0) 0x7fd36809e870 con 0x7fd3680a0280 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient(hunting): my global_id is 5117 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd354003530 con 0x7fd3680a0280 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 334963998 0 0) 0x7fd368074200 con 0x7fd36809ac90 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient(hunting): my global_id is 5130 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd36809e870 con 0x7fd36809ac90 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4093833997 0 0) 0x7fd354003530 con 0x7fd3680a0280 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 >> v1:172.21.15.63:6789/0 conn(0x7fd36809b780 legacy=0x7fd36813cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 >> v1:172.21.15.6:6789/0 conn(0x7fd36809ac90 legacy=0x7fd36809b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 monclient: found mon.noname-b 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd368141730 con 0x7fd3680a0280 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd350001620 con 0x7fd3680a0280 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: dump: 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 5 monclient: authenticate success, global_id 5117 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:24.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd350001880 con 0x7fd3680a0280 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 1 -- 172.21.15.63:0/511199519 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd350001bd0 con 0x7fd3680a0280 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd3667fc640 10 monclient: dump: 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.413 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 4 set_mon_vals no callback set 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient: shutdown 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- 172.21.15.63:0/511199519 >> v1:172.21.15.6:6790/0 conn(0x7fd3680a0280 legacy=0x7fd3681398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- 172.21.15.63:0/511199519 shutdown_connections 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- 172.21.15.63:0/511199519 >> 172.21.15.63:0/511199519 conn(0x7fd3680a5290 msgr2=0x7fd3680a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- 172.21.15.63:0/511199519 shutdown_connections 2024-09-16T05:37:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 1 -- 172.21.15.63:0/511199519 wait complete. 2024-09-16T05:37:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient: build_initial_monmap 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.403+0000 7fd36d6b9640 10 monclient: monmap: 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 1 Processor -- start 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 1 -- start start 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: init 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient(hunting): picked mon.noname-b con 0x7fd36809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient(hunting): picked mon.noname-a con 0x7fd36809b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3680a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd36809d250 con 0x7fd36809b780 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd366ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd36809ac90 legacy=0x7fd36809c430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd36813fac0 con 0x7fd36809ac90 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd36813fda0 con 0x7fd3680a0280 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient(hunting): authenticate will time out at 1101.184835s 2024-09-16T05:37:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd3677fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd3680a0280 0x7fd3681d38c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55354/0 (socket says 172.21.15.63:55354) 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd36809b780 0x7fd36809cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42824/0 (socket says 172.21.15.63:42824) 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd3677fe640 1 -- 172.21.15.63:0/1988271739 learned_addr learned my addr 172.21.15.63:0/1988271739 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844190496 0 0) 0x7fd36813fda0 con 0x7fd3680a0280 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): my global_id is 5120 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd348003530 con 0x7fd3680a0280 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2548059133 0 0) 0x7fd36809d250 con 0x7fd36809b780 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): my global_id is 5133 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd36813fda0 con 0x7fd36809b780 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2430159411 0 0) 0x7fd348003530 con 0x7fd3680a0280 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd36809d250 con 0x7fd3680a0280 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd3600043c0 con 0x7fd3680a0280 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): dump: 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1349600634 0 0) 0x7fd36813fda0 con 0x7fd36809b780 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd348003530 con 0x7fd36809b780 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd350001520 con 0x7fd36809b780 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient(hunting): dump: 2024-09-16T05:37:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1673122395 0 0) 0x7fd36809d250 con 0x7fd3680a0280 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 >> v1:172.21.15.63:6789/0 conn(0x7fd36809ac90 legacy=0x7fd36809c430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 >> v1:172.21.15.6:6789/0 conn(0x7fd36809b780 legacy=0x7fd36809cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 monclient: found mon.c 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd368140080 con 0x7fd3680a0280 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 5 monclient: authenticate success, global_id 5120 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: _renew_subs 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd368140780 con 0x7fd3680a0280 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: _renew_subs 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd36813edc0 con 0x7fd3680a0280 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd360004620 con 0x7fd3680a0280 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd360004970 con 0x7fd3680a0280 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd365ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd35f7fe640 10 monclient: dump: 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.407+0000 7fd36d6b9640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd34c005090 con 0x7fd3680a0280 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.411+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd360004b90 con 0x7fd3680a0280 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.411+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(77..77 src has 1..77) ==== 5457+0+0 (unknown 1984182297 0 0) 0x7fd360044ae0 con 0x7fd3680a0280 2024-09-16T05:37:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.411+0000 7fd366ffd640 10 monclient: get_auth_request con 0x7fd348036a70 auth_method 0 2024-09-16T05:37:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.423+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd3600155a0 con 0x7fd3680a0280 2024-09-16T05:37:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.423+0000 7fd35f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.423+0000 7fd35f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:24.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.583+0000 7fd36d6b9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:24.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.583+0000 7fd36d6b9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:24.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.583+0000 7fd36d6b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.583+0000 7fd36d6b9640 1 -- 172.21.15.63:0/1988271739 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"} v 0) -- 0x7fd34c005b90 con 0x7fd3680a0280 2024-09-16T05:37:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.587+0000 7fd35f7fe640 1 -- 172.21.15.63:0/1988271739 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}]=-39 (39) Directory not empty v77) ==== 120+0+0 (unknown 3104010711 0 0) 0x7fd360019250 con 0x7fd3680a0280 2024-09-16T05:37:24.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.587+0000 7fd35f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:24.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.587+0000 7fd35f7fe640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T05:37:24.590 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T05:37:24.595 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:24.595 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush unlink host1 foo 2024-09-16T05:37:24.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.711+0000 7f933b7fc640 10 monclient: get_monmap_and_config 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.711+0000 7f933b7fc640 10 monclient: build_initial_monmap 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.711+0000 7f933b7fc640 10 monclient: monmap: 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:24.715206+0000 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:24.715206+0000 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:24.716 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:24.717 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.711+0000 7f933b7fc640 1 Processor -- start 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.711+0000 7f933b7fc640 1 -- start start 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.711+0000 7f933b7fc640 10 monclient: init 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f933409ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f93340a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f933409b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:24.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9334074200 con 0x7f933409b780 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933409f900 con 0x7f933409ac90 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933409e870 con 0x7f93340a0280 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient(hunting): authenticate will time out at 1101.492154s 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f9339598640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f933409ac90 legacy=0x7f933409b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f9339d99640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f933409b780 0x7f933413cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42834/0 (socket says 172.21.15.63:42834) 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f9338d97640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f93340a0280 0x7f93341398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55362/0 (socket says 172.21.15.63:55362) 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f9339d99640 1 -- 172.21.15.63:0/965349034 learned_addr learned my addr 172.21.15.63:0/965349034 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050609584 0 0) 0x7f933409e870 con 0x7f93340a0280 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient(hunting): my global_id is 5123 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9320003530 con 0x7f93340a0280 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4001488278 0 0) 0x7f9334074200 con 0x7f933409b780 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient(hunting): my global_id is 5136 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f933409e870 con 0x7f933409b780 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3902776251 0 0) 0x7f9320003530 con 0x7f93340a0280 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 >> v1:172.21.15.63:6789/0 conn(0x7f933409ac90 legacy=0x7f933409b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 >> v1:172.21.15.6:6789/0 conn(0x7f933409b780 legacy=0x7f933413cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 monclient: found mon.noname-b 2024-09-16T05:37:24.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9334141730 con 0x7f93340a0280 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: _finish_auth 0 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9324001550 con 0x7f93340a0280 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: dump: 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.722 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 5 monclient: authenticate success, global_id 5123 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93240017b0 con 0x7f93340a0280 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 1 -- 172.21.15.63:0/965349034 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9324001b00 con 0x7f93340a0280 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:24.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f932bfff640 10 monclient: dump: 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:24.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 4 set_mon_vals no callback set 2024-09-16T05:37:24.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient: shutdown 2024-09-16T05:37:24.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:24.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 1 -- 172.21.15.63:0/965349034 >> v1:172.21.15.6:6790/0 conn(0x7f93340a0280 legacy=0x7f93341398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 1 -- 172.21.15.63:0/965349034 shutdown_connections 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 1 -- 172.21.15.63:0/965349034 >> 172.21.15.63:0/965349034 conn(0x7f93340a5290 msgr2=0x7f93340a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 1 -- 172.21.15.63:0/965349034 shutdown_connections 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 1 -- 172.21.15.63:0/965349034 wait complete. 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient: build_initial_monmap 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.715+0000 7f933b7fc640 10 monclient: monmap: 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:24.729 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 1 Processor -- start 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 1 -- start start 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: init 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f933409ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f933409b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f93340a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9339598640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f933409ac90 legacy=0x7f93341c98a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:24.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933413e0f0 con 0x7f933409b780 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933406df90 con 0x7f933409ac90 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933409c0f0 con 0x7f93340a0280 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient(hunting): authenticate will time out at 1101.497009s 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9339d99640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f93340a0280 0x7f93341d1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55364/0 (socket says 172.21.15.63:55364) 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9338d97640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f933409b780 0x7f93341ce880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42850/0 (socket says 172.21.15.63:42850) 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9339d99640 1 -- 172.21.15.63:0/2813591982 learned_addr learned my addr 172.21.15.63:0/2813591982 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3363232554 0 0) 0x7f933409c0f0 con 0x7f93340a0280 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): my global_id is 5126 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9318003530 con 0x7f93340a0280 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343182191 0 0) 0x7f933413e0f0 con 0x7f933409b780 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): my global_id is 5139 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f933409c0f0 con 0x7f933409b780 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4234619711 0 0) 0x7f9318003530 con 0x7f93340a0280 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f933413e0f0 con 0x7f93340a0280 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f932c001c40 con 0x7f93340a0280 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:24.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 304563684 0 0) 0x7f933409c0f0 con 0x7f933409b780 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9318003530 con 0x7f933409b780 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9324001450 con 0x7f933409b780 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient(hunting): dump: 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.734 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1824088274 0 0) 0x7f933413e0f0 con 0x7f93340a0280 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 >> v1:172.21.15.63:6789/0 conn(0x7f933409ac90 legacy=0x7f93341c98a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 >> v1:172.21.15.6:6789/0 conn(0x7f933409b780 legacy=0x7f93341ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 monclient: found mon.c 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933413be00 con 0x7f93340a0280 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:24.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 5 monclient: authenticate success, global_id 5126 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: _renew_subs 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f933413de10 con 0x7f93340a0280 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: _renew_subs 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f933413cb80 con 0x7f93340a0280 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f932c004b70 con 0x7f93340a0280 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f932c004ec0 con 0x7f93340a0280 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f9329ffb640 10 monclient: dump: 2024-09-16T05:37:24.736 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f932b7fe640 4 set_mon_vals no callback set 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.719+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9314005090 con 0x7f93340a0280 2024-09-16T05:37:24.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.723+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f932c00fc60 con 0x7f93340a0280 2024-09-16T05:37:24.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.723+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(77..77 src has 1..77) ==== 5457+0+0 (unknown 1984182297 0 0) 0x7f932c044b40 con 0x7f93340a0280 2024-09-16T05:37:24.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.723+0000 7f9339598640 10 monclient: get_auth_request con 0x7f9318036a20 auth_method 0 2024-09-16T05:37:24.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.735+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f932c015660 con 0x7f93340a0280 2024-09-16T05:37:24.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.735+0000 7f9329ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:24.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.735+0000 7f9329ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:24.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.903+0000 7f933b7fc640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:24.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.903+0000 7f933b7fc640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:24.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.903+0000 7f933b7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:24.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:24.903+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) -- 0x7f9314005b90 con 0x7f93340a0280 2024-09-16T05:37:25.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.059+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 removed item id -11 name 'host1' from crush map v78) ==== 147+0+0 (unknown 3439428393 0 0) 0x7f932c014f60 con 0x7f93340a0280 2024-09-16T05:37:25.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.059+0000 7f9329ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.059+0000 7f9329ffb640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T05:37:25.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.163+0000 7f933b7fc640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.163+0000 7f933b7fc640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.163+0000 7f933b7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:25.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.163+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) -- 0x7f93140010a0 con 0x7f93340a0280 2024-09-16T05:37:25.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.163+0000 7f9329ffb640 1 -- 172.21.15.63:0/2813591982 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v78) ==== 141+0+0 (unknown 4243608097 0 0) 0x7f932c01db70 con 0x7f93340a0280 2024-09-16T05:37:25.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.163+0000 7f9329ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.163+0000 7f9329ffb640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T05:37:25.168 INFO:tasks.workunit.client.0.smithi063.stderr:item -11 does not appear in that position 2024-09-16T05:37:25.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.167+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9318036a20 legacy=0x7f9318038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.167+0000 7f933b7fc640 10 monclient: shutdown 2024-09-16T05:37:25.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.167+0000 7f933b7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:25.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.167+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 >> v1:172.21.15.6:6790/0 conn(0x7f93340a0280 legacy=0x7f93341d1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.167+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 shutdown_connections 2024-09-16T05:37:25.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.167+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 >> 172.21.15.63:0/2813591982 conn(0x7f93340a5290 msgr2=0x7f93340a44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:25.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.167+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 shutdown_connections 2024-09-16T05:37:25.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.167+0000 7f933b7fc640 1 -- 172.21.15.63:0/2813591982 wait complete. 2024-09-16T05:37:25.184 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush unlink host1 foo 2024-09-16T05:37:25.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.299+0000 7ff0c41e2640 10 monclient: get_monmap_and_config 2024-09-16T05:37:25.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.299+0000 7ff0c41e2640 10 monclient: build_initial_monmap 2024-09-16T05:37:25.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.299+0000 7ff0c41e2640 10 monclient: monmap: 2024-09-16T05:37:25.305 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:25.305 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.305 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:25.304075+0000 2024-09-16T05:37:25.306 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:25.304075+0000 2024-09-16T05:37:25.306 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:25.306 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.306 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:25.306 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:25.306 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:25.306 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 Processor -- start 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- start start 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient: init 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient(hunting): picked mon.noname-b con 0x7ff0bc09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient(hunting): picked mon.noname-a con 0x7ff0bc09b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient(hunting): picked mon.noname-c con 0x7ff0bc139fc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc0a14f0 con 0x7ff0bc09b780 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc09c680 con 0x7ff0bc139fc0 2024-09-16T05:37:25.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc09e930 con 0x7ff0bc09ac90 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient(hunting): authenticate will time out at 1102.081147s 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c277f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff0bc139fc0 legacy=0x7ff0bc13c3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c1f7e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff0bc09ac90 0x7ff0bc09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55366/0 (socket says 172.21.15.63:55366) 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c177d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff0bc09b780 0x7ff0bc1388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42852/0 (socket says 172.21.15.63:42852) 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c1f7e640 1 -- 172.21.15.63:0/2242149839 learned_addr learned my addr 172.21.15.63:0/2242149839 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2587026612 0 0) 0x7ff0bc09e930 con 0x7ff0bc09ac90 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient(hunting): _negotiate 2024-09-16T05:37:25.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient(hunting): my global_id is 5129 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0a8003530 con 0x7ff0bc09ac90 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 342056296 0 0) 0x7ff0bc0a14f0 con 0x7ff0bc09b780 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient(hunting): _negotiate 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient(hunting): my global_id is 5142 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0bc09e930 con 0x7ff0bc09b780 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4080336683 0 0) 0x7ff0bc09e930 con 0x7ff0bc09b780 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 >> v1:172.21.15.6:6790/0 conn(0x7ff0bc09ac90 legacy=0x7ff0bc09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 >> v1:172.21.15.63:6789/0 conn(0x7ff0bc139fc0 legacy=0x7ff0bc13c3b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: _finish_hunting 0 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 monclient: found mon.noname-a 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0bc09c400 con 0x7ff0bc09b780 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: _finish_auth 0 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: _check_auth_tickets 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0ac001600 con 0x7ff0bc09b780 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:25.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: dump: 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.315 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 5 monclient: authenticate success, global_id 5142 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0ac001870 con 0x7ff0bc09b780 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 1 -- 172.21.15.63:0/2242149839 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0ac001c80 con 0x7ff0bc09b780 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c0f7c640 10 monclient: dump: 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.316 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.317 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.317 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.317 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.317 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.317 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.317 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.317 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 4 set_mon_vals no callback set 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 10 monclient: shutdown 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2242149839 >> v1:172.21.15.6:6789/0 conn(0x7ff0bc09b780 legacy=0x7ff0bc1388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2242149839 shutdown_connections 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2242149839 >> 172.21.15.63:0/2242149839 conn(0x7ff0bc0a5290 msgr2=0x7ff0bc0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2242149839 shutdown_connections 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.303+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2242149839 wait complete. 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: build_initial_monmap 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: monmap: 2024-09-16T05:37:25.318 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 1 Processor -- start 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 1 -- start start 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: init 2024-09-16T05:37:25.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient(hunting): picked mon.noname-c con 0x7ff0bc09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient(hunting): picked mon.noname-b con 0x7ff0bc09b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient(hunting): picked mon.noname-a con 0x7ff0bc139fc0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc134990 con 0x7ff0bc139fc0 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc070e40 con 0x7ff0bc09b780 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc070bc0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c177d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff0bc09b780 legacy=0x7ff0bc06f000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient(hunting): authenticate will time out at 1102.085691s 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c1f7e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff0bc09ac90 0x7ff0bc06e8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55378/0 (socket says 172.21.15.63:55378) 2024-09-16T05:37:25.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c1f7e640 1 -- 172.21.15.63:0/2042436598 learned_addr learned my addr 172.21.15.63:0/2042436598 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393970975 0 0) 0x7ff0bc070bc0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): my global_id is 5132 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0a0003530 con 0x7ff0bc09ac90 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142644850 0 0) 0x7ff0bc134990 con 0x7ff0bc139fc0 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): my global_id is 5145 2024-09-16T05:37:25.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0bc070bc0 con 0x7ff0bc139fc0 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2114393970 0 0) 0x7ff0a0003530 con 0x7ff0bc09ac90 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0bc134990 con 0x7ff0bc09ac90 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0b8001900 con 0x7ff0bc09ac90 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): dump: 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.325 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 791689066 0 0) 0x7ff0bc070bc0 con 0x7ff0bc139fc0 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0a0003530 con 0x7ff0bc139fc0 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0b4004280 con 0x7ff0bc139fc0 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient(hunting): dump: 2024-09-16T05:37:25.326 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2473531319 0 0) 0x7ff0bc134990 con 0x7ff0bc09ac90 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 >> v1:172.21.15.63:6789/0 conn(0x7ff0bc09b780 legacy=0x7ff0bc06f000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 >> v1:172.21.15.6:6789/0 conn(0x7ff0bc139fc0 legacy=0x7ff0bc1cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 monclient: found mon.c 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0bc071060 con 0x7ff0bc09ac90 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 5 monclient: authenticate success, global_id 5132 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: _renew_subs 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0bc071d30 con 0x7ff0bc09ac90 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: _renew_subs 2024-09-16T05:37:25.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.307+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0bc06f8e0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0b8001cd0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0b8004c40 con 0x7ff0bc09ac90 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b3fff640 4 set_mon_vals no callback set 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b27fc640 10 monclient: dump: 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.329 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0c41e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0c41e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0c41e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:25.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a4005090 con 0x7ff0bc09ac90 2024-09-16T05:37:25.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff0b800f9e0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(78..78 src has 1..78) ==== 5449+0+0 (unknown 2153714307 0 0) 0x7ff0b8044970 con 0x7ff0bc09ac90 2024-09-16T05:37:25.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.311+0000 7ff0c177d640 10 monclient: get_auth_request con 0x7ff0a0036a20 auth_method 0 2024-09-16T05:37:25.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.323+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff0b80154a0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.323+0000 7ff0b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:25.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.323+0000 7ff0b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:25.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.491+0000 7ff0c41e2640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.491+0000 7ff0c41e2640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.491+0000 7ff0c41e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:25.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.491+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) -- 0x7ff0a4005b90 con 0x7ff0bc09ac90 2024-09-16T05:37:25.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.495+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v78) ==== 141+0+0 (unknown 4243608097 0 0) 0x7ff0b8014da0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.495+0000 7ff0b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.495+0000 7ff0b27fc640 10 monclient: _finish_command 2 = system:0 item -11 does not appear in that position 2024-09-16T05:37:25.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.595+0000 7ff0c41e2640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.595+0000 7ff0c41e2640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.595+0000 7ff0c41e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:25.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.595+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) -- 0x7ff0a40010a0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.599+0000 7ff0b27fc640 1 -- 172.21.15.63:0/2042436598 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v78) ==== 141+0+0 (unknown 4243608097 0 0) 0x7ff0b80190f0 con 0x7ff0bc09ac90 2024-09-16T05:37:25.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.599+0000 7ff0b27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T05:37:25.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.599+0000 7ff0b27fc640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T05:37:25.603 INFO:tasks.workunit.client.0.smithi063.stderr:item -11 does not appear in that position 2024-09-16T05:37:25.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.599+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff0a0036a20 legacy=0x7ff0a0038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.599+0000 7ff0c41e2640 10 monclient: shutdown 2024-09-16T05:37:25.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.599+0000 7ff0c41e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:25.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.599+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 >> v1:172.21.15.6:6790/0 conn(0x7ff0bc09ac90 legacy=0x7ff0bc06e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.603+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 shutdown_connections 2024-09-16T05:37:25.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.603+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 >> 172.21.15.63:0/2042436598 conn(0x7ff0bc0a5290 msgr2=0x7ff0bc0a2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:25.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.603+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 shutdown_connections 2024-09-16T05:37:25.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.603+0000 7ff0c41e2640 1 -- 172.21.15.63:0/2042436598 wait complete. 2024-09-16T05:37:25.621 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:25.621 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c host1 2024-09-16T05:37:25.621 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 1 2024-09-16T05:37:25.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.735+0000 7f6c72ad3640 10 monclient: get_monmap_and_config 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.735+0000 7f6c72ad3640 10 monclient: build_initial_monmap 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.735+0000 7f6c72ad3640 10 monclient: monmap: 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:25.739146+0000 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:25.739146+0000 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:25.740 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:25.741 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:25.741 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.735+0000 7f6c72ad3640 1 Processor -- start 2024-09-16T05:37:25.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.735+0000 7f6c72ad3640 1 -- start start 2024-09-16T05:37:25.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.735+0000 7f6c72ad3640 10 monclient: init 2024-09-16T05:37:25.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:25.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:25.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient(hunting): picked mon.noname-c con 0x7f6c6c09e790 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:25.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient(hunting): picked mon.noname-a con 0x7f6c6c09fd70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:25.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient(hunting): picked mon.noname-b con 0x7f6c6c1394f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c6c09c500 con 0x7f6c6c09fd70 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c7086f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6c6c09e790 legacy=0x7f6c6c09eb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c6c0a1260 con 0x7f6c6c09e790 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c6c09c9c0 con 0x7f6c6c1394f0 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient(hunting): authenticate will time out at 1102.516298s 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6c6c09fd70 0x7f6c6c137dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42876/0 (socket says 172.21.15.63:42876) 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c71070640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6c6c1394f0 0x7f6c6c13b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55382/0 (socket says 172.21.15.63:55382) 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6bfff640 1 -- 172.21.15.63:0/601872086 learned_addr learned my addr 172.21.15.63:0/601872086 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2485179000 0 0) 0x7f6c6c09c9c0 con 0x7f6c6c1394f0 2024-09-16T05:37:25.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient(hunting): my global_id is 5135 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c54003450 con 0x7f6c6c1394f0 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2914558467 0 0) 0x7f6c6c09c500 con 0x7f6c6c09fd70 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient(hunting): my global_id is 5148 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c6c09c9c0 con 0x7f6c6c09fd70 2024-09-16T05:37:25.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1496644149 0 0) 0x7f6c54003450 con 0x7f6c6c1394f0 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 >> v1:172.21.15.63:6789/0 conn(0x7f6c6c09e790 legacy=0x7f6c6c09eb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 >> v1:172.21.15.6:6789/0 conn(0x7f6c6c09fd70 legacy=0x7f6c6c137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6c0a0180 con 0x7f6c6c1394f0 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:25.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c640019a0 con 0x7f6c6c1394f0 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: dump: 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.748 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 5 monclient: authenticate success, global_id 5135 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c64001c00 con 0x7f6c6c1394f0 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 1 -- 172.21.15.63:0/601872086 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c64001f50 con 0x7f6c6c1394f0 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c6b7fe640 10 monclient: dump: 2024-09-16T05:37:25.749 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 4 set_mon_vals no callback set 2024-09-16T05:37:25.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient: shutdown 2024-09-16T05:37:25.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:25.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 1 -- 172.21.15.63:0/601872086 >> v1:172.21.15.6:6790/0 conn(0x7f6c6c1394f0 legacy=0x7f6c6c13b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 1 -- 172.21.15.63:0/601872086 shutdown_connections 2024-09-16T05:37:25.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 1 -- 172.21.15.63:0/601872086 >> 172.21.15.63:0/601872086 conn(0x7f6c6c0a1b60 msgr2=0x7f6c6c0a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:25.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 1 -- 172.21.15.63:0/601872086 shutdown_connections 2024-09-16T05:37:25.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 1 -- 172.21.15.63:0/601872086 wait complete. 2024-09-16T05:37:25.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient: build_initial_monmap 2024-09-16T05:37:25.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.739+0000 7f6c72ad3640 10 monclient: monmap: 2024-09-16T05:37:25.755 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:25.755 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:25.755 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:25.755 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:25.755 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:25.755 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 1 Processor -- start 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 1 -- start start 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient: init 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:25.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient(hunting): picked mon.noname-b con 0x7f6c6c09e790 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient(hunting): picked mon.noname-c con 0x7f6c6c09fd70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient(hunting): picked mon.noname-a con 0x7f6c6c1394f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c6c138d60 con 0x7f6c6c1394f0 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c7086f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6c6c09e790 legacy=0x7f6c6c133ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c6c1349c0 con 0x7f6c6c09e790 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c6c136830 con 0x7f6c6c09fd70 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient(hunting): authenticate will time out at 1102.521030s 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c6bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6c6c09fd70 0x7f6c6c1342b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55398/0 (socket says 172.21.15.63:55398) 2024-09-16T05:37:25.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c6bfff640 1 -- 172.21.15.63:0/1469308611 learned_addr learned my addr 172.21.15.63:0/1469308611 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c71070640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6c6c1394f0 0x7f6c6c1cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42886/0 (socket says 172.21.15.63:42886) 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3412842724 0 0) 0x7f6c6c0a0180 con 0x7f6c6c1394f0 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): my global_id is 5151 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c58003530 con 0x7f6c6c1394f0 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1496331719 0 0) 0x7f6c6c136830 con 0x7f6c6c09fd70 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:25.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): my global_id is 5138 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c6c0a0180 con 0x7f6c6c09fd70 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 19566951 0 0) 0x7f6c58003530 con 0x7f6c6c1394f0 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c6c136830 con 0x7f6c6c1394f0 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c640018a0 con 0x7f6c6c1394f0 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): dump: 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.759 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1788450422 0 0) 0x7f6c6c0a0180 con 0x7f6c6c09fd70 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c58003530 con 0x7f6c6c09fd70 2024-09-16T05:37:25.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c600016b0 con 0x7f6c6c09fd70 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient(hunting): dump: 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.761 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 590967567 0 0) 0x7f6c6c136830 con 0x7f6c6c1394f0 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 >> v1:172.21.15.6:6790/0 conn(0x7f6c6c09fd70 legacy=0x7f6c6c1342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 >> v1:172.21.15.63:6789/0 conn(0x7f6c6c09e790 legacy=0x7f6c6c133ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 monclient: found mon.a 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6c1374f0 con 0x7f6c6c1394f0 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:25.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 5 monclient: authenticate success, global_id 5151 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient: _renew_subs 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c6c135940 con 0x7f6c6c1394f0 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient: _renew_subs 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c6c1300e0 con 0x7f6c6c1394f0 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c64001b60 con 0x7f6c6c1394f0 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:25.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c64004fc0 con 0x7f6c6c1394f0 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c6affd640 4 set_mon_vals no callback set 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.743+0000 7f6c697fa640 10 monclient: dump: 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:25.764 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.747+0000 7f6c72ad3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.747+0000 7f6c72ad3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.747+0000 7f6c72ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.747+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c4c005090 con 0x7f6c6c1394f0 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.747+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6c6400fd60 con 0x7f6c6c1394f0 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.747+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(78..78 src has 1..78) ==== 5449+0+0 (unknown 2153714307 0 0) 0x7f6c64044c20 con 0x7f6c6c1394f0 2024-09-16T05:37:25.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.747+0000 7f6c7086f640 10 monclient: get_auth_request con 0x7f6c58036ac0 auth_method 0 2024-09-16T05:37:25.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.759+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6c6c138d60 con 0x7f6c6c1394f0 2024-09-16T05:37:25.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.759+0000 7f6c697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:25.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.759+0000 7f6c697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:25.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.919+0000 7f6c72ad3640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:25.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.923+0000 7f6c72ad3640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:25.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.923+0000 7f6c72ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:25.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.923+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f6c4c005b90 con 0x7f6c6c1394f0 2024-09-16T05:37:25.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.923+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) ==== 56+0+1340 (unknown 596248356 0 3168403622) 0x7f6c64015810 con 0x7f6c6c1394f0 2024-09-16T05:37:25.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.923+0000 7f6c697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:25.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:25.923+0000 7f6c697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:26.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.015+0000 7f6c72ad3640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:26.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.015+0000 7f6c72ad3640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:26.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.015+0000 7f6c72ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.015+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f6c4c005d60 con 0x7f6c6c1394f0 2024-09-16T05:37:26.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c697fa640 1 -- 172.21.15.63:0/1469308611 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) ==== 56+0+1340 (unknown 3183532764 0 3168403622) 0x7f6c64015110 con 0x7f6c6c1394f0 2024-09-16T05:37:26.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c697fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:26.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c697fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:26.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6c58036ac0 legacy=0x7f6c58038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c72ad3640 10 monclient: shutdown 2024-09-16T05:37:26.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c72ad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:26.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 >> v1:172.21.15.6:6789/0 conn(0x7f6c6c1394f0 legacy=0x7f6c6c1cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 shutdown_connections 2024-09-16T05:37:26.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 >> 172.21.15.63:0/1469308611 conn(0x7f6c6c0a1b60 msgr2=0x7f6c6c0a38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:26.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 shutdown_connections 2024-09-16T05:37:26.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.019+0000 7f6c72ad3640 1 -- 172.21.15.63:0/1469308611 wait complete. 2024-09-16T05:37:26.037 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush rm foo 2024-09-16T05:37:26.037 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:26.037 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm foo 2024-09-16T05:37:26.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.143+0000 7f72fc0ba640 10 monclient: get_monmap_and_config 2024-09-16T05:37:26.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.143+0000 7f72fc0ba640 10 monclient: build_initial_monmap 2024-09-16T05:37:26.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.143+0000 7f72fc0ba640 10 monclient: monmap: 2024-09-16T05:37:26.148 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:26.148 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.148 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:26.147578+0000 2024-09-16T05:37:26.148 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:26.147578+0000 2024-09-16T05:37:26.148 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:26.149 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.149 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:26.149 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:26.149 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:26.149 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 Processor -- start 2024-09-16T05:37:26.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- start start 2024-09-16T05:37:26.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient: init 2024-09-16T05:37:26.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient(hunting): picked mon.noname-b con 0x7f72f40a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient(hunting): picked mon.noname-a con 0x7f72f406c5f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient(hunting): picked mon.noname-c con 0x7f72f409d680 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f409ec50 con 0x7f72f406c5f0 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f406c350 con 0x7f72f409d680 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f406ccd0 con 0x7f72f40a1a00 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient(hunting): authenticate will time out at 1102.925106s 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fa657640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f72f409d680 legacy=0x7f72f413d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f9655640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f72f406c5f0 0x7f72f4139a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42888/0 (socket says 172.21.15.63:42888) 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f9655640 1 -- 172.21.15.63:0/3205649074 learned_addr learned my addr 172.21.15.63:0/3205649074 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:26.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f9e56640 1 --1- 172.21.15.63:0/3205649074 >> v1:172.21.15.6:6790/0 conn(0x7f72f40a1a00 0x7f72f40a1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55412/0 (socket says 172.21.15.63:55412) 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4108693414 0 0) 0x7f72e4001050 con 0x7f72f406c5f0 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient(hunting): my global_id is 5154 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72e0003530 con 0x7f72f406c5f0 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2257853985 0 0) 0x7f72f406ccd0 con 0x7f72f40a1a00 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient(hunting): my global_id is 5141 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72e4001050 con 0x7f72f40a1a00 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 180864375 0 0) 0x7f72e0003530 con 0x7f72f406c5f0 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 >> v1:172.21.15.6:6790/0 conn(0x7f72f40a1a00 legacy=0x7f72f40a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 >> v1:172.21.15.63:6789/0 conn(0x7f72f409d680 legacy=0x7f72f413d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: _finish_hunting 0 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 monclient: found mon.noname-a 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f40a24f0 con 0x7f72f406c5f0 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: _finish_auth 0 2024-09-16T05:37:26.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: _check_auth_tickets 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72e4001720 con 0x7f72f406c5f0 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: dump: 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.159 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 5 monclient: authenticate success, global_id 5154 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72e4001980 con 0x7f72f406c5f0 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 1 -- 172.21.15.63:0/3205649074 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72e4001d50 con 0x7f72f406c5f0 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 4 set_mon_vals no callback set 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72f8e54640 10 monclient: dump: 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 10 monclient: shutdown 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- 172.21.15.63:0/3205649074 >> v1:172.21.15.6:6789/0 conn(0x7f72f406c5f0 legacy=0x7f72f4139a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- 172.21.15.63:0/3205649074 shutdown_connections 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- 172.21.15.63:0/3205649074 >> 172.21.15.63:0/3205649074 conn(0x7f72f40a3570 msgr2=0x7f72f4075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- 172.21.15.63:0/3205649074 shutdown_connections 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.147+0000 7f72fc0ba640 1 -- 172.21.15.63:0/3205649074 wait complete. 2024-09-16T05:37:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: build_initial_monmap 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: monmap: 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 1 Processor -- start 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 1 -- start start 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: init 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient(hunting): picked mon.noname-b con 0x7f72f409d680 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient(hunting): picked mon.noname-c con 0x7f72f40a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient(hunting): picked mon.noname-a con 0x7f72f413ca70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f40a4da0 con 0x7f72f413ca70 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f409d1e0 con 0x7f72f409d680 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72f9e56640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f72f409d680 legacy=0x7f72f413bc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f413d760 con 0x7f72f40a1a00 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient(hunting): authenticate will time out at 1102.929162s 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72f9655640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f72f40a1a00 0x7f72f413c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55422/0 (socket says 172.21.15.63:55422) 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72f9655640 1 -- 172.21.15.63:0/2442040980 learned_addr learned my addr 172.21.15.63:0/2442040980 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893698013 0 0) 0x7f72f40a4da0 con 0x7f72f413ca70 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): my global_id is 5157 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72d8003530 con 0x7f72f413ca70 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3364296234 0 0) 0x7f72f413d760 con 0x7f72f40a1a00 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): my global_id is 5144 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72f40a4da0 con 0x7f72f40a1a00 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3516700560 0 0) 0x7f72d8003530 con 0x7f72f413ca70 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72f413d760 con 0x7f72f413ca70 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72ec004340 con 0x7f72f413ca70 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): dump: 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.166 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3483491582 0 0) 0x7f72f40a4da0 con 0x7f72f40a1a00 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72d8003530 con 0x7f72f40a1a00 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72e40016e0 con 0x7f72f40a1a00 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:26.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient(hunting): dump: 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3245824410 0 0) 0x7f72f413d760 con 0x7f72f413ca70 2024-09-16T05:37:26.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 >> v1:172.21.15.6:6790/0 conn(0x7f72f40a1a00 legacy=0x7f72f413c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 >> v1:172.21.15.63:6789/0 conn(0x7f72f409d680 legacy=0x7f72f413bc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 monclient: found mon.a 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f411e0c0 con 0x7f72f413ca70 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 5 monclient: authenticate success, global_id 5157 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: _renew_subs 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72f411e3a0 con 0x7f72f413ca70 2024-09-16T05:37:26.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: _renew_subs 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72fc0ba640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72f41d2eb0 con 0x7f72f413ca70 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72ec0045a0 con 0x7f72f413ca70 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72ec0048f0 con 0x7f72f413ca70 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ebfff640 4 set_mon_vals no callback set 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.151+0000 7f72ea7fc640 10 monclient: dump: 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.155+0000 7f72fc0ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.155+0000 7f72fc0ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.155+0000 7f72fc0ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.155+0000 7f72fc0ba640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72dc005090 con 0x7f72f413ca70 2024-09-16T05:37:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.155+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f72ec004a90 con 0x7f72f413ca70 2024-09-16T05:37:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.155+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(79..79 src has 1..79) ==== 5449+0+0 (unknown 2367597548 0 0) 0x7f72ec044a20 con 0x7f72f413ca70 2024-09-16T05:37:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.155+0000 7f72f9e56640 10 monclient: get_auth_request con 0x7f72d8036a20 auth_method 0 2024-09-16T05:37:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.167+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f72ec0154f0 con 0x7f72f413ca70 2024-09-16T05:37:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.167+0000 7f72ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.167+0000 7f72ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:26.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.331+0000 7f72fc0ba640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:26.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.331+0000 7f72fc0ba640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:26.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.331+0000 7f72fc0ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.331+0000 7f72fc0ba640 1 -- 172.21.15.63:0/2442040980 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7f72dc005380 con 0x7f72f413ca70 2024-09-16T05:37:26.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.335+0000 7f72ea7fc640 1 -- 172.21.15.63:0/2442040980 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-39 (39) Directory not empty v79) ==== 99+0+0 (unknown 730519691 0 0) 0x7f72ec0191a0 con 0x7f72f413ca70 2024-09-16T05:37:26.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.335+0000 7f72ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:26.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.335+0000 7f72ea7fc640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T05:37:26.337 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T05:37:26.342 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:26.342 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush rm bar 2024-09-16T05:37:26.342 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:26.342 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm bar 2024-09-16T05:37:26.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient: get_monmap_and_config 2024-09-16T05:37:26.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient: build_initial_monmap 2024-09-16T05:37:26.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient: monmap: 2024-09-16T05:37:26.451 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:26.451 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.451 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:26.449630+0000 2024-09-16T05:37:26.451 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:26.449630+0000 2024-09-16T05:37:26.451 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:26.452 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.452 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:26.452 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:26.452 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:26.452 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 1 Processor -- start 2024-09-16T05:37:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 1 -- start start 2024-09-16T05:37:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient: init 2024-09-16T05:37:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient(hunting): picked mon.noname-c con 0x7fcfe00a3d80 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:26.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient(hunting): picked mon.noname-b con 0x7fcfe00a4870 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:26.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient(hunting): picked mon.noname-a con 0x7fcfe013d4a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:26.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe00a0880 con 0x7fcfe013d4a0 2024-09-16T05:37:26.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe00a5340 con 0x7fcfe00a3d80 2024-09-16T05:37:26.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe00a10b0 con 0x7fcfe00a4870 2024-09-16T05:37:26.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:26.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe6f18640 10 monclient(hunting): authenticate will time out at 1103.226503s 2024-09-16T05:37:26.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfe4cb4640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcfe00a3d80 legacy=0x7fcfe00a4160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:26.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcfe00a4870 0x7fcfe013bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55428/0 (socket says 172.21.15.63:55428) 2024-09-16T05:37:26.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdffff640 1 -- 172.21.15.63:0/1881700690 learned_addr learned my addr 172.21.15.63:0/1881700690 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:26.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355837165 0 0) 0x7fcfe00a0880 con 0x7fcfe013d4a0 2024-09-16T05:37:26.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 10 monclient(hunting): my global_id is 5160 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfc8003530 con 0x7fcfe013d4a0 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1989979910 0 0) 0x7fcfe00a10b0 con 0x7fcfe00a4870 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 10 monclient(hunting): my global_id is 5147 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfe00a0880 con 0x7fcfe00a4870 2024-09-16T05:37:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1501666929 0 0) 0x7fcfc8003530 con 0x7fcfe013d4a0 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.447+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 >> v1:172.21.15.6:6790/0 conn(0x7fcfe00a4870 legacy=0x7fcfe013bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 >> v1:172.21.15.63:6789/0 conn(0x7fcfe00a3d80 legacy=0x7fcfe00a4160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 1 monclient: found mon.noname-a 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe00a1390 con 0x7fcfe013d4a0 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:26.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 5 monclient: authenticate success, global_id 5160 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfd0001620 con 0x7fcfe013d4a0 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: dump: 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.466 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcfd0001880 con 0x7fcfe013d4a0 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 1 -- 172.21.15.63:0/1881700690 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfd0001c10 con 0x7fcfe013d4a0 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 4 set_mon_vals no callback set 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdf7fe640 10 monclient: dump: 2024-09-16T05:37:26.467 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: shutdown 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:26.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1881700690 >> v1:172.21.15.6:6789/0 conn(0x7fcfe013d4a0 legacy=0x7fcfe013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1881700690 shutdown_connections 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1881700690 >> 172.21.15.63:0/1881700690 conn(0x7fcfe0075470 msgr2=0x7fcfe0075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1881700690 shutdown_connections 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1881700690 wait complete. 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: build_initial_monmap 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: monmap: 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:26.469 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:26.470 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:26.470 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:26.470 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.470 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:26.470 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:26.471 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:26.471 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 Processor -- start 2024-09-16T05:37:26.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- start start 2024-09-16T05:37:26.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: init 2024-09-16T05:37:26.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient(hunting): picked mon.noname-b con 0x7fcfe00a3d80 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient(hunting): picked mon.noname-c con 0x7fcfe00a4870 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient(hunting): picked mon.noname-a con 0x7fcfe013d4a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe0072690 con 0x7fcfe013d4a0 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe0139240 con 0x7fcfe00a3d80 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe00a2e40 con 0x7fcfe00a4870 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient(hunting): authenticate will time out at 1103.230206s 2024-09-16T05:37:26.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe4cb4640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcfe00a3d80 legacy=0x7fcfe0070370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcfe00a4870 0x7fcfe0070a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55432/0 (socket says 172.21.15.63:55432) 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe54b5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcfe013d4a0 0x7fcfe01d3670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42926/0 (socket says 172.21.15.63:42926) 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdffff640 1 -- 172.21.15.63:0/1879585870 learned_addr learned my addr 172.21.15.63:0/1879585870 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 847712886 0 0) 0x7fcfe0072690 con 0x7fcfe013d4a0 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): my global_id is 5163 2024-09-16T05:37:26.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfcc003530 con 0x7fcfe013d4a0 2024-09-16T05:37:26.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955907779 0 0) 0x7fcfe00a2e40 con 0x7fcfe00a4870 2024-09-16T05:37:26.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): my global_id is 5150 2024-09-16T05:37:26.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfe0072690 con 0x7fcfe00a4870 2024-09-16T05:37:26.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4078605652 0 0) 0x7fcfe0072690 con 0x7fcfe00a4870 2024-09-16T05:37:26.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfe00a2e40 con 0x7fcfe00a4870 2024-09-16T05:37:26.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfd4001a20 con 0x7fcfe00a4870 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): dump: 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.478 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.479 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.479 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.479 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.479 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:26.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 807410255 0 0) 0x7fcfcc003530 con 0x7fcfe013d4a0 2024-09-16T05:37:26.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfe0072690 con 0x7fcfe013d4a0 2024-09-16T05:37:26.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfd0001520 con 0x7fcfe013d4a0 2024-09-16T05:37:26.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:26.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:26.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient(hunting): dump: 2024-09-16T05:37:26.481 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.481 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.481 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.481 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.481 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.481 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.482 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.482 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.482 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.482 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 940103912 0 0) 0x7fcfe00a2e40 con 0x7fcfe00a4870 2024-09-16T05:37:26.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 >> v1:172.21.15.63:6789/0 conn(0x7fcfe00a3d80 legacy=0x7fcfe0070370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:26.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 >> v1:172.21.15.6:6789/0 conn(0x7fcfe013d4a0 legacy=0x7fcfe01d3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 monclient: found mon.c 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe013d160 con 0x7fcfe00a4870 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 5 monclient: authenticate success, global_id 5150 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: _renew_subs 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.451+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe00a0570 con 0x7fcfe00a4870 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfe6f18640 10 monclient: _renew_subs 2024-09-16T05:37:26.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfe6f18640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcfe0138f60 con 0x7fcfe00a4870 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcfd4001c80 con 0x7fcfe00a4870 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfd4004c40 con 0x7fcfe00a4870 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdeffd640 4 set_mon_vals no callback set 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdd7fa640 10 monclient: dump: 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.484 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.485 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.485 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.485 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.485 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.485 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.485 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfe6f18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfe6f18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfe6f18640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:26.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfc0005090 con 0x7fcfe00a4870 2024-09-16T05:37:26.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcfd400f9e0 con 0x7fcfe00a4870 2024-09-16T05:37:26.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(79..79 src has 1..79) ==== 5449+0+0 (unknown 2367597548 0 0) 0x7fcfd4044940 con 0x7fcfe00a4870 2024-09-16T05:37:26.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.455+0000 7fcfe4cb4640 10 monclient: get_auth_request con 0x7fcfcc036a70 auth_method 0 2024-09-16T05:37:26.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.467+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fcfd40154d0 con 0x7fcfe00a4870 2024-09-16T05:37:26.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.467+0000 7fcfdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.467+0000 7fcfdd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:26.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.627+0000 7fcfe6f18640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:26.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.627+0000 7fcfe6f18640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:26.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.627+0000 7fcfe6f18640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:26.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.627+0000 7fcfe6f18640 1 -- 172.21.15.63:0/1879585870 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) -- 0x7fcfc0005380 con 0x7fcfe00a4870 2024-09-16T05:37:26.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.631+0000 7fcfdd7fa640 1 -- 172.21.15.63:0/1879585870 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v79) ==== 99+0+0 (unknown 2325897860 0 0) 0x7fcfd4014dd0 con 0x7fcfe00a4870 2024-09-16T05:37:26.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.631+0000 7fcfdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:26.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.631+0000 7fcfdd7fa640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T05:37:26.633 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T05:37:26.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:26.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush unlink host1 bar 2024-09-16T05:37:26.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 10 monclient: get_monmap_and_config 2024-09-16T05:37:26.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 10 monclient: build_initial_monmap 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 10 monclient: monmap: 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:26.747024+0000 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:26.747024+0000 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:26.748 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 1 Processor -- start 2024-09-16T05:37:26.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 1 -- start start 2024-09-16T05:37:26.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 10 monclient: init 2024-09-16T05:37:26.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:26.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 10 monclient(hunting): picked mon.noname-c con 0x7fee1809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.743+0000 7fee1d09b640 10 monclient(hunting): picked mon.noname-a con 0x7fee1809b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient(hunting): picked mon.noname-b con 0x7fee18139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee180a14f0 con 0x7fee1809b780 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee16d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fee1809ac90 legacy=0x7fee1809b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1809c680 con 0x7fee1809ac90 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1809e930 con 0x7fee18139fc0 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient(hunting): authenticate will time out at 1103.524118s 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fee1809b780 0x7fee181388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42934/0 (socket says 172.21.15.63:42934) 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1759e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fee18139fc0 0x7fee1813c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55444/0 (socket says 172.21.15.63:55444) 2024-09-16T05:37:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1659c640 1 -- 172.21.15.63:0/1887859734 learned_addr learned my addr 172.21.15.63:0/1887859734 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 472276507 0 0) 0x7fee1809e930 con 0x7fee18139fc0 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient(hunting): my global_id is 5153 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee00003530 con 0x7fee18139fc0 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094865738 0 0) 0x7fee180a14f0 con 0x7fee1809b780 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient(hunting): my global_id is 5166 2024-09-16T05:37:26.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee1809e930 con 0x7fee1809b780 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2968760639 0 0) 0x7fee00003530 con 0x7fee18139fc0 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 >> v1:172.21.15.63:6789/0 conn(0x7fee1809ac90 legacy=0x7fee1809b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 >> v1:172.21.15.6:6789/0 conn(0x7fee1809b780 legacy=0x7fee181388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 monclient: found mon.noname-b 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1809c400 con 0x7fee18139fc0 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: _finish_auth 0 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee08001620 con 0x7fee18139fc0 2024-09-16T05:37:26.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 5 monclient: authenticate success, global_id 5153 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: dump: 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.755 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee08001880 con 0x7fee18139fc0 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 4 set_mon_vals no callback set 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 1 -- 172.21.15.63:0/1887859734 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee08001bd0 con 0x7fee18139fc0 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee15d9b640 10 monclient: dump: 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.756 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient: shutdown 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- 172.21.15.63:0/1887859734 >> v1:172.21.15.6:6790/0 conn(0x7fee18139fc0 legacy=0x7fee1813c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- 172.21.15.63:0/1887859734 shutdown_connections 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- 172.21.15.63:0/1887859734 >> 172.21.15.63:0/1887859734 conn(0x7fee180a5290 msgr2=0x7fee180a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- 172.21.15.63:0/1887859734 shutdown_connections 2024-09-16T05:37:26.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- 172.21.15.63:0/1887859734 wait complete. 2024-09-16T05:37:26.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient: build_initial_monmap 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient: monmap: 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:26.762 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 Processor -- start 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- start start 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient: init 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient(hunting): picked mon.noname-a con 0x7fee1809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient(hunting): picked mon.noname-b con 0x7fee1809b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient(hunting): picked mon.noname-c con 0x7fee18139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1811faa0 con 0x7fee1809ac90 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1811fd80 con 0x7fee1809b780 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee181399f0 con 0x7fee18139fc0 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fee1809b780 legacy=0x7fee181ce5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:26.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.747+0000 7fee1d09b640 10 monclient(hunting): authenticate will time out at 1103.527888s 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1759e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fee18139fc0 0x7fee181d1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55460/0 (socket says 172.21.15.63:55460) 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1759e640 1 -- 172.21.15.63:0/122168383 learned_addr learned my addr 172.21.15.63:0/122168383 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880095580 0 0) 0x7fee181399f0 con 0x7fee18139fc0 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): my global_id is 5156 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee04003530 con 0x7fee18139fc0 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894691490 0 0) 0x7fee1811faa0 con 0x7fee1809ac90 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:26.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): my global_id is 5169 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee181399f0 con 0x7fee1809ac90 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3859928403 0 0) 0x7fee181399f0 con 0x7fee1809ac90 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee1811faa0 con 0x7fee1809ac90 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee10004450 con 0x7fee1809ac90 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): dump: 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.765 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1236773168 0 0) 0x7fee04003530 con 0x7fee18139fc0 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee181399f0 con 0x7fee18139fc0 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee08001560 con 0x7fee18139fc0 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:26.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient(hunting): dump: 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 589825519 0 0) 0x7fee1811faa0 con 0x7fee1809ac90 2024-09-16T05:37:26.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 >> v1:172.21.15.6:6790/0 conn(0x7fee18139fc0 legacy=0x7fee181d1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 >> v1:172.21.15.63:6789/0 conn(0x7fee1809b780 legacy=0x7fee181ce5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 monclient: found mon.a 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee180a3db0 con 0x7fee1809ac90 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 5 monclient: authenticate success, global_id 5169 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 10 monclient: _renew_subs 2024-09-16T05:37:26.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee1811f7c0 con 0x7fee1809ac90 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 10 monclient: _renew_subs 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fee1811ce00 con 0x7fee1809ac90 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee10004a60 con 0x7fee1809ac90 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee10004e30 con 0x7fee1809ac90 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1559a640 4 set_mon_vals no callback set 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 10 monclient: dump: 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:26.769 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf8005090 con 0x7fee1809ac90 2024-09-16T05:37:26.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fee10005050 con 0x7fee1809ac90 2024-09-16T05:37:26.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(79..79 src has 1..79) ==== 5449+0+0 (unknown 2367597548 0 0) 0x7fee10044c20 con 0x7fee1809ac90 2024-09-16T05:37:26.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.751+0000 7fee1659c640 10 monclient: get_auth_request con 0x7fee04036a00 auth_method 0 2024-09-16T05:37:26.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.763+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fee100157b0 con 0x7fee1809ac90 2024-09-16T05:37:26.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.763+0000 7fedff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:26.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.763+0000 7fedff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:26.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.935+0000 7fee1d09b640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T05:37:26.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.935+0000 7fee1d09b640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T05:37:26.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.935+0000 7fee1d09b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:26.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:26.935+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) -- 0x7fedf8005b90 con 0x7fee1809ac90 2024-09-16T05:37:27.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.075+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 removed item id -11 name 'host1' from crush map v80) ==== 147+0+0 (unknown 4259604224 0 0) 0x7fee100150b0 con 0x7fee1809ac90 2024-09-16T05:37:27.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.075+0000 7fedff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T05:37:27.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.075+0000 7fedff7fe640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T05:37:27.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.179+0000 7fee1d09b640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T05:37:27.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.179+0000 7fee1d09b640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T05:37:27.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.179+0000 7fee1d09b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:27.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.179+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) -- 0x7fedf80010a0 con 0x7fee1809ac90 2024-09-16T05:37:27.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.179+0000 7fedff7fe640 1 -- 172.21.15.63:0/122168383 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 item -11 does not appear in that position v80) ==== 141+0+0 (unknown 2349056600 0 0) 0x7fee1001dc60 con 0x7fee1809ac90 2024-09-16T05:37:27.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.179+0000 7fedff7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T05:37:27.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.179+0000 7fedff7fe640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T05:37:27.183 INFO:tasks.workunit.client.0.smithi063.stderr:item -11 does not appear in that position 2024-09-16T05:37:27.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.183+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 >> v1:172.21.15.63:6812/3643207211 conn(0x7fee04036a00 legacy=0x7fee04038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.183+0000 7fee1d09b640 10 monclient: shutdown 2024-09-16T05:37:27.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.183+0000 7fee1d09b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:27.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.183+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 >> v1:172.21.15.6:6789/0 conn(0x7fee1809ac90 legacy=0x7fee18132ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.183+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 shutdown_connections 2024-09-16T05:37:27.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.183+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 >> 172.21.15.63:0/122168383 conn(0x7fee180a5290 msgr2=0x7fee180a4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:27.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.183+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 shutdown_connections 2024-09-16T05:37:27.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.183+0000 7fee1d09b640 1 -- 172.21.15.63:0/122168383 wait complete. 2024-09-16T05:37:27.200 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c host1 2024-09-16T05:37:27.200 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:27.201 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 1 2024-09-16T05:37:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.311+0000 7f4b51a82640 10 monclient: get_monmap_and_config 2024-09-16T05:37:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.311+0000 7f4b51a82640 10 monclient: build_initial_monmap 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.311+0000 7f4b51a82640 10 monclient: monmap: 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:27.316015+0000 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:27.316015+0000 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:27.317 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 Processor -- start 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- start start 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient: init 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient(hunting): picked mon.noname-b con 0x7f4b4c09ee70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient(hunting): picked mon.noname-c con 0x7f4b4c0a45c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient(hunting): picked mon.noname-a con 0x7f4b4c09f960 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4c0a4ee0 con 0x7f4b4c09f960 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4c1413d0 con 0x7f4b4c0a45c0 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4c1425b0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient(hunting): authenticate will time out at 1104.093467s 2024-09-16T05:37:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4affd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4b4c0a45c0 legacy=0x7f4b4c09f250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:27.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4b4c09ee70 0x7f4b4c0a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55472/0 (socket says 172.21.15.63:55472) 2024-09-16T05:37:27.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b4c09f960 0x7f4b4c13eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42956/0 (socket says 172.21.15.63:42956) 2024-09-16T05:37:27.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4b7fe640 1 -- 172.21.15.63:0/755747904 learned_addr learned my addr 172.21.15.63:0/755747904 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:27.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 119516602 0 0) 0x7f4b4c1425b0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient(hunting): my global_id is 5159 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b34003530 con 0x7f4b4c09ee70 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2704241017 0 0) 0x7f4b4c0a4ee0 con 0x7f4b4c09f960 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient(hunting): my global_id is 5172 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b4c1425b0 con 0x7f4b4c09f960 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3502406839 0 0) 0x7f4b34003530 con 0x7f4b4c09ee70 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 >> v1:172.21.15.63:6789/0 conn(0x7f4b4c0a45c0 legacy=0x7f4b4c09f250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:27.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 >> v1:172.21.15.6:6789/0 conn(0x7f4b4c09f960 legacy=0x7f4b4c13eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 monclient: found mon.noname-b 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4c143790 con 0x7f4b4c09ee70 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b440019a0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:27.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: dump: 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 5 monclient: authenticate success, global_id 5159 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:27.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b44001c50 con 0x7f4b4c09ee70 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 1 -- 172.21.15.63:0/755747904 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b44002020 con 0x7f4b4c09ee70 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 4 set_mon_vals no callback set 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b4a7fc640 10 monclient: dump: 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.328 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 10 monclient: shutdown 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- 172.21.15.63:0/755747904 >> v1:172.21.15.6:6790/0 conn(0x7f4b4c09ee70 legacy=0x7f4b4c0a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- 172.21.15.63:0/755747904 shutdown_connections 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- 172.21.15.63:0/755747904 >> 172.21.15.63:0/755747904 conn(0x7f4b4c0a5d10 msgr2=0x7f4b4c0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- 172.21.15.63:0/755747904 shutdown_connections 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.315+0000 7f4b51a82640 1 -- 172.21.15.63:0/755747904 wait complete. 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: build_initial_monmap 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: monmap: 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.330 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 1 Processor -- start 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 1 -- start start 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: init 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient(hunting): picked mon.noname-a con 0x7f4b4c09ee70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient(hunting): picked mon.noname-c con 0x7f4b4c09f960 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient(hunting): picked mon.noname-b con 0x7f4b4c0a45c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4c1389b0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4c138c90 con 0x7f4b4c0a45c0 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4c1420f0 con 0x7f4b4c09f960 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient(hunting): authenticate will time out at 1104.097166s 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b4bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4b4c0a45c0 legacy=0x7f4b4c1d36f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b4b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b4c09ee70 0x7f4b4c06fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42958/0 (socket says 172.21.15.63:42958) 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b4b7fe640 1 -- 172.21.15.63:0/3601913884 learned_addr learned my addr 172.21.15.63:0/3601913884 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857381394 0 0) 0x7f4b4c1389b0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): my global_id is 5175 2024-09-16T05:37:27.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b38003560 con 0x7f4b4c09ee70 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814526440 0 0) 0x7f4b4c1420f0 con 0x7f4b4c09f960 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): my global_id is 5162 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b4c1389b0 con 0x7f4b4c09f960 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 257402968 0 0) 0x7f4b38003560 con 0x7f4b4c09ee70 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b4c1420f0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b44001fa0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): dump: 2024-09-16T05:37:27.333 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4162458081 0 0) 0x7f4b4c1389b0 con 0x7f4b4c09f960 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b38003560 con 0x7f4b4c09f960 2024-09-16T05:37:27.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b400040d0 con 0x7f4b4c09f960 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient(hunting): dump: 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.335 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3717289369 0 0) 0x7f4b4c1420f0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 >> v1:172.21.15.6:6790/0 conn(0x7f4b4c09f960 legacy=0x7f4b4c070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 >> v1:172.21.15.63:6789/0 conn(0x7f4b4c0a45c0 legacy=0x7f4b4c1d36f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 monclient: found mon.a 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4c0a0500 con 0x7f4b4c09ee70 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient: _finish_auth 0 2024-09-16T05:37:27.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 5 monclient: authenticate success, global_id 5175 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: _renew_subs 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4c072aa0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: _renew_subs 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b4c071d50 con 0x7f4b4c09ee70 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b44001c50 con 0x7f4b4c09ee70 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b49ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b44005170 con 0x7f4b4c09ee70 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:27.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b33fff640 10 monclient: dump: 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.319+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b2c005090 con 0x7f4b4c09ee70 2024-09-16T05:37:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.323+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4b44005310 con 0x7f4b4c09ee70 2024-09-16T05:37:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.323+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(80..80 src has 1..80) ==== 5409+0+0 (unknown 1043726573 0 0) 0x7f4b44044b00 con 0x7f4b4c09ee70 2024-09-16T05:37:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.323+0000 7f4b4affd640 10 monclient: get_auth_request con 0x7f4b380369c0 auth_method 0 2024-09-16T05:37:27.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.335+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4b440158b0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.335+0000 7f4b33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:27.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.335+0000 7f4b33fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:27.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.495+0000 7f4b51a82640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:27.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.495+0000 7f4b51a82640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:27.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.495+0000 7f4b51a82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:27.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.495+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f4b2c005b90 con 0x7f4b4c09ee70 2024-09-16T05:37:27.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.495+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v80) ==== 56+0+1340 (unknown 1137015514 0 2557657599) 0x7f4b440151b0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.495+0000 7f4b33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:27.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.495+0000 7f4b33fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:27.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.591+0000 7f4b51a82640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.591+0000 7f4b51a82640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.591+0000 7f4b51a82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.591+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f4b2c0008e0 con 0x7f4b4c09ee70 2024-09-16T05:37:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.591+0000 7f4b33fff640 1 -- 172.21.15.63:0/3601913884 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v80) ==== 56+0+1340 (unknown 1137015514 0 2557657599) 0x7f4b44040820 con 0x7f4b4c09ee70 2024-09-16T05:37:27.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.591+0000 7f4b33fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:27.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.591+0000 7f4b33fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:27.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.595+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4b380369c0 legacy=0x7f4b38038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.595+0000 7f4b51a82640 10 monclient: shutdown 2024-09-16T05:37:27.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.595+0000 7f4b51a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:27.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.595+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 >> v1:172.21.15.6:6789/0 conn(0x7f4b4c09ee70 legacy=0x7f4b4c06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.595+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 shutdown_connections 2024-09-16T05:37:27.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.595+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 >> 172.21.15.63:0/3601913884 conn(0x7f4b4c0a5d10 msgr2=0x7f4b4c0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:27.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.595+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 shutdown_connections 2024-09-16T05:37:27.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.595+0000 7f4b51a82640 1 -- 172.21.15.63:0/3601913884 wait complete. 2024-09-16T05:37:27.613 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm osd.7 host1 2024-09-16T05:37:27.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient: get_monmap_and_config 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient: build_initial_monmap 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient: monmap: 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:27.729246+0000 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:27.729246+0000 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:27.730 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:27.731 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 1 Processor -- start 2024-09-16T05:37:27.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 1 -- start start 2024-09-16T05:37:27.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient: init 2024-09-16T05:37:27.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:27.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:27.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient(hunting): picked mon.noname-b con 0x7f02780a1920 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:27.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient(hunting): picked mon.noname-c con 0x7f02780a2410 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:27.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient(hunting): picked mon.noname-a con 0x7f027809bbd0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:27.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027806cae0 con 0x7f027809bbd0 2024-09-16T05:37:27.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027809aa60 con 0x7f02780a2410 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027809b7b0 con 0x7f02780a1920 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f028038e640 10 monclient(hunting): authenticate will time out at 1104.507060s 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f027d929640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f02780a2410 legacy=0x7f027813bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f027e12a640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f02780a1920 0x7f02780a1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55486/0 (socket says 172.21.15.63:55486) 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f027e92b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f027809bbd0 0x7f027813f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42972/0 (socket says 172.21.15.63:42972) 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f027e12a640 1 -- 172.21.15.63:0/3512826905 learned_addr learned my addr 172.21.15.63:0/3512826905 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.727+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1551384389 0 0) 0x7f027809b7b0 con 0x7f02780a1920 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient(hunting): _negotiate 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient(hunting): my global_id is 5165 2024-09-16T05:37:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0264003530 con 0x7f02780a1920 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 531748093 0 0) 0x7f02700013d0 con 0x7f027809bbd0 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient(hunting): _negotiate 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient(hunting): my global_id is 5178 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f027809b7b0 con 0x7f027809bbd0 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3729013454 0 0) 0x7f0264003530 con 0x7f02780a1920 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 >> v1:172.21.15.63:6789/0 conn(0x7f02780a2410 legacy=0x7f027813bd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 >> v1:172.21.15.6:6789/0 conn(0x7f027809bbd0 legacy=0x7f027813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: _finish_hunting 0 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 monclient: found mon.noname-b 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027806c3e0 con 0x7f02780a1920 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: _finish_auth 0 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: _check_auth_tickets 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0274001620 con 0x7f02780a1920 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: dump: 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.739 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 5 monclient: authenticate success, global_id 5165 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0274001880 con 0x7f02780a1920 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 1 -- 172.21.15.63:0/3512826905 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0274001bd0 con 0x7f02780a1920 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:27.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d128640 10 monclient: dump: 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 4 set_mon_vals no callback set 2024-09-16T05:37:27.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient: shutdown 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- 172.21.15.63:0/3512826905 >> v1:172.21.15.6:6790/0 conn(0x7f02780a1920 legacy=0x7f02780a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- 172.21.15.63:0/3512826905 shutdown_connections 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- 172.21.15.63:0/3512826905 >> 172.21.15.63:0/3512826905 conn(0x7f0278075470 msgr2=0x7f0278075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- 172.21.15.63:0/3512826905 shutdown_connections 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- 172.21.15.63:0/3512826905 wait complete. 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient: build_initial_monmap 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient: monmap: 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:27.742 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 Processor -- start 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- start start 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient: init 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient(hunting): picked mon.noname-b con 0x7f027809bbd0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient(hunting): picked mon.noname-a con 0x7f02780a1920 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:27.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient(hunting): picked mon.noname-c con 0x7f02780a2410 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027811eb60 con 0x7f02780a1920 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027811d150 con 0x7f027809bbd0 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027e12a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f027809bbd0 legacy=0x7f027813e3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027811d3d0 con 0x7f02780a2410 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f028038e640 10 monclient(hunting): authenticate will time out at 1104.510846s 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027e92b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f02780a2410 0x7f027813f200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55498/0 (socket says 172.21.15.63:55498) 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027d929640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f02780a1920 0x7f027813eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42980/0 (socket says 172.21.15.63:42980) 2024-09-16T05:37:27.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f027e92b640 1 -- 172.21.15.63:0/2268274956 learned_addr learned my addr 172.21.15.63:0/2268274956 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1067591815 0 0) 0x7f027811d3d0 con 0x7f02780a2410 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 10 monclient(hunting): my global_id is 5168 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f025c003530 con 0x7f02780a2410 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556678782 0 0) 0x7f027811eb60 con 0x7f02780a1920 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 10 monclient(hunting): my global_id is 5181 2024-09-16T05:37:27.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.731+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f027811d3d0 con 0x7f02780a1920 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1853729214 0 0) 0x7f025c003530 con 0x7f02780a2410 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f027811eb60 con 0x7f02780a2410 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0270001cc0 con 0x7f02780a2410 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient(hunting): dump: 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.748 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 381055184 0 0) 0x7f027811d3d0 con 0x7f02780a1920 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f025c003530 con 0x7f02780a1920 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f02680040d0 con 0x7f02780a1920 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient(hunting): dump: 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.749 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1344685918 0 0) 0x7f027811eb60 con 0x7f02780a2410 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 >> v1:172.21.15.63:6789/0 conn(0x7f027809bbd0 legacy=0x7f027813e3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 >> v1:172.21.15.6:6789/0 conn(0x7f02780a1920 legacy=0x7f027813eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 monclient: found mon.c 2024-09-16T05:37:27.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027809d910 con 0x7f02780a2410 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient: _finish_auth 0 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 5 monclient: authenticate success, global_id 5168 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 10 monclient: _renew_subs 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f027809d100 con 0x7f02780a2410 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 10 monclient: _renew_subs 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f02781d49c0 con 0x7f02780a2410 2024-09-16T05:37:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0270004b70 con 0x7f02780a2410 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0270004ec0 con 0x7f02780a2410 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f027c927640 4 set_mon_vals no callback set 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 10 monclient: dump: 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:27.752 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0260005090 con 0x7f02780a2410 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f027000fc60 con 0x7f02780a2410 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(80..80 src has 1..80) ==== 5409+0+0 (unknown 1043726573 0 0) 0x7f0270044bb0 con 0x7f02780a2410 2024-09-16T05:37:27.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.735+0000 7f027e12a640 10 monclient: get_auth_request con 0x7f025c036a20 auth_method 0 2024-09-16T05:37:27.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.747+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0270044e20 con 0x7f02780a2410 2024-09-16T05:37:27.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.747+0000 7f026effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:27.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.747+0000 7f026effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:27.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.907+0000 7f028038e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T05:37:27.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.907+0000 7f028038e640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T05:37:27.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.907+0000 7f028038e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:27.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:27.907+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) -- 0x7f0260005b90 con 0x7f02780a2410 2024-09-16T05:37:28.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.087+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 removed item id 7 name 'osd.7' from crush map v81) ==== 143+0+0 (unknown 3056957958 0 0) 0x7f027806cae0 con 0x7f02780a2410 2024-09-16T05:37:28.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.087+0000 7f026effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T05:37:28.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.087+0000 7f026effd640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-16T05:37:28.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.183+0000 7f028038e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T05:37:28.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.183+0000 7f028038e640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T05:37:28.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.183+0000 7f028038e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:28.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.183+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) -- 0x7f02600010a0 con 0x7f02780a2410 2024-09-16T05:37:28.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f026effd640 1 -- 172.21.15.63:0/2268274956 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 device 'osd.7' does not appear in the crush map v81) ==== 145+0+0 (unknown 3395045225 0 0) 0x7f0270015760 con 0x7f02780a2410 2024-09-16T05:37:28.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f026effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T05:37:28.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f026effd640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-16T05:37:28.190 INFO:tasks.workunit.client.0.smithi063.stderr:device 'osd.7' does not appear in the crush map 2024-09-16T05:37:28.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 >> v1:172.21.15.63:6812/3643207211 conn(0x7f025c036a20 legacy=0x7f025c038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:28.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f028038e640 10 monclient: shutdown 2024-09-16T05:37:28.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f028038e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:28.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 >> v1:172.21.15.6:6790/0 conn(0x7f02780a2410 legacy=0x7f027813f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:28.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 shutdown_connections 2024-09-16T05:37:28.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 >> 172.21.15.63:0/2268274956 conn(0x7f0278075470 msgr2=0x7f027809def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:28.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 shutdown_connections 2024-09-16T05:37:28.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.187+0000 7f028038e640 1 -- 172.21.15.63:0/2268274956 wait complete. 2024-09-16T05:37:28.205 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm host1 2024-09-16T05:37:28.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient: get_monmap_and_config 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient: build_initial_monmap 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient: monmap: 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:28.318156+0000 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:28.318156+0000 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:28.319 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:28.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 1 Processor -- start 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 1 -- start start 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient: init 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient(hunting): picked mon.noname-b con 0x7f5304136ac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient(hunting): picked mon.noname-c con 0x7f530409fd80 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient(hunting): picked mon.noname-a con 0x7f530413d0b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5304072cb0 con 0x7f530413d0b0 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5304136670 con 0x7f530409fd80 2024-09-16T05:37:28.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53040a14f0 con 0x7f5304136ac0 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f530a79a640 10 monclient(hunting): authenticate will time out at 1105.094909s 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f52fb7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f530409fd80 legacy=0x7f530413b990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f5308d37640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f530413d0b0 0x7f530413f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42982/0 (socket says 172.21.15.63:42982) 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f5308d37640 1 -- 172.21.15.63:0/3924448974 learned_addr learned my addr 172.21.15.63:0/3924448974 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646698391 0 0) 0x7f53040a14f0 con 0x7f5304136ac0 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 10 monclient(hunting): my global_id is 5171 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52f0003530 con 0x7f5304136ac0 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450508401 0 0) 0x7f52fc0013d0 con 0x7f530413d0b0 2024-09-16T05:37:28.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:28.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:28.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:28.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 10 monclient(hunting): my global_id is 5184 2024-09-16T05:37:28.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.315+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53040a14f0 con 0x7f530413d0b0 2024-09-16T05:37:28.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 751071327 0 0) 0x7f53040a14f0 con 0x7f530413d0b0 2024-09-16T05:37:28.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 >> v1:172.21.15.6:6790/0 conn(0x7f5304136ac0 legacy=0x7f5304136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:28.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 >> v1:172.21.15.63:6789/0 conn(0x7f530409fd80 legacy=0x7f530413b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:28.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:28.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 1 monclient: found mon.noname-a 2024-09-16T05:37:28.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:28.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:28.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53040a0630 con 0x7f530413d0b0 2024-09-16T05:37:28.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52fc001a80 con 0x7f530413d0b0 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: dump: 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:28.333 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 5 monclient: authenticate success, global_id 5184 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52fc001ce0 con 0x7f530413d0b0 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 1 -- 172.21.15.63:0/3924448974 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52fc004c30 con 0x7f530413d0b0 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53037fe640 10 monclient: dump: 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:28.334 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 4 set_mon_vals no callback set 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient: shutdown 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- 172.21.15.63:0/3924448974 >> v1:172.21.15.6:6789/0 conn(0x7f530413d0b0 legacy=0x7f530413f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- 172.21.15.63:0/3924448974 shutdown_connections 2024-09-16T05:37:28.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- 172.21.15.63:0/3924448974 >> 172.21.15.63:0/3924448974 conn(0x7f53040a1c90 msgr2=0x7f53040a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- 172.21.15.63:0/3924448974 shutdown_connections 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- 172.21.15.63:0/3924448974 wait complete. 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient: build_initial_monmap 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient: monmap: 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:28.336 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 Processor -- start 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- start start 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient: init 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient(hunting): picked mon.noname-c con 0x7f530409fd80 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient(hunting): picked mon.noname-a con 0x7f5304136ac0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient(hunting): picked mon.noname-b con 0x7f530413d0b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f530411f9a0 con 0x7f5304136ac0 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5304120e50 con 0x7f530413d0b0 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f530413c7b0 con 0x7f530409fd80 2024-09-16T05:37:28.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f530a79a640 10 monclient(hunting): authenticate will time out at 1105.098781s 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f5308d37640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f530413d0b0 legacy=0x7f53041d3e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f5303fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f530409fd80 0x7f53041353f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55524/0 (socket says 172.21.15.63:55524) 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f52fb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5304136ac0 0x7f53041d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:42994/0 (socket says 172.21.15.63:42994) 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f5303fff640 1 -- 172.21.15.63:0/2369013225 learned_addr learned my addr 172.21.15.63:0/2369013225 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1412220620 0 0) 0x7f530413c7b0 con 0x7f530409fd80 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 10 monclient(hunting): my global_id is 5174 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52e4003530 con 0x7f530409fd80 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088036648 0 0) 0x7f530411f9a0 con 0x7f5304136ac0 2024-09-16T05:37:28.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 10 monclient(hunting): my global_id is 5187 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.319+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f530413c7b0 con 0x7f5304136ac0 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4094333511 0 0) 0x7f52e4003530 con 0x7f530409fd80 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f530411f9a0 con 0x7f530409fd80 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 927580493 0 0) 0x7f530413c7b0 con 0x7f5304136ac0 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52e4003530 con 0x7f5304136ac0 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52f4001a40 con 0x7f530409fd80 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient(hunting): dump: 2024-09-16T05:37:28.339 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52ec004020 con 0x7f5304136ac0 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:28.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient(hunting): dump: 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2263748831 0 0) 0x7f52e4003530 con 0x7f5304136ac0 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 >> v1:172.21.15.6:6790/0 conn(0x7f530409fd80 legacy=0x7f53041353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:28.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 >> v1:172.21.15.63:6789/0 conn(0x7f530413d0b0 legacy=0x7f53041d3e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 monclient: found mon.a 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5304120b70 con 0x7f5304136ac0 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 5 monclient: authenticate success, global_id 5187 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 10 monclient: _renew_subs 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f530411f720 con 0x7f5304136ac0 2024-09-16T05:37:28.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 10 monclient: _renew_subs 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f530411e250 con 0x7f5304136ac0 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52ec004200 con 0x7f5304136ac0 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52ec004550 con 0x7f5304136ac0 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f5302ffd640 4 set_mon_vals no callback set 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 10 monclient: dump: 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.343 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e8005090 con 0x7f5304136ac0 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f52ec004770 con 0x7f5304136ac0 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(81..81 src has 1..81) ==== 5388+0+0 (unknown 121091825 0 0) 0x7f52ec044660 con 0x7f5304136ac0 2024-09-16T05:37:28.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.323+0000 7f5303fff640 10 monclient: get_auth_request con 0x7f52e40369a0 auth_method 0 2024-09-16T05:37:28.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.335+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f52ec015170 con 0x7f5304136ac0 2024-09-16T05:37:28.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.335+0000 7f53017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:28.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.335+0000 7f53017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:28.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.499+0000 7f530a79a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T05:37:28.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.499+0000 7f530a79a640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T05:37:28.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.499+0000 7f530a79a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:28.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:28.499+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) -- 0x7f52e8005380 con 0x7f5304136ac0 2024-09-16T05:37:29.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.090+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 removed item id -11 name 'host1' from crush map v82) ==== 124+0+0 (unknown 1851892854 0 0) 0x7f52ec018e20 con 0x7f5304136ac0 2024-09-16T05:37:29.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.090+0000 7f53017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T05:37:29.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.090+0000 7f53017fa640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T05:37:29.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T05:37:29.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T05:37:29.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) -- 0x7f52e80010a0 con 0x7f5304136ac0 2024-09-16T05:37:29.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f53017fa640 1 -- 172.21.15.63:0/2369013225 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 device 'host1' does not appear in the crush map v82) ==== 124+0+0 (unknown 4127511703 0 0) 0x7f52ec01d6e0 con 0x7f5304136ac0 2024-09-16T05:37:29.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f53017fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T05:37:29.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f53017fa640 10 monclient: _finish_command 3 = system:0 device 'host1' does not appear in the crush map 2024-09-16T05:37:29.198 INFO:tasks.workunit.client.0.smithi063.stderr:device 'host1' does not appear in the crush map 2024-09-16T05:37:29.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 >> v1:172.21.15.63:6812/3643207211 conn(0x7f52e40369a0 legacy=0x7f52e4038e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 10 monclient: shutdown 2024-09-16T05:37:29.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:29.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 >> v1:172.21.15.6:6789/0 conn(0x7f5304136ac0 legacy=0x7f53041d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 shutdown_connections 2024-09-16T05:37:29.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 >> 172.21.15.63:0/2369013225 conn(0x7f53040a1c90 msgr2=0x7f530409cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:29.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 shutdown_connections 2024-09-16T05:37:29.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.194+0000 7f530a79a640 1 -- 172.21.15.63:0/2369013225 wait complete. 2024-09-16T05:37:29.215 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:29.215 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c host1 2024-09-16T05:37:29.215 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 0 2024-09-16T05:37:29.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 10 monclient: get_monmap_and_config 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 10 monclient: build_initial_monmap 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 10 monclient: monmap: 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:29.331016+0000 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:29.331016+0000 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:29.332 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 1 Processor -- start 2024-09-16T05:37:29.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 1 -- start start 2024-09-16T05:37:29.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 10 monclient: init 2024-09-16T05:37:29.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:29.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:29.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 10 monclient(hunting): picked mon.noname-b con 0x7eff6009e790 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:29.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.326+0000 7eff6638e640 10 monclient(hunting): picked mon.noname-c con 0x7eff6009fd70 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:29.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient(hunting): picked mon.noname-a con 0x7eff601394f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6009c500 con 0x7eff601394f0 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff600a1260 con 0x7eff6009fd70 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6009c9c0 con 0x7eff6009e790 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7eff6009fd70 legacy=0x7eff60137dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient(hunting): authenticate will time out at 1106.108081s 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7eff6009e790 0x7eff6009eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55540/0 (socket says 172.21.15.63:55540) 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5ffff640 1 -- 172.21.15.63:0/1153198226 learned_addr learned my addr 172.21.15.63:0/1153198226 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3725769920 0 0) 0x7eff6009c9c0 con 0x7eff6009e790 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:29.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient(hunting): my global_id is 5177 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff4c003530 con 0x7eff6009e790 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2989291277 0 0) 0x7eff6009c500 con 0x7eff601394f0 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient(hunting): my global_id is 5190 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff6009c9c0 con 0x7eff601394f0 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2159771789 0 0) 0x7eff4c003530 con 0x7eff6009e790 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 >> v1:172.21.15.63:6789/0 conn(0x7eff6009fd70 legacy=0x7eff60137dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 >> v1:172.21.15.6:6789/0 conn(0x7eff601394f0 legacy=0x7eff6013b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 monclient: found mon.noname-b 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff600a0180 con 0x7eff6009e790 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: _finish_auth 0 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff480019a0 con 0x7eff6009e790 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 5 monclient: authenticate success, global_id 5177 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:29.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: dump: 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff48001bd0 con 0x7eff6009e790 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 4 set_mon_vals no callback set 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 1 -- 172.21.15.63:0/1153198226 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff48001fa0 con 0x7eff6009e790 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff5effd640 10 monclient: dump: 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.341 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient: shutdown 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- 172.21.15.63:0/1153198226 >> v1:172.21.15.6:6790/0 conn(0x7eff6009e790 legacy=0x7eff6009eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- 172.21.15.63:0/1153198226 shutdown_connections 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- 172.21.15.63:0/1153198226 >> 172.21.15.63:0/1153198226 conn(0x7eff600a1b60 msgr2=0x7eff600a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- 172.21.15.63:0/1153198226 shutdown_connections 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- 172.21.15.63:0/1153198226 wait complete. 2024-09-16T05:37:29.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient: build_initial_monmap 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient: monmap: 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 Processor -- start 2024-09-16T05:37:29.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- start start 2024-09-16T05:37:29.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient: init 2024-09-16T05:37:29.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:29.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:29.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient(hunting): picked mon.noname-a con 0x7eff6009e790 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:29.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient(hunting): picked mon.noname-c con 0x7eff6009fd70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:29.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient(hunting): picked mon.noname-b con 0x7eff601394f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:29.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6009c0f0 con 0x7eff6009e790 2024-09-16T05:37:29.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff60071c50 con 0x7eff601394f0 2024-09-16T05:37:29.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff600713b0 con 0x7eff6009fd70 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6638e640 10 monclient(hunting): authenticate will time out at 1106.111812s 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.330+0000 7eff6492b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7eff601394f0 legacy=0x7eff601d2e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7eff6009e790 0x7eff60072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43016/0 (socket says 172.21.15.63:43016) 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5ffff640 1 -- 172.21.15.63:0/2283045308 learned_addr learned my addr 172.21.15.63:0/2283045308 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160659152 0 0) 0x7eff6009c0f0 con 0x7eff6009e790 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): my global_id is 5193 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff40003530 con 0x7eff6009e790 2024-09-16T05:37:29.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 684281791 0 0) 0x7eff600713b0 con 0x7eff6009fd70 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): my global_id is 5180 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff6009c0f0 con 0x7eff6009fd70 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1875158289 0 0) 0x7eff40003530 con 0x7eff6009e790 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff600713b0 con 0x7eff6009e790 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff480018e0 con 0x7eff6009e790 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): dump: 2024-09-16T05:37:29.350 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2718908085 0 0) 0x7eff6009c0f0 con 0x7eff6009fd70 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff40003530 con 0x7eff6009fd70 2024-09-16T05:37:29.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff50004110 con 0x7eff6009fd70 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient(hunting): dump: 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.352 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3349739943 0 0) 0x7eff600713b0 con 0x7eff6009e790 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 >> v1:172.21.15.6:6790/0 conn(0x7eff6009fd70 legacy=0x7eff601cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 >> v1:172.21.15.63:6789/0 conn(0x7eff601394f0 legacy=0x7eff601d2e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 monclient: found mon.a 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff600704c0 con 0x7eff6009e790 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient: _finish_auth 0 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:29.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 5 monclient: authenticate success, global_id 5193 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 10 monclient: _renew_subs 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff601d4520 con 0x7eff6009e790 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 10 monclient: _renew_subs 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7eff601d48c0 con 0x7eff6009e790 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff48001bd0 con 0x7eff6009e790 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff48004fc0 con 0x7eff6009e790 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:29.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 10 monclient: dump: 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5e7fc640 4 set_mon_vals no callback set 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:29.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff44005090 con 0x7eff6009e790 2024-09-16T05:37:29.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7eff480051e0 con 0x7eff6009e790 2024-09-16T05:37:29.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(82..82 src has 1..82) ==== 5253+0+0 (unknown 857999504 0 0) 0x7eff48044b70 con 0x7eff6009e790 2024-09-16T05:37:29.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.334+0000 7eff5f7fe640 10 monclient: get_auth_request con 0x7eff40036a30 auth_method 0 2024-09-16T05:37:29.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.346+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7eff48015760 con 0x7eff6009e790 2024-09-16T05:37:29.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.346+0000 7eff5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:29.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.346+0000 7eff5cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:29.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.510+0000 7eff6638e640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:29.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.510+0000 7eff6638e640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:29.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.510+0000 7eff6638e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.510+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7eff44005b90 con 0x7eff6009e790 2024-09-16T05:37:29.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.514+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) ==== 56+0+1273 (unknown 2059188187 0 3412332199) 0x7eff480193b0 con 0x7eff6009e790 2024-09-16T05:37:29.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.514+0000 7eff5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:29.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.514+0000 7eff5cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:29.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.606+0000 7eff6638e640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:29.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.606+0000 7eff6638e640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:29.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.606+0000 7eff6638e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.606+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7eff440008e0 con 0x7eff6009e790 2024-09-16T05:37:29.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff5cff9640 1 -- 172.21.15.63:0/2283045308 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) ==== 56+0+1273 (unknown 2059188187 0 3412332199) 0x7eff4801dc70 con 0x7eff6009e790 2024-09-16T05:37:29.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff5cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:29.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff5cff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:29.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 >> v1:172.21.15.63:6812/3643207211 conn(0x7eff40036a30 legacy=0x7eff40038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff6638e640 10 monclient: shutdown 2024-09-16T05:37:29.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff6638e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:29.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 >> v1:172.21.15.6:6789/0 conn(0x7eff6009e790 legacy=0x7eff60072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 shutdown_connections 2024-09-16T05:37:29.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 >> 172.21.15.63:0/2283045308 conn(0x7eff600a1b60 msgr2=0x7eff60132ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:29.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 shutdown_connections 2024-09-16T05:37:29.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.610+0000 7eff6638e640 1 -- 172.21.15.63:0/2283045308 wait complete. 2024-09-16T05:37:29.629 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd tree-from host1 2024-09-16T05:37:29.629 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:29.629 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree-from host1 2024-09-16T05:37:29.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.742+0000 7f7921a25640 10 monclient: get_monmap_and_config 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.742+0000 7f7921a25640 10 monclient: build_initial_monmap 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.742+0000 7f7921a25640 10 monclient: monmap: 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:29.748079+0000 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:29.748079+0000 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:29.749 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 Processor -- start 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- start start 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient: init 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient(hunting): picked mon.noname-c con 0x7f791c09ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient(hunting): picked mon.noname-a con 0x7f791c09b070 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient(hunting): picked mon.noname-b con 0x7f791c13a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791c0722a0 con 0x7f791c09b070 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791c0a0510 con 0x7f791c09ac90 2024-09-16T05:37:29.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791c09e660 con 0x7f791c13a9a0 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient(hunting): authenticate will time out at 1106.525021s 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791affd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f791c09ac90 legacy=0x7f791c09f6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f791c13a9a0 0x7f791c13cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55564/0 (socket says 172.21.15.63:55564) 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791b7fe640 1 -- 172.21.15.63:0/2969631004 learned_addr learned my addr 172.21.15.63:0/2969631004 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f79127fc640 1 --1- 172.21.15.63:0/2969631004 >> v1:172.21.15.6:6789/0 conn(0x7f791c09b070 0x7f791c09b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43020/0 (socket says 172.21.15.63:43020) 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3574856874 0 0) 0x7f791c0722a0 con 0x7f791c09b070 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient(hunting): my global_id is 5196 2024-09-16T05:37:29.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7908003530 con 0x7f791c09b070 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124063778 0 0) 0x7f791c09e660 con 0x7f791c13a9a0 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient(hunting): my global_id is 5183 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f791c0722a0 con 0x7f791c13a9a0 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1084805767 0 0) 0x7f7908003530 con 0x7f791c09b070 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 >> v1:172.21.15.6:6790/0 conn(0x7f791c13a9a0 legacy=0x7f791c13cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 >> v1:172.21.15.63:6789/0 conn(0x7f791c09ac90 legacy=0x7f791c09f6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 monclient: found mon.noname-a 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:29.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c1414e0 con 0x7f791c09b070 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7904001620 con 0x7f791c09b070 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 5 monclient: authenticate success, global_id 5196 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:29.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: dump: 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79040018d0 con 0x7f791c09b070 2024-09-16T05:37:29.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 1 -- 172.21.15.63:0/2969631004 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7904001ce0 con 0x7f791c09b070 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f791a7fc640 10 monclient: dump: 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.757 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.758 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.758 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:29.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 4 set_mon_vals no callback set 2024-09-16T05:37:29.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 10 monclient: shutdown 2024-09-16T05:37:29.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:29.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- 172.21.15.63:0/2969631004 >> v1:172.21.15.6:6789/0 conn(0x7f791c09b070 legacy=0x7f791c09b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- 172.21.15.63:0/2969631004 shutdown_connections 2024-09-16T05:37:29.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- 172.21.15.63:0/2969631004 >> 172.21.15.63:0/2969631004 conn(0x7f791c0a1bf0 msgr2=0x7f791c0a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- 172.21.15.63:0/2969631004 shutdown_connections 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.746+0000 7f7921a25640 1 -- 172.21.15.63:0/2969631004 wait complete. 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: build_initial_monmap 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: monmap: 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:29.763 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 1 Processor -- start 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 1 -- start start 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: init 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient(hunting): picked mon.noname-a con 0x7f791c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient(hunting): picked mon.noname-b con 0x7f791c09b070 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient(hunting): picked mon.noname-c con 0x7f791c13a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791c11b600 con 0x7f791c09ac90 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791c1cc230 con 0x7f791c09b070 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791c135530 con 0x7f791c13a9a0 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:29.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient(hunting): authenticate will time out at 1106.529131s 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f79127fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f791c09b070 legacy=0x7f791c1cbb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f791b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f791c13a9a0 0x7f791c1d3670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55572/0 (socket says 172.21.15.63:55572) 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f791b7fe640 1 -- 172.21.15.63:0/2815273848 learned_addr learned my addr 172.21.15.63:0/2815273848 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2642729747 0 0) 0x7f791c135530 con 0x7f791c13a9a0 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): my global_id is 5186 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78fc003530 con 0x7f791c13a9a0 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2478997249 0 0) 0x7f791c11b600 con 0x7f791c09ac90 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:29.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): my global_id is 5199 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f791c135530 con 0x7f791c09ac90 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2907561306 0 0) 0x7f791c135530 con 0x7f791c09ac90 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f791c11b600 con 0x7f791c09ac90 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3056087817 0 0) 0x7f78fc003530 con 0x7f791c13a9a0 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f791c135530 con 0x7f791c13a9a0 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7914001da0 con 0x7f791c13a9a0 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.766 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f790c0040d0 con 0x7f791c09ac90 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:29.767 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3155590296 0 0) 0x7f791c11b600 con 0x7f791c09ac90 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 >> v1:172.21.15.6:6790/0 conn(0x7f791c13a9a0 legacy=0x7f791c1d3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:29.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 >> v1:172.21.15.63:6789/0 conn(0x7f791c09b070 legacy=0x7f791c1cbb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 monclient: found mon.a 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c13e090 con 0x7f791c09ac90 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient: _finish_auth 0 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 5 monclient: authenticate success, global_id 5199 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: _renew_subs 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f791c141270 con 0x7f791c09ac90 2024-09-16T05:37:29.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: _renew_subs 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f791c070df0 con 0x7f791c09ac90 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f790c004670 con 0x7f791c09ac90 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f790c0049c0 con 0x7f791c09ac90 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7919ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7918ff9640 10 monclient: dump: 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.770 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.750+0000 7f7921a25640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7900005090 con 0x7f791c09ac90 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.754+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f790c00f7e0 con 0x7f791c09ac90 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.754+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(82..82 src has 1..82) ==== 5253+0+0 (unknown 857999504 0 0) 0x7f790c044710 con 0x7f791c09ac90 2024-09-16T05:37:29.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.754+0000 7f79127fc640 10 monclient: get_auth_request con 0x7f78fc0369e0 auth_method 0 2024-09-16T05:37:29.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.766+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f790c015360 con 0x7f791c09ac90 2024-09-16T05:37:29.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.766+0000 7f7918ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:29.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.766+0000 7f7918ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:29.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.930+0000 7f7921a25640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T05:37:29.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.930+0000 7f7921a25640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T05:37:29.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.930+0000 7f7921a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:29.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.930+0000 7f7921a25640 1 -- 172.21.15.63:0/2815273848 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host1"} v 0) -- 0x7f7900005380 con 0x7f791c09ac90 2024-09-16T05:37:29.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.930+0000 7f7918ff9640 1 -- 172.21.15.63:0/2815273848 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host1"}]=-2 bucket 'host1' does not exist v82) ==== 109+0+0 (unknown 435488527 0 0) 0x7f790c014c60 con 0x7f791c09ac90 2024-09-16T05:37:29.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.930+0000 7f7918ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T05:37:29.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:29.930+0000 7f7918ff9640 10 monclient: _finish_command 2 = mon:2 bucket 'host1' does not exist 2024-09-16T05:37:29.934 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOENT: bucket 'host1' does not exist 2024-09-16T05:37:29.939 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:29.939 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree-from host2 2024-09-16T05:37:30.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.050+0000 7f269c8c5640 10 monclient: get_monmap_and_config 2024-09-16T05:37:30.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.050+0000 7f269c8c5640 10 monclient: build_initial_monmap 2024-09-16T05:37:30.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.050+0000 7f269c8c5640 10 monclient: monmap: 2024-09-16T05:37:30.057 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:30.057 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.057 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:30.056394+0000 2024-09-16T05:37:30.057 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:30.056394+0000 2024-09-16T05:37:30.057 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:30.058 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.058 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:30.058 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:30.058 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:30.058 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 Processor -- start 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 -- start start 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient: init 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2698075510 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient(hunting): picked mon.noname-b con 0x7f2698098cb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient(hunting): picked mon.noname-c con 0x7f269813b330 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269809efa0 con 0x7f2698075510 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2698132e80 con 0x7f269813b330 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269809f280 con 0x7f2698098cb0 2024-09-16T05:37:30.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient(hunting): authenticate will time out at 1106.833854s 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f2696d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f269813b330 legacy=0x7f269813d720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f2695d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2698098cb0 0x7f2698139c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55588/0 (socket says 172.21.15.63:55588) 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f2695d9b640 1 -- 172.21.15.63:0/1983192462 learned_addr learned my addr 172.21.15.63:0/1983192462 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269659c640 1 --1- 172.21.15.63:0/1983192462 >> v1:172.21.15.6:6789/0 conn(0x7f2698075510 0x7f26980981b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43036/0 (socket says 172.21.15.63:43036) 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 930739939 0 0) 0x7f2688001120 con 0x7f2698075510 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient(hunting): my global_id is 5202 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2680003530 con 0x7f2698075510 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3772053122 0 0) 0x7f269809f280 con 0x7f2698098cb0 2024-09-16T05:37:30.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient(hunting): my global_id is 5189 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2688001120 con 0x7f2698098cb0 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 332012819 0 0) 0x7f2680003530 con 0x7f2698075510 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 >> v1:172.21.15.6:6790/0 conn(0x7f2698098cb0 legacy=0x7f2698139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 >> v1:172.21.15.63:6789/0 conn(0x7f269813b330 legacy=0x7f269813d720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: _finish_hunting 0 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 monclient: found mon.noname-a 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269809f560 con 0x7f2698075510 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: _finish_auth 0 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: _check_auth_tickets 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f26880017f0 con 0x7f2698075510 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: dump: 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.065 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 5 monclient: authenticate success, global_id 5202 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2688001b10 con 0x7f2698075510 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 1 -- 172.21.15.63:0/1983192462 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f26880047f0 con 0x7f2698075510 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269559a640 10 monclient: dump: 2024-09-16T05:37:30.066 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 4 set_mon_vals no callback set 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 10 monclient: shutdown 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:30.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 -- 172.21.15.63:0/1983192462 >> v1:172.21.15.6:6789/0 conn(0x7f2698075510 legacy=0x7f26980981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 -- 172.21.15.63:0/1983192462 shutdown_connections 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 -- 172.21.15.63:0/1983192462 >> 172.21.15.63:0/1983192462 conn(0x7f26980a2520 msgr2=0x7f26980a2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.054+0000 7f269c8c5640 1 -- 172.21.15.63:0/1983192462 shutdown_connections 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 1 -- 172.21.15.63:0/1983192462 wait complete. 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: build_initial_monmap 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: monmap: 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.068 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:30.069 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:30.069 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:30.069 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 1 Processor -- start 2024-09-16T05:37:30.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 1 -- start start 2024-09-16T05:37:30.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: init 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient(hunting): picked mon.noname-c con 0x7f2698075510 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2698098cb0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient(hunting): picked mon.noname-b con 0x7f269813b330 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2698131ab0 con 0x7f2698098cb0 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269812bb50 con 0x7f269813b330 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269809bf20 con 0x7f2698075510 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient(hunting): authenticate will time out at 1106.837586s 2024-09-16T05:37:30.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2696d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f269813b330 legacy=0x7f26981d3c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2698075510 0x7f26981313a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55604/0 (socket says 172.21.15.63:55604) 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269659c640 1 -- 172.21.15.63:0/3582470810 learned_addr learned my addr 172.21.15.63:0/3582470810 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3844128570 0 0) 0x7f2698131ab0 con 0x7f2698098cb0 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): my global_id is 5205 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2678003530 con 0x7f2698098cb0 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1875250229 0 0) 0x7f269809bf20 con 0x7f2698075510 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): my global_id is 5192 2024-09-16T05:37:30.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2698131ab0 con 0x7f2698075510 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1874363336 0 0) 0x7f2678003530 con 0x7f2698098cb0 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f269809bf20 con 0x7f2698098cb0 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f268c001900 con 0x7f2698098cb0 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.075 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 626583249 0 0) 0x7f2698131ab0 con 0x7f2698075510 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2678003530 con 0x7f2698075510 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2688001cc0 con 0x7f2698075510 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:30.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2611210703 0 0) 0x7f269809bf20 con 0x7f2698098cb0 2024-09-16T05:37:30.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 >> v1:172.21.15.6:6790/0 conn(0x7f2698075510 legacy=0x7f26981313a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 >> v1:172.21.15.63:6789/0 conn(0x7f269813b330 legacy=0x7f26981d3c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 monclient: found mon.a 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26980a4f30 con 0x7f2698098cb0 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 5 monclient: authenticate success, global_id 5205 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: _renew_subs 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f269809c810 con 0x7f2698098cb0 2024-09-16T05:37:30.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: _renew_subs 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2698131e90 con 0x7f2698098cb0 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f268c0047f0 con 0x7f2698098cb0 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f268c004bc0 con 0x7f2698098cb0 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2694d99640 4 set_mon_vals no callback set 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.058+0000 7f2686ffd640 10 monclient: dump: 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.079 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.062+0000 7f269c8c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.062+0000 7f269c8c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.062+0000 7f269c8c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.062+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f267c005090 con 0x7f2698098cb0 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.062+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f268c00f9e0 con 0x7f2698098cb0 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.062+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(82..82 src has 1..82) ==== 5253+0+0 (unknown 857999504 0 0) 0x7f268c044810 con 0x7f2698098cb0 2024-09-16T05:37:30.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.062+0000 7f269659c640 10 monclient: get_auth_request con 0x7f26780369b0 auth_method 0 2024-09-16T05:37:30.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.074+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f268c015460 con 0x7f2698098cb0 2024-09-16T05:37:30.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.074+0000 7f2686ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.074+0000 7f2686ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:30.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.238+0000 7f269c8c5640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T05:37:30.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.238+0000 7f269c8c5640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T05:37:30.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.238+0000 7f269c8c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.238+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) -- 0x7f267c005380 con 0x7f2698098cb0 2024-09-16T05:37:30.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.238+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v82) ==== 80+0+177 (unknown 808184818 0 2845383371) 0x7f268c014d60 con 0x7f2698098cb0 2024-09-16T05:37:30.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.238+0000 7f2686ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T05:37:30.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.238+0000 7f2686ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:30.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T05:37:30.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 10 monclient: _send_command 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T05:37:30.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) -- 0x7f267c005770 con 0x7f2698098cb0 2024-09-16T05:37:30.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f2686ffd640 1 -- 172.21.15.63:0/3582470810 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v82) ==== 80+0+177 (unknown 808184818 0 2845383371) 0x7f268c01d910 con 0x7f2698098cb0 2024-09-16T05:37:30.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f2686ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T05:37:30.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f2686ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:30.343 INFO:tasks.workunit.client.0.smithi063.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T05:37:30.343 INFO:tasks.workunit.client.0.smithi063.stdout:-16 1.00000 host host2 2024-09-16T05:37:30.343 INFO:tasks.workunit.client.0.smithi063.stdout: 9 1.00000 osd.9 down 1.00000 1.00000 2024-09-16T05:37:30.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 >> v1:172.21.15.63:6812/3643207211 conn(0x7f26780369b0 legacy=0x7f2678038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 10 monclient: shutdown 2024-09-16T05:37:30.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:30.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 >> v1:172.21.15.6:6789/0 conn(0x7f2698098cb0 legacy=0x7f26981d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 shutdown_connections 2024-09-16T05:37:30.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 >> 172.21.15.63:0/3582470810 conn(0x7f26980a2520 msgr2=0x7f26980a3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:30.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 shutdown_connections 2024-09-16T05:37:30.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.338+0000 7f269c8c5640 1 -- 172.21.15.63:0/3582470810 wait complete. 2024-09-16T05:37:30.360 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd tree-from osd.9 2024-09-16T05:37:30.360 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:30.360 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree-from osd.9 2024-09-16T05:37:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient: get_monmap_and_config 2024-09-16T05:37:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient: build_initial_monmap 2024-09-16T05:37:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient: monmap: 2024-09-16T05:37:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:30.480659+0000 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:30.480659+0000 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:30.482 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 1 Processor -- start 2024-09-16T05:37:30.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 1 -- start start 2024-09-16T05:37:30.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient: init 2024-09-16T05:37:30.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:30.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:30.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient(hunting): picked mon.noname-c con 0x7fcc14136da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:30.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient(hunting): picked mon.noname-b con 0x7fcc140a4ea0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:30.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient(hunting): picked mon.noname-a con 0x7fcc1409fe80 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc140a48e0 con 0x7fcc1409fe80 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc140a4440 con 0x7fcc14136da0 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc140a4b60 con 0x7fcc140a4ea0 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient(hunting): authenticate will time out at 1107.257622s 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc18cfb640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcc14136da0 legacy=0x7fcc14137180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc13fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcc140a4ea0 0x7fcc1413bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55618/0 (socket says 172.21.15.63:55618) 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc13fff640 1 -- 172.21.15.63:0/2420791240 learned_addr learned my addr 172.21.15.63:0/2420791240 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1845034510 0 0) 0x7fcc140a4b60 con 0x7fcc140a4ea0 2024-09-16T05:37:30.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient(hunting): my global_id is 5195 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf8003530 con 0x7fcc140a4ea0 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1514443601 0 0) 0x7fcc140a48e0 con 0x7fcc1409fe80 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient(hunting): my global_id is 5208 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc140a4b60 con 0x7fcc1409fe80 2024-09-16T05:37:30.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 350648439 0 0) 0x7fcbf8003530 con 0x7fcc140a4ea0 2024-09-16T05:37:30.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 >> v1:172.21.15.63:6789/0 conn(0x7fcc14136da0 legacy=0x7fcc14137180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:30.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 >> v1:172.21.15.6:6789/0 conn(0x7fcc1409fe80 legacy=0x7fcc1413f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:30.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 monclient: found mon.noname-b 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc14137890 con 0x7fcc140a4ea0 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc00001620 con 0x7fcc140a4ea0 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: dump: 2024-09-16T05:37:30.496 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 5 monclient: authenticate success, global_id 5195 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc00001940 con 0x7fcc140a4ea0 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc1af5f640 4 set_mon_vals no callback set 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 1 -- 172.21.15.63:0/2420791240 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc00001d10 con 0x7fcc140a4ea0 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.478+0000 7fcc137fe640 10 monclient: dump: 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.498 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: shutdown 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:30.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- 172.21.15.63:0/2420791240 >> v1:172.21.15.6:6790/0 conn(0x7fcc140a4ea0 legacy=0x7fcc1413bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- 172.21.15.63:0/2420791240 shutdown_connections 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- 172.21.15.63:0/2420791240 >> 172.21.15.63:0/2420791240 conn(0x7fcc140a5e80 msgr2=0x7fcc140a6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- 172.21.15.63:0/2420791240 shutdown_connections 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- 172.21.15.63:0/2420791240 wait complete. 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: build_initial_monmap 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: monmap: 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:30.500 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 Processor -- start 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- start start 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: init 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:30.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1409fe80 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient(hunting): picked mon.noname-c con 0x7fcc140a4ea0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient(hunting): picked mon.noname-a con 0x7fcc14136da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc14125bc0 con 0x7fcc14136da0 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc141230d0 con 0x7fcc1409fe80 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc140a7280 con 0x7fcc140a4ea0 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient(hunting): authenticate will time out at 1107.261660s 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc18cfb640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcc1409fe80 legacy=0x7fcc141361c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc13fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcc140a4ea0 0x7fcc141d0330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55624/0 (socket says 172.21.15.63:55624) 2024-09-16T05:37:30.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc13fff640 1 -- 172.21.15.63:0/3292165836 learned_addr learned my addr 172.21.15.63:0/3292165836 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3615902259 0 0) 0x7fcc14125bc0 con 0x7fcc14136da0 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): my global_id is 5211 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbfc003530 con 0x7fcc14136da0 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183195833 0 0) 0x7fcc140a7280 con 0x7fcc140a4ea0 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): my global_id is 5198 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc14125bc0 con 0x7fcc140a4ea0 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3353896205 0 0) 0x7fcc14125bc0 con 0x7fcc140a4ea0 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc140a7280 con 0x7fcc140a4ea0 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc00001600 con 0x7fcc140a4ea0 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): dump: 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.504 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2915494302 0 0) 0x7fcbfc003530 con 0x7fcc14136da0 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc14125bc0 con 0x7fcc14136da0 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc040017c0 con 0x7fcc14136da0 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:30.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient(hunting): dump: 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.506 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2155537731 0 0) 0x7fcc140a7280 con 0x7fcc140a4ea0 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 >> v1:172.21.15.63:6789/0 conn(0x7fcc1409fe80 legacy=0x7fcc141361c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 >> v1:172.21.15.6:6789/0 conn(0x7fcc14136da0 legacy=0x7fcc141d3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 monclient: found mon.c 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc140a0570 con 0x7fcc140a4ea0 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:30.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 5 monclient: authenticate success, global_id 5198 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: _renew_subs 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc141238d0 con 0x7fcc140a4ea0 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: _renew_subs 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc1af5f640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc141368d0 con 0x7fcc140a4ea0 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc00001940 con 0x7fcc140a4ea0 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.482+0000 7fcc117fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc00004e70 con 0x7fcc140a4ea0 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc117fa640 10 monclient: dump: 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.509 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc12ffd640 4 set_mon_vals no callback set 2024-09-16T05:37:30.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc1af5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc1af5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc1af5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:30.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc1af5f640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf0005090 con 0x7fcc140a4ea0 2024-09-16T05:37:30.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcc0000fc10 con 0x7fcc140a4ea0 2024-09-16T05:37:30.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(82..82 src has 1..82) ==== 5253+0+0 (unknown 857999504 0 0) 0x7fcc00044930 con 0x7fcc140a4ea0 2024-09-16T05:37:30.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.486+0000 7fcc18cfb640 10 monclient: get_auth_request con 0x7fcbfc036aa0 auth_method 0 2024-09-16T05:37:30.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.498+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fcc000155e0 con 0x7fcc140a4ea0 2024-09-16T05:37:30.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.498+0000 7fcc117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.498+0000 7fcc117fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:30.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.666+0000 7fcc1af5f640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T05:37:30.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.666+0000 7fcc1af5f640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T05:37:30.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.666+0000 7fcc1af5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:30.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.666+0000 7fcc1af5f640 1 -- 172.21.15.63:0/3292165836 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree-from", "bucket": "osd.9"} v 0) -- 0x7fcbf0005380 con 0x7fcc140a4ea0 2024-09-16T05:37:30.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.666+0000 7fcc117fa640 1 -- 172.21.15.63:0/3292165836 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "osd.9"}]=-22 "osd.9" is not a bucket v82) ==== 103+0+0 (unknown 1301753818 0 0) 0x7fcc00014ee0 con 0x7fcc140a4ea0 2024-09-16T05:37:30.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.666+0000 7fcc117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T05:37:30.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.666+0000 7fcc117fa640 10 monclient: _finish_command 2 = mon:22 "osd.9" is not a bucket 2024-09-16T05:37:30.673 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: "osd.9" is not a bucket 2024-09-16T05:37:30.677 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:30.677 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush rm bar 2024-09-16T05:37:30.677 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:30.677 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm bar 2024-09-16T05:37:30.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient: get_monmap_and_config 2024-09-16T05:37:30.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient: build_initial_monmap 2024-09-16T05:37:30.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient: monmap: 2024-09-16T05:37:30.799 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:30.799 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.800 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:30.798732+0000 2024-09-16T05:37:30.800 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:30.798732+0000 2024-09-16T05:37:30.800 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:30.800 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.800 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:30.800 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:30.800 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:30.800 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 1 Processor -- start 2024-09-16T05:37:30.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 1 -- start start 2024-09-16T05:37:30.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient: init 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient(hunting): picked mon.noname-b con 0x7fe4a40a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient(hunting): picked mon.noname-c con 0x7fe4a409b450 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient(hunting): picked mon.noname-a con 0x7fe4a409b8a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4a409e970 con 0x7fe4a409b8a0 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4a409c370 con 0x7fe4a409b450 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4a409d620 con 0x7fe4a40a08a0 2024-09-16T05:37:30.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4a3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe4a409b450 legacy=0x7fe4a41389b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.794+0000 7fe4ab208640 10 monclient(hunting): authenticate will time out at 1107.575844s 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a97a5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe4a409b8a0 0x7fe4a413c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43068/0 (socket says 172.21.15.63:43068) 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a8fa4640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe4a40a08a0 0x7fe4a409ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55628/0 (socket says 172.21.15.63:55628) 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a97a5640 1 -- 172.21.15.63:0/3291478546 learned_addr learned my addr 172.21.15.63:0/3291478546 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492186278 0 0) 0x7fe4a409e970 con 0x7fe4a409b8a0 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient(hunting): my global_id is 5214 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe48c003530 con 0x7fe4a409b8a0 2024-09-16T05:37:30.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3988110790 0 0) 0x7fe4a409d620 con 0x7fe4a40a08a0 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient(hunting): my global_id is 5201 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4a409e970 con 0x7fe4a40a08a0 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3518897473 0 0) 0x7fe4a409e970 con 0x7fe4a40a08a0 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 >> v1:172.21.15.63:6789/0 conn(0x7fe4a409b450 legacy=0x7fe4a41389b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 >> v1:172.21.15.6:6789/0 conn(0x7fe4a409b8a0 legacy=0x7fe4a413c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 monclient: found mon.noname-b 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:30.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a409d900 con 0x7fe4a40a08a0 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe494001600 con 0x7fe4a40a08a0 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: dump: 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.807 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 5 monclient: authenticate success, global_id 5201 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4940018f0 con 0x7fe4a40a08a0 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 1 -- 172.21.15.63:0/3291478546 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe494001cc0 con 0x7fe4a40a08a0 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a37fe640 10 monclient: dump: 2024-09-16T05:37:30.808 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 4 set_mon_vals no callback set 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient: shutdown 2024-09-16T05:37:30.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- 172.21.15.63:0/3291478546 >> v1:172.21.15.6:6790/0 conn(0x7fe4a40a08a0 legacy=0x7fe4a409ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- 172.21.15.63:0/3291478546 shutdown_connections 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- 172.21.15.63:0/3291478546 >> 172.21.15.63:0/3291478546 conn(0x7fe4a40a1c90 msgr2=0x7fe4a40a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- 172.21.15.63:0/3291478546 shutdown_connections 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- 172.21.15.63:0/3291478546 wait complete. 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient: build_initial_monmap 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient: monmap: 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.810 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:30.811 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:30.811 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:30.811 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 Processor -- start 2024-09-16T05:37:30.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- start start 2024-09-16T05:37:30.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient: init 2024-09-16T05:37:30.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:30.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:30.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient(hunting): picked mon.noname-c con 0x7fe4a409b450 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient(hunting): picked mon.noname-b con 0x7fe4a409b8a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient(hunting): picked mon.noname-a con 0x7fe4a40a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4a409d070 con 0x7fe4a40a08a0 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4a4139c80 con 0x7fe4a409b8a0 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4a406e980 con 0x7fe4a409b450 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4ab208640 10 monclient(hunting): authenticate will time out at 1107.579836s 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.798+0000 7fe4a3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe4a409b8a0 legacy=0x7fe4a406e270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a8fa4640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe4a409b450 0x7fe4a406db60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55638/0 (socket says 172.21.15.63:55638) 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a97a5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe4a40a08a0 0x7fe4a41d3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43074/0 (socket says 172.21.15.63:43074) 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a8fa4640 1 -- 172.21.15.63:0/1583036587 learned_addr learned my addr 172.21.15.63:0/1583036587 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:30.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 577087270 0 0) 0x7fe4a409d070 con 0x7fe4a40a08a0 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): my global_id is 5217 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe490003530 con 0x7fe4a40a08a0 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642492802 0 0) 0x7fe4a406e980 con 0x7fe4a409b450 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): my global_id is 5204 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4a409d070 con 0x7fe4a409b450 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3755296468 0 0) 0x7fe490003530 con 0x7fe4a40a08a0 2024-09-16T05:37:30.817 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4a406e980 con 0x7fe4a40a08a0 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe49c0014c0 con 0x7fe4a40a08a0 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): dump: 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.818 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2943799707 0 0) 0x7fe4a409d070 con 0x7fe4a409b450 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe490003530 con 0x7fe4a409b450 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe494001680 con 0x7fe4a409b450 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient(hunting): dump: 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.819 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3176613423 0 0) 0x7fe4a406e980 con 0x7fe4a40a08a0 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 >> v1:172.21.15.6:6790/0 conn(0x7fe4a409b450 legacy=0x7fe4a406db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 >> v1:172.21.15.63:6789/0 conn(0x7fe4a409b8a0 legacy=0x7fe4a406e270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 monclient: found mon.a 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:30.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a406eea0 con 0x7fe4a40a08a0 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 5 monclient: authenticate success, global_id 5217 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 10 monclient: _renew_subs 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a40a3f30 con 0x7fe4a40a08a0 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 10 monclient: _renew_subs 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4a406f430 con 0x7fe4a40a08a0 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe49c004b70 con 0x7fe4a40a08a0 2024-09-16T05:37:30.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe49c004ec0 con 0x7fe4a40a08a0 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a2ffd640 4 set_mon_vals no callback set 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 10 monclient: dump: 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:30.822 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4ab208640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe484005090 con 0x7fe4a40a08a0 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe49c00fc60 con 0x7fe4a40a08a0 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(82..82 src has 1..82) ==== 5253+0+0 (unknown 857999504 0 0) 0x7fe49c044a50 con 0x7fe4a40a08a0 2024-09-16T05:37:30.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.802+0000 7fe4a8fa4640 10 monclient: get_auth_request con 0x7fe490036a70 auth_method 0 2024-09-16T05:37:30.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.814+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe49c015700 con 0x7fe4a40a08a0 2024-09-16T05:37:30.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.814+0000 7fe4a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:30.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.814+0000 7fe4a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:30.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.990+0000 7fe4ab208640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:30.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.990+0000 7fe4ab208640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:30.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.990+0000 7fe4ab208640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:30.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.990+0000 7fe4ab208640 1 -- 172.21.15.63:0/1583036587 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) -- 0x7fe484005380 con 0x7fe4a40a08a0 2024-09-16T05:37:31.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.998+0000 7fe4a17fa640 1 -- 172.21.15.63:0/1583036587 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v83) ==== 99+0+0 (unknown 4057533042 0 0) 0x7fe49c015000 con 0x7fe4a40a08a0 2024-09-16T05:37:31.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.998+0000 7fe4a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:31.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:30.998+0000 7fe4a17fa640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T05:37:31.001 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T05:37:31.008 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:31.008 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush unlink host2 2024-09-16T05:37:31.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient: get_monmap_and_config 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient: build_initial_monmap 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient: monmap: 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:31.130221+0000 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:31.130221+0000 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:31.131 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:31.132 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:31.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 1 Processor -- start 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 1 -- start start 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient: init 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient(hunting): picked mon.noname-b con 0x7f12b8136ac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient(hunting): picked mon.noname-a con 0x7f12b809fd80 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient(hunting): picked mon.noname-c con 0x7f12b813d0b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b8072cb0 con 0x7f12b809fd80 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b8136670 con 0x7f12b813d0b0 2024-09-16T05:37:31.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b80a14f0 con 0x7f12b8136ac0 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bf77d640 10 monclient(hunting): authenticate will time out at 1107.907325s 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bdd1a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f12b813d0b0 legacy=0x7f12b813f4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bcd18640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12b809fd80 0x7f12b813b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43086/0 (socket says 172.21.15.63:43086) 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.126+0000 7f12bcd18640 1 -- 172.21.15.63:0/2015246180 learned_addr learned my addr 172.21.15.63:0/2015246180 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2782512573 0 0) 0x7f12b80a14f0 con 0x7f12b8136ac0 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient(hunting): my global_id is 5207 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a4003530 con 0x7f12b8136ac0 2024-09-16T05:37:31.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763908149 0 0) 0x7f12a8001120 con 0x7f12b809fd80 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient(hunting): my global_id is 5220 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12b80a14f0 con 0x7f12b809fd80 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2406656200 0 0) 0x7f12a4003530 con 0x7f12b8136ac0 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 >> v1:172.21.15.63:6789/0 conn(0x7f12b813d0b0 legacy=0x7f12b813f4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 >> v1:172.21.15.6:6789/0 conn(0x7f12b809fd80 legacy=0x7f12b813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 monclient: found mon.noname-b 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:31.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b80a0630 con 0x7f12b8136ac0 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: _finish_auth 0 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12b4001550 con 0x7f12b8136ac0 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: dump: 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:31.138 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 5 monclient: authenticate success, global_id 5207 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12b40017b0 con 0x7f12b8136ac0 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 1 -- 172.21.15.63:0/2015246180 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12b4001b80 con 0x7f12b8136ac0 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:31.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12affff640 10 monclient: dump: 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 4 set_mon_vals no callback set 2024-09-16T05:37:31.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient: shutdown 2024-09-16T05:37:31.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:31.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/2015246180 >> v1:172.21.15.6:6790/0 conn(0x7f12b8136ac0 legacy=0x7f12b8136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:31.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/2015246180 shutdown_connections 2024-09-16T05:37:31.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/2015246180 >> 172.21.15.63:0/2015246180 conn(0x7f12b80a1c90 msgr2=0x7f12b80a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:31.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/2015246180 shutdown_connections 2024-09-16T05:37:31.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/2015246180 wait complete. 2024-09-16T05:37:31.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient: build_initial_monmap 2024-09-16T05:37:31.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient: monmap: 2024-09-16T05:37:31.145 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:31.145 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:31.145 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:31.145 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 Processor -- start 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- start start 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient: init 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:31.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient(hunting): picked mon.noname-b con 0x7f12b809fd80 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient(hunting): picked mon.noname-c con 0x7f12b8136ac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient(hunting): picked mon.noname-a con 0x7f12b813d0b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b811d0c0 con 0x7f12b813d0b0 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bd519640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f12b809fd80 legacy=0x7f12b81339f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b811f3b0 con 0x7f12b809fd80 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b813c8b0 con 0x7f12b8136ac0 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bf77d640 10 monclient(hunting): authenticate will time out at 1107.911522s 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bdd1a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12b813d0b0 0x7f12b81d40e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43094/0 (socket says 172.21.15.63:43094) 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bcd18640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f12b8136ac0 0x7f12b81d09b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55662/0 (socket says 172.21.15.63:55662) 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.130+0000 7f12bdd1a640 1 -- 172.21.15.63:0/3977081028 learned_addr learned my addr 172.21.15.63:0/3977081028 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940418104 0 0) 0x7f12b813c8b0 con 0x7f12b8136ac0 2024-09-16T05:37:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): my global_id is 5210 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f129c003530 con 0x7f12b8136ac0 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 380035261 0 0) 0x7f12b811d0c0 con 0x7f12b813d0b0 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): my global_id is 5223 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12b813c8b0 con 0x7f12b813d0b0 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 21716420 0 0) 0x7f129c003530 con 0x7f12b8136ac0 2024-09-16T05:37:31.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12b811d0c0 con 0x7f12b8136ac0 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12a8001930 con 0x7f12b8136ac0 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): dump: 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1705030488 0 0) 0x7f12b813c8b0 con 0x7f12b813d0b0 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f129c003530 con 0x7f12b813d0b0 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12b0004300 con 0x7f12b813d0b0 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient(hunting): dump: 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2062678468 0 0) 0x7f12b811d0c0 con 0x7f12b8136ac0 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 >> v1:172.21.15.63:6789/0 conn(0x7f12b809fd80 legacy=0x7f12b81339f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 >> v1:172.21.15.6:6789/0 conn(0x7f12b813d0b0 legacy=0x7f12b81d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 monclient: found mon.c 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b809da40 con 0x7f12b8136ac0 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 5 monclient: authenticate success, global_id 5210 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 10 monclient: _renew_subs 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12b809c0f0 con 0x7f12b8136ac0 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 10 monclient: _renew_subs 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f12b8134100 con 0x7f12b8136ac0 2024-09-16T05:37:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12a80047f0 con 0x7f12b8136ac0 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12a8004b40 con 0x7f12b8136ac0 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 10 monclient: dump: 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:31.153 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12af7fe640 4 set_mon_vals no callback set 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1298005090 con 0x7f12b8136ac0 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f12a8004da0 con 0x7f12b8136ac0 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(83..83 src has 1..83) ==== 5253+0+0 (unknown 1441598717 0 0) 0x7f12a8044690 con 0x7f12b8136ac0 2024-09-16T05:37:31.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.134+0000 7f12bd519640 10 monclient: get_auth_request con 0x7f129c036a30 auth_method 0 2024-09-16T05:37:31.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.146+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f12b8072cb0 con 0x7f12b8136ac0 2024-09-16T05:37:31.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.146+0000 7f12adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:31.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.146+0000 7f12adffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:31.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.314+0000 7f12bf77d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T05:37:31.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.314+0000 7f12bf77d640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T05:37:31.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.314+0000 7f12bf77d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:31.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:31.314+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) -- 0x7f1298005380 con 0x7f12b8136ac0 2024-09-16T05:37:32.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:32.114+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v84) ==== 128+0+0 (unknown 3030795535 0 0) 0x7f12a80152e0 con 0x7f12b8136ac0 2024-09-16T05:37:32.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:32.114+0000 7f12adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T05:37:32.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:32.114+0000 7f12adffb640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T05:37:32.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:32.222+0000 7f12bf77d640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T05:37:32.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:32.222+0000 7f12bf77d640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T05:37:32.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:32.222+0000 7f12bf77d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:32.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:32.222+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) -- 0x7f12980010a0 con 0x7f12b8136ac0 2024-09-16T05:37:33.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.126+0000 7f12adffb640 1 -- 172.21.15.63:0/3977081028 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v85) ==== 128+0+0 (unknown 3711208365 0 0) 0x7f12a801d790 con 0x7f12b8136ac0 2024-09-16T05:37:33.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.126+0000 7f12adffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T05:37:33.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.126+0000 7f12adffb640 10 monclient: _finish_command 3 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T05:37:33.132 INFO:tasks.workunit.client.0.smithi063.stderr:removed item id -16 name 'host2' from crush map 2024-09-16T05:37:33.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 >> v1:172.21.15.63:6812/3643207211 conn(0x7f129c036a30 legacy=0x7f129c038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:33.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.130+0000 7f12bf77d640 10 monclient: shutdown 2024-09-16T05:37:33.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.130+0000 7f12bf77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:33.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 >> v1:172.21.15.6:6790/0 conn(0x7f12b8136ac0 legacy=0x7f12b81d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:33.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 shutdown_connections 2024-09-16T05:37:33.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 >> 172.21.15.63:0/3977081028 conn(0x7f12b80a1c90 msgr2=0x7f12b80a3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:33.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 shutdown_connections 2024-09-16T05:37:33.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.130+0000 7f12bf77d640 1 -- 172.21.15.63:0/3977081028 wait complete. 2024-09-16T05:37:33.152 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient: get_monmap_and_config 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient: build_initial_monmap 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient: monmap: 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:33.270423+0000 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:33.270423+0000 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:33.271 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:33.272 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:33.272 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:33.272 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:33.272 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:33.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 1 Processor -- start 2024-09-16T05:37:33.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 1 -- start start 2024-09-16T05:37:33.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient: init 2024-09-16T05:37:33.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:33.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:33.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient(hunting): picked mon.noname-a con 0x7f61e809c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient(hunting): picked mon.noname-b con 0x7f61e80a09c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient(hunting): picked mon.noname-c con 0x7f61e809d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e80a0ed0 con 0x7f61e809c300 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e809d950 con 0x7f61e809d3a0 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e809ebd0 con 0x7f61e80a09c0 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61f0032640 10 monclient(hunting): authenticate will time out at 1110.047940s 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.266+0000 7f61ee5cf640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f61e809d3a0 legacy=0x7f61e813c150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:33.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ed5cd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f61e80a09c0 0x7f61e8138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:55666/0 (socket says 172.21.15.63:55666) 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ed5cd640 1 -- 172.21.15.63:0/1366878362 learned_addr learned my addr 172.21.15.63:0/1366878362 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4137135388 0 0) 0x7f61e809ebd0 con 0x7f61e80a09c0 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient(hunting): my global_id is 5213 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61d0003530 con 0x7f61e80a09c0 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3350936279 0 0) 0x7f61e80a0ed0 con 0x7f61e809c300 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:33.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient(hunting): my global_id is 5226 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e809ebd0 con 0x7f61e809c300 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 488677893 0 0) 0x7f61e809ebd0 con 0x7f61e809c300 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 >> v1:172.21.15.6:6790/0 conn(0x7f61e80a09c0 legacy=0x7f61e8138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 >> v1:172.21.15.63:6789/0 conn(0x7f61e809d3a0 legacy=0x7f61e813c150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 monclient: found mon.noname-a 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e80a2470 con 0x7f61e809c300 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: _finish_auth 0 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:33.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f61d80018b0 con 0x7f61e809c300 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: dump: 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:33.279 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 5 monclient: authenticate success, global_id 5226 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61d8001b90 con 0x7f61e809c300 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 1 -- 172.21.15.63:0/1366878362 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f61d8001fa0 con 0x7f61e809c300 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 4 set_mon_vals no callback set 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ecdcc640 10 monclient: dump: 2024-09-16T05:37:33.280 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient: shutdown 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- 172.21.15.63:0/1366878362 >> v1:172.21.15.6:6789/0 conn(0x7f61e809c300 legacy=0x7f61e809c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- 172.21.15.63:0/1366878362 shutdown_connections 2024-09-16T05:37:33.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- 172.21.15.63:0/1366878362 >> 172.21.15.63:0/1366878362 conn(0x7f61e80a1e90 msgr2=0x7f61e8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:33.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- 172.21.15.63:0/1366878362 shutdown_connections 2024-09-16T05:37:33.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- 172.21.15.63:0/1366878362 wait complete. 2024-09-16T05:37:33.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient: build_initial_monmap 2024-09-16T05:37:33.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient: monmap: 2024-09-16T05:37:33.286 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:33.286 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:33.286 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:33.286 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:33.286 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:33.286 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:33.287 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:33.287 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:33.287 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:33.287 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:33.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 Processor -- start 2024-09-16T05:37:33.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- start start 2024-09-16T05:37:33.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient: init 2024-09-16T05:37:33.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient(hunting): picked mon.noname-a con 0x7f61e809c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient(hunting): picked mon.noname-b con 0x7f61e809d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient(hunting): picked mon.noname-c con 0x7f61e8134170 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e80a45f0 con 0x7f61e809c300 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e8121140 con 0x7f61e809d3a0 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61e8121420 con 0x7f61e8134170 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61f0032640 10 monclient(hunting): authenticate will time out at 1110.051730s 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.270+0000 7f61ed5cd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f61e809d3a0 legacy=0x7f61e81d02e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:33.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61eddce640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61e809c300 0x7f61e8133a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:43126/0 (socket says 172.21.15.63:43126) 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61eddce640 1 -- 172.21.15.63:0/2882787745 learned_addr learned my addr 172.21.15.63:0/2882787745 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661537167 0 0) 0x7f61e80a2470 con 0x7f61e809c300 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): my global_id is 5229 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61d4003530 con 0x7f61e809c300 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 231876597 0 0) 0x7f61e8121420 con 0x7f61e8134170 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:33.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): my global_id is 5216 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e80a2470 con 0x7f61e8134170 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1681207499 0 0) 0x7f61d4003530 con 0x7f61e809c300 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61e8121420 con 0x7f61e809c300 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f61d8003a20 con 0x7f61e809c300 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): dump: 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:33.290 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1261726132 0 0) 0x7f61e80a2470 con 0x7f61e8134170 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61d4003530 con 0x7f61e8134170 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f61e0003f40 con 0x7f61e8134170 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:33.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient(hunting): dump: 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:33.292 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2842483464 0 0) 0x7f61e8121420 con 0x7f61e809c300 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 >> v1:172.21.15.6:6790/0 conn(0x7f61e8134170 legacy=0x7f61e81d3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 >> v1:172.21.15.63:6789/0 conn(0x7f61e809d3a0 legacy=0x7f61e81d02e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 monclient: found mon.a 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e80a40a0 con 0x7f61e809c300 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:33.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 5 monclient: authenticate success, global_id 5229 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 10 monclient: _renew_subs 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f61e809c0f0 con 0x7f61e809c300 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 10 monclient: _renew_subs 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f61e811fa10 con 0x7f61e809c300 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61d8001b90 con 0x7f61e809c300 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:33.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f61d80040d0 con 0x7f61e809c300 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e7fff640 4 set_mon_vals no callback set 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 10 monclient: dump: 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:33.295 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c8005090 con 0x7f61e809c300 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f61d80042f0 con 0x7f61e809c300 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(85..85 src has 1..85) ==== 5205+0+0 (unknown 2950801141 0 0) 0x7f61d8044c00 con 0x7f61e809c300 2024-09-16T05:37:33.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.274+0000 7f61ed5cd640 10 monclient: get_auth_request con 0x7f61d4036990 auth_method 0 2024-09-16T05:37:33.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.286+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f61d8044e20 con 0x7f61e809c300 2024-09-16T05:37:33.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.286+0000 7f61e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:33.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.286+0000 7f61e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:33.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.458+0000 7f61f0032640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T05:37:33.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.458+0000 7f61f0032640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T05:37:33.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.458+0000 7f61f0032640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:33.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:33.458+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) -- 0x7f61c8005380 con 0x7f61e809c300 2024-09-16T05:37:34.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.130+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} v86) ==== 251+0+0 (unknown 3704264822 0 0) 0x7f61e80a45f0 con 0x7f61e809c300 2024-09-16T05:37:34.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.130+0000 7f61e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T05:37:34.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.130+0000 7f61e67fc640 10 monclient: _finish_command 2 = system:0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} 2024-09-16T05:37:34.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.230+0000 7f61f0032640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T05:37:34.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.230+0000 7f61f0032640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T05:37:34.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.230+0000 7f61f0032640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:34.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.230+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) -- 0x7f61c8001170 con 0x7f61e809c300 2024-09-16T05:37:34.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.230+0000 7f61e67fc640 1 -- 172.21.15.63:0/2882787745 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 bucket 'host-for-test' already exists v86) ==== 200+0+0 (unknown 2473766080 0 0) 0x7f61d8015940 con 0x7f61e809c300 2024-09-16T05:37:34.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.230+0000 7f61e67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T05:37:34.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.230+0000 7f61e67fc640 10 monclient: _finish_command 3 = system:0 bucket 'host-for-test' already exists 2024-09-16T05:37:34.236 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'host-for-test' already exists 2024-09-16T05:37:34.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.234+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 >> v1:172.21.15.63:6812/3643207211 conn(0x7f61d4036990 legacy=0x7f61d4038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.234+0000 7f61f0032640 10 monclient: shutdown 2024-09-16T05:37:34.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.234+0000 7f61f0032640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:34.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.234+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 >> v1:172.21.15.6:6789/0 conn(0x7f61e809c300 legacy=0x7f61e8133a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.234+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 shutdown_connections 2024-09-16T05:37:34.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.234+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 >> 172.21.15.63:0/2882787745 conn(0x7f61e80a1e90 msgr2=0x7f61e80a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:34.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.234+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 shutdown_connections 2024-09-16T05:37:34.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.234+0000 7f61f0032640 1 -- 172.21.15.63:0/2882787745 wait complete. 2024-09-16T05:37:34.254 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:34.254 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep host-for-test 2024-09-16T05:37:34.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient: get_monmap_and_config 2024-09-16T05:37:34.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient: build_initial_monmap 2024-09-16T05:37:34.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient: monmap: 2024-09-16T05:37:34.375 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:34.374864+0000 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:34.374864+0000 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:34.376 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 1 Processor -- start 2024-09-16T05:37:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 1 -- start start 2024-09-16T05:37:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient: init 2024-09-16T05:37:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient(hunting): picked mon.noname-c con 0x7f575c09e790 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:34.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient(hunting): picked mon.noname-b con 0x7f575c09fd70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient(hunting): picked mon.noname-a con 0x7f575c1394f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575c09c500 con 0x7f575c1394f0 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575c0a1260 con 0x7f575c09e790 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575c09c9c0 con 0x7f575c09fd70 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f5761096640 10 monclient(hunting): authenticate will time out at 1111.151879s 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.370+0000 7f575ad9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f575c09e790 legacy=0x7f575c09eb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f575a59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f575c09fd70 0x7f575c137dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48434/0 (socket says 172.21.15.63:48434) 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f575a59c640 1 -- 172.21.15.63:0/876311683 learned_addr learned my addr 172.21.15.63:0/876311683 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3211394322 0 0) 0x7f575c09c500 con 0x7f575c1394f0 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:34.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient(hunting): my global_id is 5232 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5744003450 con 0x7f575c1394f0 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2803437932 0 0) 0x7f575c09c9c0 con 0x7f575c09fd70 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient(hunting): my global_id is 5219 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f575c09c500 con 0x7f575c09fd70 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1628553519 0 0) 0x7f575c09c500 con 0x7f575c09fd70 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 >> v1:172.21.15.63:6789/0 conn(0x7f575c09e790 legacy=0x7f575c09eb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:34.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 >> v1:172.21.15.6:6789/0 conn(0x7f575c1394f0 legacy=0x7f575c13b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 monclient: found mon.noname-b 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c0a0180 con 0x7f575c09fd70 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: _finish_auth 0 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f57500014b0 con 0x7f575c09fd70 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 5 monclient: authenticate success, global_id 5219 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:34.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: dump: 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:34.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f57500017a0 con 0x7f575c09fd70 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 4 set_mon_vals no callback set 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 1 -- 172.21.15.63:0/876311683 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5750001b70 con 0x7f575c09fd70 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5759d9b640 10 monclient: dump: 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.384 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient: shutdown 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- 172.21.15.63:0/876311683 >> v1:172.21.15.6:6790/0 conn(0x7f575c09fd70 legacy=0x7f575c137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- 172.21.15.63:0/876311683 shutdown_connections 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- 172.21.15.63:0/876311683 >> 172.21.15.63:0/876311683 conn(0x7f575c0a1b60 msgr2=0x7f575c0a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- 172.21.15.63:0/876311683 shutdown_connections 2024-09-16T05:37:34.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- 172.21.15.63:0/876311683 wait complete. 2024-09-16T05:37:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient: build_initial_monmap 2024-09-16T05:37:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient: monmap: 2024-09-16T05:37:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:34.389 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 Processor -- start 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- start start 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient: init 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:34.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient(hunting): picked mon.noname-c con 0x7f575c09e790 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient(hunting): picked mon.noname-b con 0x7f575c09fd70 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient(hunting): picked mon.noname-a con 0x7f575c1394f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575c0709c0 con 0x7f575c1394f0 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575c071820 con 0x7f575c09fd70 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f575a59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f575c09fd70 legacy=0x7f575c0702b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575c072710 con 0x7f575c09e790 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.374+0000 7f5761096640 10 monclient(hunting): authenticate will time out at 1111.155879s 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f575ad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f575c09e790 0x7f575c06fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48440/0 (socket says 172.21.15.63:48440) 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f575ad9d640 1 -- 172.21.15.63:0/3862163904 learned_addr learned my addr 172.21.15.63:0/3862163904 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567855500 0 0) 0x7f575c0709c0 con 0x7f575c1394f0 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:34.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): my global_id is 5235 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5748003530 con 0x7f575c1394f0 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2382326483 0 0) 0x7f575c072710 con 0x7f575c09e790 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): my global_id is 5222 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f575c0709c0 con 0x7f575c09e790 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4061953664 0 0) 0x7f575c0709c0 con 0x7f575c09e790 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f575c072710 con 0x7f575c09e790 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2279611704 0 0) 0x7f5748003530 con 0x7f575c1394f0 2024-09-16T05:37:34.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f575c0709c0 con 0x7f575c1394f0 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f574c0040e0 con 0x7f575c09e790 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): dump: 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.393 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5754001c20 con 0x7f575c1394f0 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient(hunting): dump: 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.394 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2440352083 0 0) 0x7f575c072710 con 0x7f575c09e790 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 >> v1:172.21.15.63:6789/0 conn(0x7f575c09fd70 legacy=0x7f575c0702b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 >> v1:172.21.15.6:6789/0 conn(0x7f575c1394f0 legacy=0x7f575c1d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 monclient: found mon.c 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c11b610 con 0x7f575c09e790 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:34.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 5 monclient: authenticate success, global_id 5222 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 10 monclient: _renew_subs 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f575c071120 con 0x7f575c09e790 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 10 monclient: _renew_subs 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f575c137fe0 con 0x7f575c09e790 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f574c004340 con 0x7f575c09e790 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f575959a640 4 set_mon_vals no callback set 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f574c004690 con 0x7f575c09e790 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:34.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 10 monclient: dump: 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:34.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:34.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f573c005090 con 0x7f575c09e790 2024-09-16T05:37:34.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f574c0048b0 con 0x7f575c09e790 2024-09-16T05:37:34.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(86..86 src has 1..86) ==== 5730+0+0 (unknown 2662021698 0 0) 0x7f574c0448d0 con 0x7f575c09e790 2024-09-16T05:37:34.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.378+0000 7f575a59c640 10 monclient: get_auth_request con 0x7f5748036a10 auth_method 0 2024-09-16T05:37:34.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.390+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f574c015280 con 0x7f575c09e790 2024-09-16T05:37:34.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.390+0000 7f57437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:34.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.390+0000 7f57437fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:34.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.562+0000 7f5761096640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:34.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.562+0000 7f5761096640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:34.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.562+0000 7f5761096640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.562+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f573c005b90 con 0x7f575c09e790 2024-09-16T05:37:34.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.562+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v86) ==== 56+0+1500 (unknown 139406809 0 1721953902) 0x7f574c031e20 con 0x7f575c09e790 2024-09-16T05:37:34.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.562+0000 7f57437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:34.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.562+0000 7f57437fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:34.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.662+0000 7f5761096640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:34.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.662+0000 7f5761096640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:34.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.662+0000 7f5761096640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.662+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f573c006700 con 0x7f575c09e790 2024-09-16T05:37:34.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.662+0000 7f57437fe640 1 -- 172.21.15.63:0/3862163904 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v86) ==== 56+0+1500 (unknown 139406809 0 1721953902) 0x7f573c006700 con 0x7f575c09e790 2024-09-16T05:37:34.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.662+0000 7f57437fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:34.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.662+0000 7f57437fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:34.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.666+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5748036a10 legacy=0x7f5748038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.666+0000 7f5761096640 10 monclient: shutdown 2024-09-16T05:37:34.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.666+0000 7f5761096640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:34.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.666+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 >> v1:172.21.15.6:6790/0 conn(0x7f575c09e790 legacy=0x7f575c06fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.666+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 shutdown_connections 2024-09-16T05:37:34.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.666+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 >> 172.21.15.63:0/3862163904 conn(0x7f575c0a1b60 msgr2=0x7f575c0a3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:34.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.666+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 shutdown_connections 2024-09-16T05:37:34.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.666+0000 7f5761096640 1 -- 172.21.15.63:0/3862163904 wait complete. 2024-09-16T05:37:34.685 INFO:tasks.workunit.client.0.smithi063.stdout:-11 0 host host-for-test 2024-09-16T05:37:34.685 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep rack-for-test 2024-09-16T05:37:34.685 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.798+0000 7fe73ccca640 10 monclient: get_monmap_and_config 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.798+0000 7fe73ccca640 10 monclient: build_initial_monmap 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.798+0000 7fe73ccca640 10 monclient: monmap: 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:34.803461+0000 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:34.803461+0000 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:34.804 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.805 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:34.805 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:34.805 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:34.805 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.798+0000 7fe73ccca640 1 Processor -- start 2024-09-16T05:37:34.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.798+0000 7fe73ccca640 1 -- start start 2024-09-16T05:37:34.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.798+0000 7fe73ccca640 10 monclient: init 2024-09-16T05:37:34.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient(hunting): picked mon.noname-c con 0x7fe73809acb0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient(hunting): picked mon.noname-b con 0x7fe7380a13f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient(hunting): picked mon.noname-a con 0x7fe73809b7a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7380a0fd0 con 0x7fe73809b7a0 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe73809f110 con 0x7fe73809acb0 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe73809ec90 con 0x7fe7380a13f0 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient(hunting): authenticate will time out at 1111.580429s 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe73809acb0 legacy=0x7fe73809b090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe735d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe7380a13f0 0x7fe738139ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48456/0 (socket says 172.21.15.63:48456) 2024-09-16T05:37:34.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe735d9b640 1 -- 172.21.15.63:0/381751992 learned_addr learned my addr 172.21.15.63:0/381751992 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4111231459 0 0) 0x7fe73809ec90 con 0x7fe7380a13f0 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient(hunting): my global_id is 5225 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe720003530 con 0x7fe7380a13f0 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310767528 0 0) 0x7fe7380a0fd0 con 0x7fe73809b7a0 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient(hunting): my global_id is 5238 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe73809ec90 con 0x7fe73809b7a0 2024-09-16T05:37:34.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4114184915 0 0) 0x7fe73809ec90 con 0x7fe73809b7a0 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 >> v1:172.21.15.6:6790/0 conn(0x7fe7380a13f0 legacy=0x7fe738139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 >> v1:172.21.15.63:6789/0 conn(0x7fe73809acb0 legacy=0x7fe73809b090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: _finish_hunting 0 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 monclient: found mon.noname-a 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7380a0730 con 0x7fe73809b7a0 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: _finish_auth 0 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: _check_auth_tickets 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe730001980 con 0x7fe73809b7a0 2024-09-16T05:37:34.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: dump: 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 5 monclient: authenticate success, global_id 5238 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe730001be0 con 0x7fe73809b7a0 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 1 -- 172.21.15.63:0/381751992 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe730001fb0 con 0x7fe73809b7a0 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73559a640 10 monclient: dump: 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.813 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 4 set_mon_vals no callback set 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient: shutdown 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- 172.21.15.63:0/381751992 >> v1:172.21.15.6:6789/0 conn(0x7fe73809b7a0 legacy=0x7fe73813d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- 172.21.15.63:0/381751992 shutdown_connections 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- 172.21.15.63:0/381751992 >> 172.21.15.63:0/381751992 conn(0x7fe7380a1b60 msgr2=0x7fe7380a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:34.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- 172.21.15.63:0/381751992 shutdown_connections 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- 172.21.15.63:0/381751992 wait complete. 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient: build_initial_monmap 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient: monmap: 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:34.815 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 Processor -- start 2024-09-16T05:37:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 1 -- start start 2024-09-16T05:37:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.802+0000 7fe73ccca640 10 monclient: init 2024-09-16T05:37:34.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:34.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:34.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient(hunting): picked mon.noname-c con 0x7fe73809acb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient(hunting): picked mon.noname-b con 0x7fe73809b7a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient(hunting): picked mon.noname-a con 0x7fe7380a13f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe738121dc0 con 0x7fe7380a13f0 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe735d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe73809b7a0 legacy=0x7fe738120000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7381220a0 con 0x7fe73809b7a0 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe738120710 con 0x7fe73809acb0 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient(hunting): authenticate will time out at 1111.584590s 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe73809acb0 0x7fe73811f8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48458/0 (socket says 172.21.15.63:48458) 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73659c640 1 -- 172.21.15.63:0/3011942683 learned_addr learned my addr 172.21.15.63:0/3011942683 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169854877 0 0) 0x7fe738120710 con 0x7fe73809acb0 2024-09-16T05:37:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): my global_id is 5228 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe718003530 con 0x7fe73809acb0 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141921453 0 0) 0x7fe738121dc0 con 0x7fe7380a13f0 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): my global_id is 5241 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe738120710 con 0x7fe7380a13f0 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 991166372 0 0) 0x7fe718003530 con 0x7fe73809acb0 2024-09-16T05:37:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe738121dc0 con 0x7fe73809acb0 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1549493365 0 0) 0x7fe738120710 con 0x7fe7380a13f0 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe718003530 con 0x7fe7380a13f0 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe728004160 con 0x7fe73809acb0 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe730001880 con 0x7fe7380a13f0 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2686638488 0 0) 0x7fe738121dc0 con 0x7fe73809acb0 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 >> v1:172.21.15.63:6789/0 conn(0x7fe73809b7a0 legacy=0x7fe738120000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 >> v1:172.21.15.6:6789/0 conn(0x7fe7380a13f0 legacy=0x7fe7381d3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 monclient: found mon.c 2024-09-16T05:37:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe738120bf0 con 0x7fe73809acb0 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 5 monclient: authenticate success, global_id 5228 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient: _renew_subs 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe73809e2d0 con 0x7fe73809acb0 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient: _renew_subs 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe738135330 con 0x7fe73809acb0 2024-09-16T05:37:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe7280043c0 con 0x7fe73809acb0 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe728004790 con 0x7fe73809acb0 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe734d99640 4 set_mon_vals no callback set 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 10 monclient: dump: 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe71c005090 con 0x7fe73809acb0 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.806+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe7280049b0 con 0x7fe73809acb0 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.810+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(86..86 src has 1..86) ==== 5730+0+0 (unknown 2662021698 0 0) 0x7fe7280449b0 con 0x7fe73809acb0 2024-09-16T05:37:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.810+0000 7fe735d9b640 10 monclient: get_auth_request con 0x7fe718036a10 auth_method 0 2024-09-16T05:37:34.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.822+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe728015420 con 0x7fe73809acb0 2024-09-16T05:37:34.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.822+0000 7fe726ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:34.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.822+0000 7fe726ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:34.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.986+0000 7fe73ccca640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:34.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.986+0000 7fe73ccca640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:34.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.986+0000 7fe73ccca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:34.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.986+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fe71c005b90 con 0x7fe73809acb0 2024-09-16T05:37:34.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.986+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v86) ==== 56+0+1500 (unknown 139406809 0 1721953902) 0x7fe728014d20 con 0x7fe73809acb0 2024-09-16T05:37:34.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.986+0000 7fe726ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:34.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:34.986+0000 7fe726ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:35.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:35.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:35.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fe71c006700 con 0x7fe73809acb0 2024-09-16T05:37:35.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe726ffd640 1 -- 172.21.15.63:0/3011942683 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v86) ==== 56+0+1500 (unknown 139406809 0 1721953902) 0x7fe71c006700 con 0x7fe73809acb0 2024-09-16T05:37:35.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe726ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:35.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe726ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:35.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe718036a10 legacy=0x7fe718038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 10 monclient: shutdown 2024-09-16T05:37:35.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:35.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 >> v1:172.21.15.6:6790/0 conn(0x7fe73809acb0 legacy=0x7fe73811f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 shutdown_connections 2024-09-16T05:37:35.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 >> 172.21.15.63:0/3011942683 conn(0x7fe7380a1b60 msgr2=0x7fe7380a3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:35.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 shutdown_connections 2024-09-16T05:37:35.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.082+0000 7fe73ccca640 1 -- 172.21.15.63:0/3011942683 wait complete. 2024-09-16T05:37:35.102 INFO:tasks.workunit.client.0.smithi063.stdout:-12 0 rack rack-for-test 2024-09-16T05:37:35.102 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:35.102 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep root-for-test 2024-09-16T05:37:35.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient: get_monmap_and_config 2024-09-16T05:37:35.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient: build_initial_monmap 2024-09-16T05:37:35.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient: monmap: 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:35.214848+0000 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:35.214848+0000 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:35.216 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 1 Processor -- start 2024-09-16T05:37:35.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 1 -- start start 2024-09-16T05:37:35.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient: init 2024-09-16T05:37:35.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:35.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:35.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient(hunting): picked mon.noname-b con 0x7f80ac09ee70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:35.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient(hunting): picked mon.noname-a con 0x7f80ac0a45c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:35.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient(hunting): picked mon.noname-c con 0x7f80ac09f960 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80ac0a4ee0 con 0x7f80ac0a45c0 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80ac1413d0 con 0x7f80ac09f960 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80ac1425b0 con 0x7f80ac09ee70 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b28e1640 10 monclient(hunting): authenticate will time out at 1111.991797s 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.210+0000 7f80b0e7e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f80ac09f960 legacy=0x7f80ac13eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80abfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f80ac09ee70 0x7f80ac0a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48462/0 (socket says 172.21.15.63:48462) 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80abfff640 1 -- 172.21.15.63:0/16115203 learned_addr learned my addr 172.21.15.63:0/16115203 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80ab7fe640 1 --1- 172.21.15.63:0/16115203 >> v1:172.21.15.6:6789/0 conn(0x7f80ac0a45c0 0x7f80ac09f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53310/0 (socket says 172.21.15.63:53310) 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1692059718 0 0) 0x7f80ac1425b0 con 0x7f80ac09ee70 2024-09-16T05:37:35.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient(hunting): my global_id is 5231 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8094003530 con 0x7f80ac09ee70 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720859730 0 0) 0x7f80ac0a4ee0 con 0x7f80ac0a45c0 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient(hunting): my global_id is 5244 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80ac1425b0 con 0x7f80ac0a45c0 2024-09-16T05:37:35.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3018357698 0 0) 0x7f8094003530 con 0x7f80ac09ee70 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 >> v1:172.21.15.63:6789/0 conn(0x7f80ac09f960 legacy=0x7f80ac13eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 >> v1:172.21.15.6:6789/0 conn(0x7f80ac0a45c0 legacy=0x7f80ac09f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 monclient: found mon.noname-b 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ac143790 con 0x7f80ac09ee70 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f80a40019a0 con 0x7f80ac09ee70 2024-09-16T05:37:35.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: dump: 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.223 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 5 monclient: authenticate success, global_id 5231 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f80a4001c00 con 0x7f80ac09ee70 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 4 set_mon_vals no callback set 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 1 -- 172.21.15.63:0/16115203 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f80a4001f50 con 0x7f80ac09ee70 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80aaffd640 10 monclient: dump: 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.224 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient: shutdown 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- 172.21.15.63:0/16115203 >> v1:172.21.15.6:6790/0 conn(0x7f80ac09ee70 legacy=0x7f80ac0a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- 172.21.15.63:0/16115203 shutdown_connections 2024-09-16T05:37:35.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- 172.21.15.63:0/16115203 >> 172.21.15.63:0/16115203 conn(0x7f80ac0a5d10 msgr2=0x7f80ac0a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:35.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- 172.21.15.63:0/16115203 shutdown_connections 2024-09-16T05:37:35.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- 172.21.15.63:0/16115203 wait complete. 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient: build_initial_monmap 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient: monmap: 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:35.229 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 Processor -- start 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- start start 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient: init 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient(hunting): picked mon.noname-a con 0x7f80ac09ee70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient(hunting): picked mon.noname-b con 0x7f80ac09f960 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient(hunting): picked mon.noname-c con 0x7f80ac0a45c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80ac13c360 con 0x7f80ac09ee70 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80ac1cc4c0 con 0x7f80ac09f960 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80ac143160 con 0x7f80ac0a45c0 2024-09-16T05:37:35.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b28e1640 10 monclient(hunting): authenticate will time out at 1111.995504s 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80ab7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f80ac09f960 legacy=0x7f80ac1cbdb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b0e7e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f80ac0a45c0 0x7f80ac1d3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48476/0 (socket says 172.21.15.63:48476) 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80abfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f80ac09ee70 0x7f80ac1cb6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53318/0 (socket says 172.21.15.63:53318) 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.214+0000 7f80b0e7e640 1 -- 172.21.15.63:0/1238193673 learned_addr learned my addr 172.21.15.63:0/1238193673 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 553741822 0 0) 0x7f80ac143160 con 0x7f80ac0a45c0 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): my global_id is 5234 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8098003530 con 0x7f80ac0a45c0 2024-09-16T05:37:35.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342485478 0 0) 0x7f80ac13c360 con 0x7f80ac09ee70 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): my global_id is 5247 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80ac143160 con 0x7f80ac09ee70 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2378754211 0 0) 0x7f8098003530 con 0x7f80ac0a45c0 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80ac13c360 con 0x7f80ac0a45c0 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f809c004010 con 0x7f80ac0a45c0 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:35.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3919525665 0 0) 0x7f80ac143160 con 0x7f80ac09ee70 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8098003530 con 0x7f80ac09ee70 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f80a40018a0 con 0x7f80ac09ee70 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.234 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2244893960 0 0) 0x7f80ac13c360 con 0x7f80ac0a45c0 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 >> v1:172.21.15.63:6789/0 conn(0x7f80ac09f960 legacy=0x7f80ac1cbdb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 >> v1:172.21.15.6:6789/0 conn(0x7f80ac09ee70 legacy=0x7f80ac1cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 monclient: found mon.c 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ac0a0500 con 0x7f80ac0a45c0 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient: _finish_auth 0 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 5 monclient: authenticate success, global_id 5234 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 10 monclient: _renew_subs 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f80ac0a3140 con 0x7f80ac0a45c0 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 10 monclient: _renew_subs 2024-09-16T05:37:35.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f80ac13ac30 con 0x7f80ac0a45c0 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f809c0041b0 con 0x7f80ac0a45c0 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f809c004580 con 0x7f80ac0a45c0 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a9ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 10 monclient: dump: 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.237 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f808c005090 con 0x7f80ac0a45c0 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f809c0047e0 con 0x7f80ac0a45c0 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(86..86 src has 1..86) ==== 5730+0+0 (unknown 2662021698 0 0) 0x7f809c044820 con 0x7f80ac0a45c0 2024-09-16T05:37:35.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.218+0000 7f80abfff640 10 monclient: get_auth_request con 0x7f8098036a30 auth_method 0 2024-09-16T05:37:35.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.230+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f809c015290 con 0x7f80ac0a45c0 2024-09-16T05:37:35.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.230+0000 7f80a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:35.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.230+0000 7f80a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:35.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.390+0000 7f80b28e1640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:35.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.390+0000 7f80b28e1640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:35.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.390+0000 7f80b28e1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.390+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f808c005b90 con 0x7f80ac0a45c0 2024-09-16T05:37:35.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.390+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v86) ==== 56+0+1500 (unknown 139406809 0 1721953902) 0x7f809c014b90 con 0x7f80ac0a45c0 2024-09-16T05:37:35.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.390+0000 7f80a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:35.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.390+0000 7f80a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:35.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.490+0000 7f80b28e1640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:35.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.490+0000 7f80b28e1640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:35.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.490+0000 7f80b28e1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.490+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f808c006700 con 0x7f80ac0a45c0 2024-09-16T05:37:35.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.490+0000 7f80a8ff9640 1 -- 172.21.15.63:0/1238193673 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v86) ==== 56+0+1500 (unknown 139406809 0 1721953902) 0x7f808c006700 con 0x7f80ac0a45c0 2024-09-16T05:37:35.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.490+0000 7f80a8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:35.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.490+0000 7f80a8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:35.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.494+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8098036a30 legacy=0x7f8098038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.494+0000 7f80b28e1640 10 monclient: shutdown 2024-09-16T05:37:35.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.494+0000 7f80b28e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:35.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.494+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 >> v1:172.21.15.6:6790/0 conn(0x7f80ac0a45c0 legacy=0x7f80ac1d3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.494+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 shutdown_connections 2024-09-16T05:37:35.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.494+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 >> 172.21.15.63:0/1238193673 conn(0x7f80ac0a5d10 msgr2=0x7f80ac0a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:35.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.494+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 shutdown_connections 2024-09-16T05:37:35.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.494+0000 7f80b28e1640 1 -- 172.21.15.63:0/1238193673 wait complete. 2024-09-16T05:37:35.510 INFO:tasks.workunit.client.0.smithi063.stdout:-14 0 root root-for-test 2024-09-16T05:37:35.511 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm host-for-test 2024-09-16T05:37:35.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient: get_monmap_and_config 2024-09-16T05:37:35.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient: build_initial_monmap 2024-09-16T05:37:35.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient: monmap: 2024-09-16T05:37:35.631 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:35.632 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.632 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:35.630939+0000 2024-09-16T05:37:35.632 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:35.630939+0000 2024-09-16T05:37:35.632 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:35.632 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.632 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:35.633 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:35.633 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:35.633 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 1 Processor -- start 2024-09-16T05:37:35.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 1 -- start start 2024-09-16T05:37:35.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient: init 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient(hunting): picked mon.noname-c con 0x7f0a3c09ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient(hunting): picked mon.noname-a con 0x7f0a3c06dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient(hunting): picked mon.noname-b con 0x7f0a3c0a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3c09f800 con 0x7f0a3c06dd30 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3c0a0ab0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3c0a0d90 con 0x7f0a3c0a0210 2024-09-16T05:37:35.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a4250b640 10 monclient(hunting): authenticate will time out at 1112.407760s 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.626+0000 7f0a3bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a3c09ac90 legacy=0x7f0a3c09b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a40aa8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0a3c0a0210 0x7f0a3c13d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48492/0 (socket says 172.21.15.63:48492) 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a40aa8640 1 -- 172.21.15.63:0/927631572 learned_addr learned my addr 172.21.15.63:0/927631572 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3143653631 0 0) 0x7f0a3c0a0d90 con 0x7f0a3c0a0210 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient(hunting): my global_id is 5237 2024-09-16T05:37:35.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a24003530 con 0x7f0a3c0a0210 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217555349 0 0) 0x7f0a3c09f800 con 0x7f0a3c06dd30 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient(hunting): my global_id is 5250 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a3c0a0d90 con 0x7f0a3c06dd30 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 135874241 0 0) 0x7f0a24003530 con 0x7f0a3c0a0210 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 >> v1:172.21.15.63:6789/0 conn(0x7f0a3c09ac90 legacy=0x7f0a3c09b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:35.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 >> v1:172.21.15.6:6789/0 conn(0x7f0a3c06dd30 legacy=0x7f0a3c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 monclient: found mon.noname-b 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c09e770 con 0x7f0a3c0a0210 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: _finish_auth 0 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a340019a0 con 0x7f0a3c0a0210 2024-09-16T05:37:35.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: dump: 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 5 monclient: authenticate success, global_id 5237 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a34001c00 con 0x7f0a3c0a0210 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 1 -- 172.21.15.63:0/927631572 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a34002010 con 0x7f0a3c0a0210 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3affd640 10 monclient: dump: 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.643 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 4 set_mon_vals no callback set 2024-09-16T05:37:35.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient: shutdown 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- 172.21.15.63:0/927631572 >> v1:172.21.15.6:6790/0 conn(0x7f0a3c0a0210 legacy=0x7f0a3c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- 172.21.15.63:0/927631572 shutdown_connections 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- 172.21.15.63:0/927631572 >> 172.21.15.63:0/927631572 conn(0x7f0a3c0a1c90 msgr2=0x7f0a3c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- 172.21.15.63:0/927631572 shutdown_connections 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- 172.21.15.63:0/927631572 wait complete. 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient: build_initial_monmap 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient: monmap: 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:35.645 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 Processor -- start 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- start start 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient: init 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:35.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient(hunting): picked mon.noname-b con 0x7f0a3c06dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient(hunting): picked mon.noname-a con 0x7f0a3c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient(hunting): picked mon.noname-c con 0x7f0a3c0a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3c121ca0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3c120780 con 0x7f0a3c06dd30 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3c120c40 con 0x7f0a3c0a0210 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a3bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0a3c06dd30 legacy=0x7f0a3c1345c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.630+0000 7f0a4250b640 10 monclient(hunting): authenticate will time out at 1112.411661s 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a40aa8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0a3c0a0210 0x7f0a3c1d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48504/0 (socket says 172.21.15.63:48504) 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a40aa8640 1 -- 172.21.15.63:0/2803997820 learned_addr learned my addr 172.21.15.63:0/2803997820 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2523475438 0 0) 0x7f0a3c120c40 con 0x7f0a3c0a0210 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:35.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:35.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:35.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): my global_id is 5240 2024-09-16T05:37:35.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a28003530 con 0x7f0a3c0a0210 2024-09-16T05:37:35.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1259112431 0 0) 0x7f0a3c121ca0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:35.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:35.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:35.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): my global_id is 5253 2024-09-16T05:37:35.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a3c120c40 con 0x7f0a3c09ac90 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3500146172 0 0) 0x7f0a3c120c40 con 0x7f0a3c09ac90 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a3c121ca0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4089601119 0 0) 0x7f0a28003530 con 0x7f0a3c0a0210 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a3c120c40 con 0x7f0a3c0a0210 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a30001840 con 0x7f0a3c09ac90 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.651 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a34001f50 con 0x7f0a3c0a0210 2024-09-16T05:37:35.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient(hunting): dump: 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 170152614 0 0) 0x7f0a3c121ca0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 >> v1:172.21.15.6:6790/0 conn(0x7f0a3c0a0210 legacy=0x7f0a3c1d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 >> v1:172.21.15.63:6789/0 conn(0x7f0a3c06dd30 legacy=0x7f0a3c1345c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 monclient: found mon.a 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c11fc20 con 0x7f0a3c09ac90 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient: _finish_auth 0 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 5 monclient: authenticate success, global_id 5253 2024-09-16T05:37:35.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 10 monclient: _renew_subs 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3c1c90f0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 10 monclient: _renew_subs 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0a3c134ea0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a30001c50 con 0x7f0a3c09ac90 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0a30003a50 con 0x7f0a3c09ac90 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a3a7fc640 4 set_mon_vals no callback set 2024-09-16T05:37:35.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 10 monclient: dump: 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:35.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:35.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a1c005090 con 0x7f0a3c09ac90 2024-09-16T05:37:35.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0a3000e7f0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(86..86 src has 1..86) ==== 5730+0+0 (unknown 2662021698 0 0) 0x7f0a30043770 con 0x7f0a3c09ac90 2024-09-16T05:37:35.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.634+0000 7f0a3bfff640 10 monclient: get_auth_request con 0x7f0a28036a00 auth_method 0 2024-09-16T05:37:35.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.646+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0a300141e0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.646+0000 7f0a38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:35.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.646+0000 7f0a38ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:35.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.818+0000 7f0a4250b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T05:37:35.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.818+0000 7f0a4250b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T05:37:35.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.818+0000 7f0a4250b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:35.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.818+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) -- 0x7f0a1c005380 con 0x7f0a3c09ac90 2024-09-16T05:37:35.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.990+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 removed item id -11 name 'host-for-test' from crush map v87) ==== 140+0+0 (unknown 2667186757 0 0) 0x7f0a30013ae0 con 0x7f0a3c09ac90 2024-09-16T05:37:35.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.990+0000 7f0a38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T05:37:35.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:35.990+0000 7f0a38ff9640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host-for-test' from crush map 2024-09-16T05:37:36.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T05:37:36.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T05:37:36.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:36.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) -- 0x7f0a1c0010a0 con 0x7f0a3c09ac90 2024-09-16T05:37:36.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a38ff9640 1 -- 172.21.15.63:0/2803997820 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 device 'host-for-test' does not appear in the crush map v87) ==== 140+0+0 (unknown 1906055171 0 0) 0x7f0a3001c690 con 0x7f0a3c09ac90 2024-09-16T05:37:36.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a38ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T05:37:36.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a38ff9640 10 monclient: _finish_command 3 = system:0 device 'host-for-test' does not appear in the crush map 2024-09-16T05:37:36.098 INFO:tasks.workunit.client.0.smithi063.stderr:device 'host-for-test' does not appear in the crush map 2024-09-16T05:37:36.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0a28036a00 legacy=0x7f0a28038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:36.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 10 monclient: shutdown 2024-09-16T05:37:36.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:36.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 >> v1:172.21.15.6:6789/0 conn(0x7f0a3c09ac90 legacy=0x7f0a3c1cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:36.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 shutdown_connections 2024-09-16T05:37:36.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 >> 172.21.15.63:0/2803997820 conn(0x7f0a3c0a1c90 msgr2=0x7f0a3c0a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:36.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 shutdown_connections 2024-09-16T05:37:36.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.094+0000 7f0a4250b640 1 -- 172.21.15.63:0/2803997820 wait complete. 2024-09-16T05:37:36.118 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm rack-for-test 2024-09-16T05:37:36.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient: get_monmap_and_config 2024-09-16T05:37:36.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient: build_initial_monmap 2024-09-16T05:37:36.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient: monmap: 2024-09-16T05:37:36.238 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:36.238 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:36.238 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:36.237724+0000 2024-09-16T05:37:36.239 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:36.237724+0000 2024-09-16T05:37:36.239 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:36.239 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:36.239 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:36.239 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:36.239 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:36.239 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:36.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 1 Processor -- start 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 1 -- start start 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient: init 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient(hunting): picked mon.noname-a con 0x7fc1d009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient(hunting): picked mon.noname-c con 0x7fc1d006dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient(hunting): picked mon.noname-b con 0x7fc1d00a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1d009f800 con 0x7fc1d009ac90 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1d00a0ab0 con 0x7fc1d006dd30 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1d00a0d90 con 0x7fc1d00a0210 2024-09-16T05:37:36.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d63f0640 10 monclient(hunting): authenticate will time out at 1113.014645s 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1cf7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc1d006dd30 legacy=0x7fc1d01399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1cffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc1d009ac90 0x7fc1d009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53330/0 (socket says 172.21.15.63:53330) 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1cffff640 1 -- 172.21.15.63:0/609005484 learned_addr learned my addr 172.21.15.63:0/609005484 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1d498d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc1d00a0210 0x7fc1d013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48510/0 (socket says 172.21.15.63:48510) 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1584875630 0 0) 0x7fc1d00a0d90 con 0x7fc1d00a0210 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient(hunting): my global_id is 5243 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1bc003530 con 0x7fc1d00a0210 2024-09-16T05:37:36.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3588801589 0 0) 0x7fc1d009f800 con 0x7fc1d009ac90 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient(hunting): my global_id is 5256 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d00a0d90 con 0x7fc1d009ac90 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1926107970 0 0) 0x7fc1d00a0d90 con 0x7fc1d009ac90 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 >> v1:172.21.15.6:6790/0 conn(0x7fc1d00a0210 legacy=0x7fc1d013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 >> v1:172.21.15.63:6789/0 conn(0x7fc1d006dd30 legacy=0x7fc1d01399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.234+0000 7fc1ceffd640 1 monclient: found mon.noname-a 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:36.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d009e770 con 0x7fc1d009ac90 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1c00014b0 con 0x7fc1d009ac90 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: dump: 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.246 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 5 monclient: authenticate success, global_id 5256 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1c0001810 con 0x7fc1d009ac90 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 1 -- 172.21.15.63:0/609005484 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1c0001c20 con 0x7fc1d009ac90 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:36.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ceffd640 10 monclient: dump: 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:36.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 4 set_mon_vals no callback set 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient: shutdown 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- 172.21.15.63:0/609005484 >> v1:172.21.15.6:6789/0 conn(0x7fc1d009ac90 legacy=0x7fc1d009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- 172.21.15.63:0/609005484 shutdown_connections 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- 172.21.15.63:0/609005484 >> 172.21.15.63:0/609005484 conn(0x7fc1d00a1c90 msgr2=0x7fc1d00a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- 172.21.15.63:0/609005484 shutdown_connections 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- 172.21.15.63:0/609005484 wait complete. 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient: build_initial_monmap 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient: monmap: 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 Processor -- start 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- start start 2024-09-16T05:37:36.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient: init 2024-09-16T05:37:36.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:36.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient(hunting): picked mon.noname-b con 0x7fc1d006dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient(hunting): picked mon.noname-c con 0x7fc1d009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient(hunting): picked mon.noname-a con 0x7fc1d00a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1d0123a80 con 0x7fc1d00a0210 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1d01233c0 con 0x7fc1d006dd30 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1cffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc1d006dd30 legacy=0x7fc1d0134820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1d0124f30 con 0x7fc1d009ac90 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d63f0640 10 monclient(hunting): authenticate will time out at 1113.018567s 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d498d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc1d00a0210 0x7fc1d01d13c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53344/0 (socket says 172.21.15.63:53344) 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1d498d640 1 -- 172.21.15.63:0/431122012 learned_addr learned my addr 172.21.15.63:0/431122012 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:36.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1cf7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc1d009ac90 0x7fc1d01cdc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48516/0 (socket says 172.21.15.63:48516) 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2595015773 0 0) 0x7fc1d0123a80 con 0x7fc1d00a0210 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): my global_id is 5259 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1b0003530 con 0x7fc1d00a0210 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137102420 0 0) 0x7fc1d0124f30 con 0x7fc1d009ac90 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): my global_id is 5246 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d0123a80 con 0x7fc1d009ac90 2024-09-16T05:37:36.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4183147759 0 0) 0x7fc1b0003530 con 0x7fc1d00a0210 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1d0124f30 con 0x7fc1d00a0210 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1c8001c60 con 0x7fc1d00a0210 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): dump: 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 473168020 0 0) 0x7fc1d0123a80 con 0x7fc1d009ac90 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1b0003530 con 0x7fc1d009ac90 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1b8004110 con 0x7fc1d009ac90 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.238+0000 7fc1ccff9640 10 monclient(hunting): dump: 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:36.257 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3338627988 0 0) 0x7fc1d0124f30 con 0x7fc1d00a0210 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 >> v1:172.21.15.6:6790/0 conn(0x7fc1d009ac90 legacy=0x7fc1d01cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 >> v1:172.21.15.63:6789/0 conn(0x7fc1d006dd30 legacy=0x7fc1d0134820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 10 monclient: _finish_hunting 0 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 monclient: found mon.a 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d0124c50 con 0x7fc1d00a0210 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 10 monclient: _finish_auth 0 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 10 monclient: _check_auth_tickets 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 5 monclient: authenticate success, global_id 5259 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 10 monclient: _renew_subs 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d0134f30 con 0x7fc1d00a0210 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 10 monclient: _renew_subs 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:36.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1d01c8da0 con 0x7fc1d00a0210 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1c8002020 con 0x7fc1d00a0210 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc1c8004fc0 con 0x7fc1d00a0210 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ce7fc640 4 set_mon_vals no callback set 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 10 monclient: dump: 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b4005090 con 0x7fc1d00a0210 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc1c800fd60 con 0x7fc1d00a0210 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(87..87 src has 1..87) ==== 5547+0+0 (unknown 3252006535 0 0) 0x7fc1c8044cc0 con 0x7fc1d00a0210 2024-09-16T05:37:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.242+0000 7fc1cffff640 10 monclient: get_auth_request con 0x7fc1b0036a20 auth_method 0 2024-09-16T05:37:36.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.254+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc1c8015790 con 0x7fc1d00a0210 2024-09-16T05:37:36.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.254+0000 7fc1ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:36.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.254+0000 7fc1ccff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.418+0000 7fc1d63f0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T05:37:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.418+0000 7fc1d63f0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T05:37:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.418+0000 7fc1d63f0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:36.418+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) -- 0x7fc1b4005380 con 0x7fc1d00a0210 2024-09-16T05:37:37.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.142+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 removed item id -12 name 'rack-for-test' from crush map v88) ==== 140+0+0 (unknown 595558166 0 0) 0x7fc1c8015090 con 0x7fc1d00a0210 2024-09-16T05:37:37.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.142+0000 7fc1ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T05:37:37.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.142+0000 7fc1ccff9640 10 monclient: _finish_command 2 = system:0 removed item id -12 name 'rack-for-test' from crush map 2024-09-16T05:37:37.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.242+0000 7fc1d63f0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T05:37:37.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.242+0000 7fc1d63f0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T05:37:37.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.242+0000 7fc1d63f0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:37.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.242+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) -- 0x7fc1b40010a0 con 0x7fc1d00a0210 2024-09-16T05:37:37.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1ccff9640 1 -- 172.21.15.63:0/431122012 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 device 'rack-for-test' does not appear in the crush map v88) ==== 140+0+0 (unknown 857013074 0 0) 0x7fc1c80193e0 con 0x7fc1d00a0210 2024-09-16T05:37:37.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1ccff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T05:37:37.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1ccff9640 10 monclient: _finish_command 3 = system:0 device 'rack-for-test' does not appear in the crush map 2024-09-16T05:37:37.249 INFO:tasks.workunit.client.0.smithi063.stderr:device 'rack-for-test' does not appear in the crush map 2024-09-16T05:37:37.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc1b0036a20 legacy=0x7fc1b0038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:37.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1d63f0640 10 monclient: shutdown 2024-09-16T05:37:37.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1d63f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:37.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 >> v1:172.21.15.6:6789/0 conn(0x7fc1d00a0210 legacy=0x7fc1d01d13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:37.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 shutdown_connections 2024-09-16T05:37:37.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 >> 172.21.15.63:0/431122012 conn(0x7fc1d00a1c90 msgr2=0x7fc1d006e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:37.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 shutdown_connections 2024-09-16T05:37:37.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.246+0000 7fc1d63f0640 1 -- 172.21.15.63:0/431122012 wait complete. 2024-09-16T05:37:37.266 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm root-for-test 2024-09-16T05:37:37.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient: get_monmap_and_config 2024-09-16T05:37:37.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient: build_initial_monmap 2024-09-16T05:37:37.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient: monmap: 2024-09-16T05:37:37.387 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:37.387 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:37.388 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:37.386749+0000 2024-09-16T05:37:37.388 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:37.386749+0000 2024-09-16T05:37:37.388 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:37.388 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:37.388 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:37.388 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:37.388 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:37.388 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:37.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 1 Processor -- start 2024-09-16T05:37:37.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 1 -- start start 2024-09-16T05:37:37.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient: init 2024-09-16T05:37:37.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient(hunting): picked mon.noname-a con 0x7fad8009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient(hunting): picked mon.noname-b con 0x7fad8006dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient(hunting): picked mon.noname-c con 0x7fad800a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad8009f800 con 0x7fad8009ac90 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad800a0ab0 con 0x7fad800a0210 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad800a0d90 con 0x7fad8006dd30 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad85fd4640 10 monclient(hunting): authenticate will time out at 1114.163739s 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.382+0000 7fad7ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fad800a0210 legacy=0x7fad8013d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fad8009ac90 0x7fad8009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53358/0 (socket says 172.21.15.63:53358) 2024-09-16T05:37:37.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7f7fe640 1 -- 172.21.15.63:0/1756311543 learned_addr learned my addr 172.21.15.63:0/1756311543 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7effd640 1 --1- 172.21.15.63:0/1756311543 >> v1:172.21.15.6:6790/0 conn(0x7fad8006dd30 0x7fad801399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48526/0 (socket says 172.21.15.63:48526) 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2680158106 0 0) 0x7fad8009f800 con 0x7fad8009ac90 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient(hunting): my global_id is 5262 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad68003530 con 0x7fad8009ac90 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1472034519 0 0) 0x7fad800a0d90 con 0x7fad8006dd30 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:37.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient(hunting): my global_id is 5249 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad8009f800 con 0x7fad8006dd30 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2075262497 0 0) 0x7fad68003530 con 0x7fad8009ac90 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 >> v1:172.21.15.6:6790/0 conn(0x7fad8006dd30 legacy=0x7fad801399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 >> v1:172.21.15.63:6789/0 conn(0x7fad800a0210 legacy=0x7fad8013d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 monclient: found mon.noname-a 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8009e770 con 0x7fad8009ac90 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:37.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 5 monclient: authenticate success, global_id 5262 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad700019a0 con 0x7fad8009ac90 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: dump: 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:37.400 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad70001c00 con 0x7fad8009ac90 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 1 -- 172.21.15.63:0/1756311543 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad70001fd0 con 0x7fad8009ac90 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 4 set_mon_vals no callback set 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7e7fc640 10 monclient: dump: 2024-09-16T05:37:37.401 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient: shutdown 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- 172.21.15.63:0/1756311543 >> v1:172.21.15.6:6789/0 conn(0x7fad8009ac90 legacy=0x7fad8009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- 172.21.15.63:0/1756311543 shutdown_connections 2024-09-16T05:37:37.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- 172.21.15.63:0/1756311543 >> 172.21.15.63:0/1756311543 conn(0x7fad800a1c90 msgr2=0x7fad800a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- 172.21.15.63:0/1756311543 shutdown_connections 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- 172.21.15.63:0/1756311543 wait complete. 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient: build_initial_monmap 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient: monmap: 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:37.403 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 Processor -- start 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- start start 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient: init 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient(hunting): picked mon.noname-a con 0x7fad8006dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient(hunting): picked mon.noname-b con 0x7fad8009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient(hunting): picked mon.noname-c con 0x7fad800a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad8006e160 con 0x7fad8006dd30 2024-09-16T05:37:37.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7effd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fad8009ac90 legacy=0x7fad801cde30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad8011ee10 con 0x7fad8009ac90 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad80120e20 con 0x7fad800a0210 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad85fd4640 10 monclient(hunting): authenticate will time out at 1114.167715s 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.386+0000 7fad7f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fad8006dd30 0x7fad80133740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53362/0 (socket says 172.21.15.63:53362) 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad7f7fe640 1 -- 172.21.15.63:0/1074819363 learned_addr learned my addr 172.21.15.63:0/1074819363 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823392242 0 0) 0x7fad80120e20 con 0x7fad800a0210 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): my global_id is 5252 2024-09-16T05:37:37.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad6c003530 con 0x7fad800a0210 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262710109 0 0) 0x7fad8006e160 con 0x7fad8006dd30 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): my global_id is 5265 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad80120e20 con 0x7fad8006dd30 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1211765167 0 0) 0x7fad6c003530 con 0x7fad800a0210 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad8006e160 con 0x7fad800a0210 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad78003fc0 con 0x7fad800a0210 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): dump: 2024-09-16T05:37:37.406 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2251821591 0 0) 0x7fad80120e20 con 0x7fad8006dd30 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad6c003530 con 0x7fad8006dd30 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad700018a0 con 0x7fad8006dd30 2024-09-16T05:37:37.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient(hunting): dump: 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:37.408 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3015986413 0 0) 0x7fad8006e160 con 0x7fad800a0210 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 >> v1:172.21.15.63:6789/0 conn(0x7fad8009ac90 legacy=0x7fad801cde30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 >> v1:172.21.15.6:6789/0 conn(0x7fad8006dd30 legacy=0x7fad80133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 monclient: found mon.c 2024-09-16T05:37:37.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad80121100 con 0x7fad800a0210 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient: _finish_auth 0 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 5 monclient: authenticate success, global_id 5252 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 10 monclient: _renew_subs 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fad8011f6f0 con 0x7fad800a0210 2024-09-16T05:37:37.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 10 monclient: _renew_subs 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fad80134020 con 0x7fad800a0210 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad78004220 con 0x7fad800a0210 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad78004570 con 0x7fad800a0210 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad7dffb640 4 set_mon_vals no callback set 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 10 monclient: dump: 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:37.411 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad60005090 con 0x7fad800a0210 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fad78004790 con 0x7fad800a0210 2024-09-16T05:37:37.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(88..88 src has 1..88) ==== 5364+0+0 (unknown 1742545226 0 0) 0x7fad78044600 con 0x7fad800a0210 2024-09-16T05:37:37.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.390+0000 7fad7f7fe640 10 monclient: get_auth_request con 0x7fad6c036a20 auth_method 0 2024-09-16T05:37:37.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.402+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fad78018dd0 con 0x7fad800a0210 2024-09-16T05:37:37.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.402+0000 7fad67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:37.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.402+0000 7fad67fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:37.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.570+0000 7fad85fd4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T05:37:37.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.570+0000 7fad85fd4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T05:37:37.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.570+0000 7fad85fd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:37.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:37.570+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) -- 0x7fad60005380 con 0x7fad800a0210 2024-09-16T05:37:38.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.166+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 removed item id -14 name 'root-for-test' from crush map v89) ==== 140+0+0 (unknown 1788538153 0 0) 0x7fad7801d690 con 0x7fad800a0210 2024-09-16T05:37:38.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.166+0000 7fad67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T05:37:38.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.166+0000 7fad67fff640 10 monclient: _finish_command 2 = system:0 removed item id -14 name 'root-for-test' from crush map 2024-09-16T05:37:38.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.274+0000 7fad85fd4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T05:37:38.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.274+0000 7fad85fd4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T05:37:38.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.274+0000 7fad85fd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:38.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.274+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) -- 0x7fad600010a0 con 0x7fad800a0210 2024-09-16T05:37:38.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.274+0000 7fad67fff640 1 -- 172.21.15.63:0/1074819363 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 device 'root-for-test' does not appear in the crush map v89) ==== 140+0+0 (unknown 1275700875 0 0) 0x7fad78040210 con 0x7fad800a0210 2024-09-16T05:37:38.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.274+0000 7fad67fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T05:37:38.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.274+0000 7fad67fff640 10 monclient: _finish_command 3 = system:0 device 'root-for-test' does not appear in the crush map 2024-09-16T05:37:38.280 INFO:tasks.workunit.client.0.smithi063.stderr:device 'root-for-test' does not appear in the crush map 2024-09-16T05:37:38.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.278+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 >> v1:172.21.15.63:6812/3643207211 conn(0x7fad6c036a20 legacy=0x7fad6c038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:38.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.278+0000 7fad85fd4640 10 monclient: shutdown 2024-09-16T05:37:38.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.278+0000 7fad85fd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:38.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.278+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 >> v1:172.21.15.6:6790/0 conn(0x7fad800a0210 legacy=0x7fad801d1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:38.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.278+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 shutdown_connections 2024-09-16T05:37:38.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.278+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 >> 172.21.15.63:0/1074819363 conn(0x7fad800a1c90 msgr2=0x7fad80134930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:38.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.278+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 shutdown_connections 2024-09-16T05:37:38.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.278+0000 7fad85fd4640 1 -- 172.21.15.63:0/1074819363 wait complete. 2024-09-16T05:37:38.297 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2024-09-16T05:37:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.414+0000 7f9ae1838640 10 monclient: get_monmap_and_config 2024-09-16T05:37:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient: build_initial_monmap 2024-09-16T05:37:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient: monmap: 2024-09-16T05:37:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:38.421 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:38.420565+0000 2024-09-16T05:37:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:38.420565+0000 2024-09-16T05:37:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:38.422 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:38.422 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 Processor -- start 2024-09-16T05:37:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 -- start start 2024-09-16T05:37:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient: init 2024-09-16T05:37:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:38.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient(hunting): picked mon.noname-b con 0x7f9adc0a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc0a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient(hunting): picked mon.noname-a con 0x7f9adc0a1f10 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9adc0a0f40 con 0x7f9adc0a1f10 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9adc0a1220 con 0x7f9adc0a4600 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9adc0a1500 con 0x7f9adc0a3b10 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient(hunting): authenticate will time out at 1115.197562s 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ada7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9adc0a4600 legacy=0x7f9adc13bc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9adaffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9adc0a3b10 0x7f9adc0a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48550/0 (socket says 172.21.15.63:48550) 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9adaffd640 1 -- 172.21.15.63:0/1047987852 learned_addr learned my addr 172.21.15.63:0/1047987852 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934045848 0 0) 0x7f9adc0a1500 con 0x7f9adc0a3b10 2024-09-16T05:37:38.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient(hunting): my global_id is 5255 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac4003530 con 0x7f9adc0a3b10 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1649715286 0 0) 0x7f9adc0a0f40 con 0x7f9adc0a1f10 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient(hunting): my global_id is 5268 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9adc0a1500 con 0x7f9adc0a1f10 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2114037430 0 0) 0x7f9adc0a1500 con 0x7f9adc0a1f10 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 >> v1:172.21.15.6:6790/0 conn(0x7f9adc0a3b10 legacy=0x7f9adc0a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:38.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 >> v1:172.21.15.63:6789/0 conn(0x7f9adc0a4600 legacy=0x7f9adc13bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 monclient: found mon.noname-a 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9adc0a51c0 con 0x7f9adc0a1f10 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ad4001600 con 0x7f9adc0a1f10 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 5 monclient: authenticate success, global_id 5268 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:38.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: dump: 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:38.435 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ad40018f0 con 0x7f9adc0a1f10 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 4 set_mon_vals no callback set 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 1 -- 172.21.15.63:0/1047987852 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ad4001cc0 con 0x7f9adc0a1f10 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ad9ffb640 10 monclient: dump: 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:38.436 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 10 monclient: shutdown 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 -- 172.21.15.63:0/1047987852 >> v1:172.21.15.6:6789/0 conn(0x7f9adc0a1f10 legacy=0x7f9adc13f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 -- 172.21.15.63:0/1047987852 shutdown_connections 2024-09-16T05:37:38.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 -- 172.21.15.63:0/1047987852 >> 172.21.15.63:0/1047987852 conn(0x7f9adc075470 msgr2=0x7f9adc075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.418+0000 7f9ae1838640 1 -- 172.21.15.63:0/1047987852 shutdown_connections 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 1 -- 172.21.15.63:0/1047987852 wait complete. 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: build_initial_monmap 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: monmap: 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:38.438 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 1 Processor -- start 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 1 -- start start 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: init 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc0a1f10 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient(hunting): picked mon.noname-b con 0x7f9adc0a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient(hunting): picked mon.noname-a con 0x7f9adc0a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9adc139060 con 0x7f9adc0a4600 2024-09-16T05:37:38.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9adc139340 con 0x7f9adc0a3b10 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9adc072b10 con 0x7f9adc0a1f10 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient(hunting): authenticate will time out at 1115.201526s 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ada7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9adc0a3b10 legacy=0x7f9adc1d0250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9adb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9adc0a4600 0x7f9adc1d3980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53376/0 (socket says 172.21.15.63:53376) 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9adb7fe640 1 -- 172.21.15.63:0/1614415919 learned_addr learned my addr 172.21.15.63:0/1614415919 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2350225198 0 0) 0x7f9adc072b10 con 0x7f9adc0a1f10 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): my global_id is 5258 2024-09-16T05:37:38.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac8003530 con 0x7f9adc0a1f10 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1185170577 0 0) 0x7f9adc139060 con 0x7f9adc0a4600 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): my global_id is 5271 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9adc072b10 con 0x7f9adc0a4600 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4233731228 0 0) 0x7f9ac8003530 con 0x7f9adc0a1f10 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9adc139060 con 0x7f9adc0a1f10 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9acc001650 con 0x7f9adc0a1f10 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): dump: 2024-09-16T05:37:38.441 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1593338722 0 0) 0x7f9adc072b10 con 0x7f9adc0a4600 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ac8003530 con 0x7f9adc0a4600 2024-09-16T05:37:38.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ad40015c0 con 0x7f9adc0a4600 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient(hunting): dump: 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:38.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 558482178 0 0) 0x7f9adc139060 con 0x7f9adc0a1f10 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 >> v1:172.21.15.63:6789/0 conn(0x7f9adc0a3b10 legacy=0x7f9adc1d0250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 >> v1:172.21.15.6:6789/0 conn(0x7f9adc0a4600 legacy=0x7f9adc1d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 monclient: found mon.c 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9adc0a0500 con 0x7f9adc0a1f10 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient: _finish_auth 0 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 5 monclient: authenticate success, global_id 5258 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: _renew_subs 2024-09-16T05:37:38.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9adc136e10 con 0x7f9adc0a1f10 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: _renew_subs 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9adc137c60 con 0x7f9adc0a1f10 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9acc002050 con 0x7f9adc0a1f10 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ad8ff9640 4 set_mon_vals no callback set 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9acc004e20 con 0x7f9adc0a1f10 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.422+0000 7f9ac3fff640 10 monclient: dump: 2024-09-16T05:37:38.445 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.426+0000 7f9ae1838640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.426+0000 7f9ae1838640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.426+0000 7f9ae1838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.426+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9abc005090 con 0x7f9adc0a1f10 2024-09-16T05:37:38.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.426+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9acc00fc40 con 0x7f9adc0a1f10 2024-09-16T05:37:38.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.426+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(89..89 src has 1..89) ==== 5205+0+0 (unknown 3865529890 0 0) 0x7f9acc044b40 con 0x7f9adc0a1f10 2024-09-16T05:37:38.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.426+0000 7f9ada7fc640 10 monclient: get_auth_request con 0x7f9ac8036930 auth_method 0 2024-09-16T05:37:38.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.438+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9acc0157c0 con 0x7f9adc0a1f10 2024-09-16T05:37:38.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.438+0000 7f9ac3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:38.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.438+0000 7f9ac3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:38.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.598+0000 7f9ae1838640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T05:37:38.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.598+0000 7f9ae1838640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T05:37:38.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.598+0000 7f9ae1838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:38.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:38.598+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) -- 0x7f9abc005380 con 0x7f9adc0a1f10 2024-09-16T05:37:39.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.170+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 v90) ==== 145+0+0 (unknown 3777338399 0 0) 0x7f9acc0150c0 con 0x7f9adc0a1f10 2024-09-16T05:37:39.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.170+0000 7f9ac3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T05:37:39.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.170+0000 7f9ac3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:39.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.270+0000 7f9ae1838640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T05:37:39.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.270+0000 7f9ae1838640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T05:37:39.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.270+0000 7f9ae1838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.270+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) -- 0x7f9abc000cb0 con 0x7f9adc0a1f10 2024-09-16T05:37:39.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.274+0000 7f9ac3fff640 1 -- 172.21.15.63:0/1614415919 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 rule foo-rule already exists v90) ==== 173+0+0 (unknown 3921559322 0 0) 0x7f9acc01dc70 con 0x7f9adc0a1f10 2024-09-16T05:37:39.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.274+0000 7f9ac3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T05:37:39.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.274+0000 7f9ac3fff640 10 monclient: _finish_command 3 = system:0 rule foo-rule already exists 2024-09-16T05:37:39.279 INFO:tasks.workunit.client.0.smithi063.stderr:rule foo-rule already exists 2024-09-16T05:37:39.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.278+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9ac8036930 legacy=0x7f9ac8038df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:39.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.278+0000 7f9ae1838640 10 monclient: shutdown 2024-09-16T05:37:39.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.278+0000 7f9ae1838640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:39.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.278+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 >> v1:172.21.15.6:6790/0 conn(0x7f9adc0a1f10 legacy=0x7f9adc13f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:39.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.278+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 shutdown_connections 2024-09-16T05:37:39.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.278+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 >> 172.21.15.63:0/1614415919 conn(0x7f9adc075470 msgr2=0x7f9adc0a7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:39.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.278+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 shutdown_connections 2024-09-16T05:37:39.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.278+0000 7f9ae1838640 1 -- 172.21.15.63:0/1614415919 wait complete. 2024-09-16T05:37:39.296 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush rm foo 2024-09-16T05:37:39.297 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:39.297 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm foo 2024-09-16T05:37:39.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient: get_monmap_and_config 2024-09-16T05:37:39.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient: build_initial_monmap 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient: monmap: 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:39.417799+0000 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:39.417799+0000 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:39.419 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 1 Processor -- start 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 1 -- start start 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient: init 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient(hunting): picked mon.noname-a con 0x7f604809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient(hunting): picked mon.noname-c con 0x7f604806dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient(hunting): picked mon.noname-b con 0x7f60480a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604809f800 con 0x7f604809ac90 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60480a0ab0 con 0x7f604806dd30 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60480a0d90 con 0x7f60480a0210 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:39.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604fc3a640 10 monclient(hunting): authenticate will time out at 1116.194699s 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604d1d5640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f604806dd30 legacy=0x7f60481399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604e1d7640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f60480a0210 0x7f604813d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48560/0 (socket says 172.21.15.63:48560) 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604d9d6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f604809ac90 0x7f604809b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53392/0 (socket says 172.21.15.63:53392) 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604e1d7640 1 -- 172.21.15.63:0/2875664415 learned_addr learned my addr 172.21.15.63:0/2875664415 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49705606 0 0) 0x7f60480a0d90 con 0x7f60480a0210 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 10 monclient(hunting): _negotiate 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 10 monclient(hunting): my global_id is 5261 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6034003530 con 0x7f60480a0210 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310727618 0 0) 0x7f604809f800 con 0x7f604809ac90 2024-09-16T05:37:39.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 10 monclient(hunting): _negotiate 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 10 monclient(hunting): my global_id is 5274 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60480a0d90 con 0x7f604809ac90 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.414+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3844772688 0 0) 0x7f6034003530 con 0x7f60480a0210 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 >> v1:172.21.15.63:6789/0 conn(0x7f604806dd30 legacy=0x7f60481399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 >> v1:172.21.15.6:6789/0 conn(0x7f604809ac90 legacy=0x7f604809b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: _finish_hunting 0 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 1 monclient: found mon.noname-b 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604809e770 con 0x7f60480a0210 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: _finish_auth 0 2024-09-16T05:37:39.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: _check_auth_tickets 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f60400019a0 con 0x7f60480a0210 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: dump: 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.426 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 5 monclient: authenticate success, global_id 5261 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6040001cc0 con 0x7f60480a0210 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 1 -- 172.21.15.63:0/2875664415 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6040002090 con 0x7f60480a0210 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604c9d4640 10 monclient: dump: 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.427 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 4 set_mon_vals no callback set 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient: shutdown 2024-09-16T05:37:39.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- 172.21.15.63:0/2875664415 >> v1:172.21.15.6:6790/0 conn(0x7f60480a0210 legacy=0x7f604813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- 172.21.15.63:0/2875664415 shutdown_connections 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- 172.21.15.63:0/2875664415 >> 172.21.15.63:0/2875664415 conn(0x7f60480a1c90 msgr2=0x7f60480a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- 172.21.15.63:0/2875664415 shutdown_connections 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- 172.21.15.63:0/2875664415 wait complete. 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient: build_initial_monmap 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient: monmap: 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:39.429 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.430 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:39.430 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:39.430 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:39.430 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 Processor -- start 2024-09-16T05:37:39.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- start start 2024-09-16T05:37:39.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient: init 2024-09-16T05:37:39.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:39.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:39.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient(hunting): picked mon.noname-b con 0x7f604806dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:39.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient(hunting): picked mon.noname-a con 0x7f604809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:39.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient(hunting): picked mon.noname-c con 0x7f60480a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:39.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604d9d6640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f604806dd30 legacy=0x7f6048133840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:39.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6048120f20 con 0x7f604809ac90 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604811fa70 con 0x7f604806dd30 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604806e2f0 con 0x7f60480a0210 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604fc3a640 10 monclient(hunting): authenticate will time out at 1116.198558s 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604e1d7640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f60480a0210 0x7f60481d16e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48564/0 (socket says 172.21.15.63:48564) 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f604e1d7640 1 -- 172.21.15.63:0/1666499216 learned_addr learned my addr 172.21.15.63:0/1666499216 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 689846151 0 0) 0x7f604806e2f0 con 0x7f60480a0210 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): my global_id is 5264 2024-09-16T05:37:39.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f602c003530 con 0x7f60480a0210 2024-09-16T05:37:39.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778998491 0 0) 0x7f6048120f20 con 0x7f604809ac90 2024-09-16T05:37:39.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:39.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:39.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:39.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): my global_id is 5277 2024-09-16T05:37:39.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f604806e2f0 con 0x7f604809ac90 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 355121911 0 0) 0x7f602c003530 con 0x7f60480a0210 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6048120f20 con 0x7f60480a0210 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6040001960 con 0x7f60480a0210 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): dump: 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.437 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.418+0000 7f603e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2177246778 0 0) 0x7f604806e2f0 con 0x7f604809ac90 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f602c003530 con 0x7f604809ac90 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f60380040d0 con 0x7f604809ac90 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:39.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient(hunting): dump: 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 151912945 0 0) 0x7f6048120f20 con 0x7f60480a0210 2024-09-16T05:37:39.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 >> v1:172.21.15.63:6789/0 conn(0x7f604806dd30 legacy=0x7f6048133840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 >> v1:172.21.15.6:6789/0 conn(0x7f604809ac90 legacy=0x7f60481cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 monclient: found mon.c 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6048120c40 con 0x7f60480a0210 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 5 monclient: authenticate success, global_id 5264 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 10 monclient: _renew_subs 2024-09-16T05:37:39.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f604811f7f0 con 0x7f60480a0210 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 10 monclient: _renew_subs 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6048133f50 con 0x7f60480a0210 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6040001cc0 con 0x7f60480a0210 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6040005000 con 0x7f60480a0210 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 10 monclient: dump: 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.441 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603ffff640 4 set_mon_vals no callback set 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604fc3a640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6030005090 con 0x7f60480a0210 2024-09-16T05:37:39.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f604000fda0 con 0x7f60480a0210 2024-09-16T05:37:39.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(90..90 src has 1..90) ==== 5265+0+0 (unknown 201638795 0 0) 0x7f6040044ad0 con 0x7f60480a0210 2024-09-16T05:37:39.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.422+0000 7f604d9d6640 10 monclient: get_auth_request con 0x7f602c036a00 auth_method 0 2024-09-16T05:37:39.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.434+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6040015710 con 0x7f60480a0210 2024-09-16T05:37:39.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.434+0000 7f603e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:39.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.434+0000 7f603e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:39.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.602+0000 7f604fc3a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:39.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.602+0000 7f604fc3a640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:39.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.602+0000 7f604fc3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.602+0000 7f604fc3a640 1 -- 172.21.15.63:0/1666499216 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7f6030005380 con 0x7f60480a0210 2024-09-16T05:37:39.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.606+0000 7f603e7fc640 1 -- 172.21.15.63:0/1666499216 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-16 (16) Device or resource busy v90) ==== 103+0+0 (unknown 2485456370 0 0) 0x7f604001dbc0 con 0x7f60480a0210 2024-09-16T05:37:39.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.606+0000 7f603e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:39.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.606+0000 7f603e7fc640 10 monclient: _finish_command 2 = mon:16 (16) Device or resource busy 2024-09-16T05:37:39.611 INFO:tasks.workunit.client.0.smithi063.stderr:Error EBUSY: (16) Device or resource busy 2024-09-16T05:37:39.617 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:39.617 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rule rm foo-rule 2024-09-16T05:37:39.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient: get_monmap_and_config 2024-09-16T05:37:39.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient: build_initial_monmap 2024-09-16T05:37:39.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient: monmap: 2024-09-16T05:37:39.734 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:39.734 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.734 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:39.733407+0000 2024-09-16T05:37:39.734 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:39.733407+0000 2024-09-16T05:37:39.735 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:39.735 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.735 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:39.735 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:39.735 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:39.735 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 1 Processor -- start 2024-09-16T05:37:39.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 1 -- start start 2024-09-16T05:37:39.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient: init 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient(hunting): picked mon.noname-c con 0x7fd024136a50 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient(hunting): picked mon.noname-b con 0x7fd02409fd70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient(hunting): picked mon.noname-a con 0x7fd02413d0d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0240a05a0 con 0x7fd02413d0d0 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0240a0880 con 0x7fd024136a50 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0240a0b60 con 0x7fd02409fd70 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 10 monclient(hunting): authenticate will time out at 1116.510386s 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02a451640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd024136a50 legacy=0x7fd024136e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:39.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd029c50640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd02409fd70 0x7fd02413b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48574/0 (socket says 172.21.15.63:48574) 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd029c50640 1 -- 172.21.15.63:0/853298202 learned_addr learned my addr 172.21.15.63:0/853298202 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 739176971 0 0) 0x7fd0240a0b60 con 0x7fd02409fd70 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient(hunting): _negotiate 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient(hunting): my global_id is 5267 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd010003530 con 0x7fd02409fd70 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 723405107 0 0) 0x7fd0240a05a0 con 0x7fd02413d0d0 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient(hunting): _negotiate 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient(hunting): my global_id is 5280 2024-09-16T05:37:39.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0240a0b60 con 0x7fd02413d0d0 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1860757874 0 0) 0x7fd010003530 con 0x7fd02409fd70 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 >> v1:172.21.15.63:6789/0 conn(0x7fd024136a50 legacy=0x7fd024136e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 >> v1:172.21.15.6:6789/0 conn(0x7fd02413d0d0 legacy=0x7fd02413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient: _finish_hunting 0 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 monclient: found mon.noname-b 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd024137540 con 0x7fd02409fd70 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient: _finish_auth 0 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient: _check_auth_tickets 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:39.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd014001620 con 0x7fd02409fd70 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 5 monclient: authenticate success, global_id 5267 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02c6b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.730+0000 7fd02944f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02944f640 10 monclient: dump: 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.742 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd014001880 con 0x7fd02409fd70 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02944f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02944f640 1 -- 172.21.15.63:0/853298202 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd014001bd0 con 0x7fd02409fd70 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02944f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02944f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02944f640 10 monclient: dump: 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.743 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 4 set_mon_vals no callback set 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient: shutdown 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:39.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- 172.21.15.63:0/853298202 >> v1:172.21.15.6:6790/0 conn(0x7fd02409fd70 legacy=0x7fd02413b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- 172.21.15.63:0/853298202 shutdown_connections 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- 172.21.15.63:0/853298202 >> 172.21.15.63:0/853298202 conn(0x7fd0240a5290 msgr2=0x7fd0240a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- 172.21.15.63:0/853298202 shutdown_connections 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- 172.21.15.63:0/853298202 wait complete. 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient: build_initial_monmap 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient: monmap: 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:39.745 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:39.746 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:39.746 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 Processor -- start 2024-09-16T05:37:39.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- start start 2024-09-16T05:37:39.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient: init 2024-09-16T05:37:39.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:39.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient(hunting): picked mon.noname-a con 0x7fd02409fd70 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient(hunting): picked mon.noname-c con 0x7fd024136a50 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient(hunting): picked mon.noname-b con 0x7fd02413d0d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd02411f580 con 0x7fd02409fd70 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd02411f800 con 0x7fd02413d0d0 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd024120cb0 con 0x7fd024136a50 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02c6b5640 10 monclient(hunting): authenticate will time out at 1116.514444s 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02ac52640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd02413d0d0 legacy=0x7fd0241d3dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02a451640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd02409fd70 0x7fd024135150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53424/0 (socket says 172.21.15.63:53424) 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd029c50640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd024136a50 0x7fd0241d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48586/0 (socket says 172.21.15.63:48586) 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd02a451640 1 -- 172.21.15.63:0/204112360 learned_addr learned my addr 172.21.15.63:0/204112360 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:39.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228769348 0 0) 0x7fd024120cb0 con 0x7fd024136a50 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): my global_id is 5270 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd008003530 con 0x7fd024136a50 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2485259172 0 0) 0x7fd02411f580 con 0x7fd02409fd70 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): my global_id is 5283 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd024120cb0 con 0x7fd02409fd70 2024-09-16T05:37:39.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 446863388 0 0) 0x7fd008003530 con 0x7fd024136a50 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd02411f580 con 0x7fd024136a50 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd0140015e0 con 0x7fd024136a50 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): dump: 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.752 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1181300313 0 0) 0x7fd024120cb0 con 0x7fd02409fd70 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd008003530 con 0x7fd02409fd70 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd020004090 con 0x7fd02409fd70 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.734+0000 7fd01affd640 10 monclient(hunting): dump: 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.753 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2140553800 0 0) 0x7fd02411f580 con 0x7fd024136a50 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 >> v1:172.21.15.63:6789/0 conn(0x7fd02413d0d0 legacy=0x7fd0241d3dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 >> v1:172.21.15.6:6789/0 conn(0x7fd02409fd70 legacy=0x7fd024135150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 monclient: found mon.c 2024-09-16T05:37:39.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0240a3ca0 con 0x7fd024136a50 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 10 monclient: _finish_auth 0 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 5 monclient: authenticate success, global_id 5270 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 10 monclient: _renew_subs 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd02413c9f0 con 0x7fd024136a50 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 10 monclient: _renew_subs 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd0241209d0 con 0x7fd024136a50 2024-09-16T05:37:39.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd014001880 con 0x7fd024136a50 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd014004c90 con 0x7fd024136a50 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01bfff640 4 set_mon_vals no callback set 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 10 monclient: dump: 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:39.756 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd00c005090 con 0x7fd024136a50 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd014004eb0 con 0x7fd024136a50 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(90..90 src has 1..90) ==== 5265+0+0 (unknown 201638795 0 0) 0x7fd014044920 con 0x7fd024136a50 2024-09-16T05:37:39.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.738+0000 7fd02a451640 10 monclient: get_auth_request con 0x7fd008036a00 auth_method 0 2024-09-16T05:37:39.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.750+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd014015500 con 0x7fd024136a50 2024-09-16T05:37:39.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.750+0000 7fd01affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:39.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.750+0000 7fd01affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:39.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.922+0000 7fd02c6b5640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T05:37:39.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.922+0000 7fd02c6b5640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T05:37:39.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.922+0000 7fd02c6b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:39.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:39.922+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) -- 0x7fd00c005380 con 0x7fd024136a50 2024-09-16T05:37:40.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.190+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 v91) ==== 85+0+0 (unknown 1139158635 0 0) 0x7fd014019150 con 0x7fd024136a50 2024-09-16T05:37:40.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.190+0000 7fd01affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T05:37:40.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.190+0000 7fd01affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:40.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.294+0000 7fd02c6b5640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T05:37:40.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.294+0000 7fd02c6b5640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T05:37:40.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.294+0000 7fd02c6b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:40.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.294+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) -- 0x7fd00c005770 con 0x7fd024136a50 2024-09-16T05:37:40.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.294+0000 7fd01affd640 1 -- 172.21.15.63:0/204112360 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 rule foo-rule does not exist v91) ==== 113+0+0 (unknown 2444140760 0 0) 0x7fd01401da10 con 0x7fd024136a50 2024-09-16T05:37:40.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.294+0000 7fd01affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T05:37:40.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.294+0000 7fd01affd640 10 monclient: _finish_command 3 = system:0 rule foo-rule does not exist 2024-09-16T05:37:40.301 INFO:tasks.workunit.client.0.smithi063.stderr:rule foo-rule does not exist 2024-09-16T05:37:40.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.298+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd008036a00 legacy=0x7fd008038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:40.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.298+0000 7fd02c6b5640 10 monclient: shutdown 2024-09-16T05:37:40.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.298+0000 7fd02c6b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:40.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.298+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 >> v1:172.21.15.6:6790/0 conn(0x7fd024136a50 legacy=0x7fd0241d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:40.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.298+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 shutdown_connections 2024-09-16T05:37:40.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.298+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 >> 172.21.15.63:0/204112360 conn(0x7fd0240a5290 msgr2=0x7fd02409df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:40.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.298+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 shutdown_connections 2024-09-16T05:37:40.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.298+0000 7fd02c6b5640 1 -- 172.21.15.63:0/204112360 wait complete. 2024-09-16T05:37:40.318 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm bar 2024-09-16T05:37:40.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient: get_monmap_and_config 2024-09-16T05:37:40.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient: build_initial_monmap 2024-09-16T05:37:40.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient: monmap: 2024-09-16T05:37:40.439 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:40.438694+0000 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:40.438694+0000 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:40.440 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:40.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 1 Processor -- start 2024-09-16T05:37:40.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 1 -- start start 2024-09-16T05:37:40.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient: init 2024-09-16T05:37:40.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:40.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:40.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient(hunting): picked mon.noname-a con 0x7fd8fc09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:40.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient(hunting): picked mon.noname-b con 0x7fd8fc06dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:40.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient(hunting): picked mon.noname-c con 0x7fd8fc0a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc09f800 con 0x7fd8fc09ac90 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc0a0ab0 con 0x7fd8fc0a0210 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc0a0d90 con 0x7fd8fc06dd30 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd903f02640 10 monclient(hunting): authenticate will time out at 1117.215685s 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.434+0000 7fd90249f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd8fc0a0210 legacy=0x7fd8fc13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd90149d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd8fc06dd30 0x7fd8fc1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48592/0 (socket says 172.21.15.63:48592) 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd90149d640 1 -- 172.21.15.63:0/2119837915 learned_addr learned my addr 172.21.15.63:0/2119837915 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3786704780 0 0) 0x7fd8fc0a0d90 con 0x7fd8fc06dd30 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient(hunting): _negotiate 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:40.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient(hunting): my global_id is 5273 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e8003530 con 0x7fd8fc06dd30 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432149643 0 0) 0x7fd8fc09f800 con 0x7fd8fc09ac90 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient(hunting): _negotiate 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient(hunting): my global_id is 5286 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8fc0a0d90 con 0x7fd8fc09ac90 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 836059308 0 0) 0x7fd8e8003530 con 0x7fd8fc06dd30 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 >> v1:172.21.15.63:6789/0 conn(0x7fd8fc0a0210 legacy=0x7fd8fc13d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 >> v1:172.21.15.6:6789/0 conn(0x7fd8fc09ac90 legacy=0x7fd8fc09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: _finish_hunting 0 2024-09-16T05:37:40.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 monclient: found mon.noname-b 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8fc09e770 con 0x7fd8fc06dd30 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: _finish_auth 0 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: _check_auth_tickets 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8ec001620 con 0x7fd8fc06dd30 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: dump: 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.446 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 5 monclient: authenticate success, global_id 5273 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8ec001880 con 0x7fd8fc06dd30 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 1 -- 172.21.15.63:0/2119837915 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8ec001c90 con 0x7fd8fc06dd30 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:40.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd900c9c640 10 monclient: dump: 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 4 set_mon_vals no callback set 2024-09-16T05:37:40.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 10 monclient: shutdown 2024-09-16T05:37:40.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:40.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 1 -- 172.21.15.63:0/2119837915 >> v1:172.21.15.6:6790/0 conn(0x7fd8fc06dd30 legacy=0x7fd8fc1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:40.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 1 -- 172.21.15.63:0/2119837915 shutdown_connections 2024-09-16T05:37:40.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 1 -- 172.21.15.63:0/2119837915 >> 172.21.15.63:0/2119837915 conn(0x7fd8fc0a1c90 msgr2=0x7fd8fc0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:40.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 1 -- 172.21.15.63:0/2119837915 shutdown_connections 2024-09-16T05:37:40.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 1 -- 172.21.15.63:0/2119837915 wait complete. 2024-09-16T05:37:40.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 10 monclient: build_initial_monmap 2024-09-16T05:37:40.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 10 monclient: monmap: 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 1 Processor -- start 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 1 -- start start 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.438+0000 7fd903f02640 10 monclient: init 2024-09-16T05:37:40.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient(hunting): picked mon.noname-b con 0x7fd8fc06dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient(hunting): picked mon.noname-c con 0x7fd8fc09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient(hunting): picked mon.noname-a con 0x7fd8fc0a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd901c9e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd8fc06dd30 legacy=0x7fd8fc133a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc123e70 con 0x7fd8fc0a0210 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc122ca0 con 0x7fd8fc06dd30 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc1225e0 con 0x7fd8fc09ac90 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient(hunting): authenticate will time out at 1117.220376s 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd90149d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd8fc09ac90 0x7fd8fc1cde30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48594/0 (socket says 172.21.15.63:48594) 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd90149d640 1 -- 172.21.15.63:0/2643961536 learned_addr learned my addr 172.21.15.63:0/2643961536 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:40.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3540575980 0 0) 0x7fd8fc1225e0 con 0x7fd8fc09ac90 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): my global_id is 5276 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e0003530 con 0x7fd8fc09ac90 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099961261 0 0) 0x7fd8fc123e70 con 0x7fd8fc0a0210 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): my global_id is 5289 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8fc1225e0 con 0x7fd8fc0a0210 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3707069777 0 0) 0x7fd8e0003530 con 0x7fd8fc09ac90 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8fc123e70 con 0x7fd8fc09ac90 2024-09-16T05:37:40.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8ec001c10 con 0x7fd8fc09ac90 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): dump: 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:40.456 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1734226057 0 0) 0x7fd8fc1225e0 con 0x7fd8fc0a0210 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8e0003530 con 0x7fd8fc0a0210 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8f4004340 con 0x7fd8fc0a0210 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient(hunting): dump: 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:40.457 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3811796452 0 0) 0x7fd8fc123e70 con 0x7fd8fc09ac90 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 >> v1:172.21.15.63:6789/0 conn(0x7fd8fc06dd30 legacy=0x7fd8fc133a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 >> v1:172.21.15.6:6789/0 conn(0x7fd8fc0a0210 legacy=0x7fd8fc1d1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 monclient: found mon.c 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8fc124150 con 0x7fd8fc09ac90 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:40.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 5 monclient: authenticate success, global_id 5276 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient: _renew_subs 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8fc123b90 con 0x7fd8fc09ac90 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient: _renew_subs 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8fc1c8f40 con 0x7fd8fc09ac90 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8ec001880 con 0x7fd8fc09ac90 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8ec004c40 con 0x7fd8fc09ac90 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:40.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 10 monclient: dump: 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f37fe640 4 set_mon_vals no callback set 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:40.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:40.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e4005090 con 0x7fd8fc09ac90 2024-09-16T05:37:40.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd8ec004e60 con 0x7fd8fc09ac90 2024-09-16T05:37:40.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(91..91 src has 1..91) ==== 5205+0+0 (unknown 2026920316 0 0) 0x7fd8ec044700 con 0x7fd8fc09ac90 2024-09-16T05:37:40.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.442+0000 7fd901c9e640 10 monclient: get_auth_request con 0x7fd8e00369e0 auth_method 0 2024-09-16T05:37:40.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.454+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd8ec0154a0 con 0x7fd8fc09ac90 2024-09-16T05:37:40.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.454+0000 7fd8f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:40.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.454+0000 7fd8f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:40.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.622+0000 7fd903f02640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:40.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.622+0000 7fd903f02640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:40.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.622+0000 7fd903f02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:40.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:40.622+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) -- 0x7fd8e4005380 con 0x7fd8fc09ac90 2024-09-16T05:37:41.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.190+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 removed item id -19 name 'bar' from crush map v92) ==== 120+0+0 (unknown 1490001235 0 0) 0x7fd8ec014da0 con 0x7fd8fc09ac90 2024-09-16T05:37:41.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.190+0000 7fd8f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:41.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.190+0000 7fd8f27fc640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'bar' from crush map 2024-09-16T05:37:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.294+0000 7fd903f02640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.294+0000 7fd903f02640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.294+0000 7fd903f02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.294+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) -- 0x7fd8e40010a0 con 0x7fd8fc09ac90 2024-09-16T05:37:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.294+0000 7fd8f27fc640 1 -- 172.21.15.63:0/2643961536 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 device 'bar' does not appear in the crush map v92) ==== 120+0+0 (unknown 2981215460 0 0) 0x7fd8ec040410 con 0x7fd8fc09ac90 2024-09-16T05:37:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd8f27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T05:37:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd8f27fc640 10 monclient: _finish_command 3 = system:0 device 'bar' does not appear in the crush map 2024-09-16T05:37:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:device 'bar' does not appear in the crush map 2024-09-16T05:37:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd8e00369e0 legacy=0x7fd8e0038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd903f02640 10 monclient: shutdown 2024-09-16T05:37:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd903f02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 >> v1:172.21.15.6:6790/0 conn(0x7fd8fc09ac90 legacy=0x7fd8fc1cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 shutdown_connections 2024-09-16T05:37:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 >> 172.21.15.63:0/2643961536 conn(0x7fd8fc0a1c90 msgr2=0x7fd8fc134f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 shutdown_connections 2024-09-16T05:37:41.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.298+0000 7fd903f02640 1 -- 172.21.15.63:0/2643961536 wait complete. 2024-09-16T05:37:41.318 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm foo 2024-09-16T05:37:41.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.434+0000 7f6a04845640 10 monclient: get_monmap_and_config 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.434+0000 7f6a04845640 10 monclient: build_initial_monmap 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.434+0000 7f6a04845640 10 monclient: monmap: 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:41.439051+0000 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:41.439051+0000 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:41.440 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:41.441 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:41.441 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.434+0000 7f6a04845640 1 Processor -- start 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- start start 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient: init 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient(hunting): picked mon.noname-c con 0x7f6a000a3d80 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient(hunting): picked mon.noname-a con 0x7f6a000a4870 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0013d4a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:41.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a000a0880 con 0x7f6a000a4870 2024-09-16T05:37:41.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a000a5340 con 0x7f6a000a3d80 2024-09-16T05:37:41.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a000a10b0 con 0x7f6a0013d4a0 2024-09-16T05:37:41.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:41.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient(hunting): authenticate will time out at 1118.216779s 2024-09-16T05:37:41.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fe59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6a000a3d80 legacy=0x7f6a000a4160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:41.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fdd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a000a4870 0x7f6a0013bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53444/0 (socket says 172.21.15.63:53444) 2024-09-16T05:37:41.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fdd9b640 1 -- 172.21.15.63:0/1030329225 learned_addr learned my addr 172.21.15.63:0/1030329225 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:41.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fed9d640 1 --1- 172.21.15.63:0/1030329225 >> v1:172.21.15.6:6790/0 conn(0x7f6a0013d4a0 0x7f6a0013f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48596/0 (socket says 172.21.15.63:48596) 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1667377032 0 0) 0x7f6a000a10b0 con 0x7f6a0013d4a0 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient(hunting): my global_id is 5279 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69e8003530 con 0x7f6a0013d4a0 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3557850090 0 0) 0x7f6a000a0880 con 0x7f6a000a4870 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient(hunting): my global_id is 5292 2024-09-16T05:37:41.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a000a10b0 con 0x7f6a000a4870 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 831691199 0 0) 0x7f69e8003530 con 0x7f6a0013d4a0 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 >> v1:172.21.15.63:6789/0 conn(0x7f6a000a3d80 legacy=0x7f6a000a4160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 >> v1:172.21.15.6:6789/0 conn(0x7f6a000a4870 legacy=0x7f6a0013bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: _finish_hunting 0 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 monclient: found mon.noname-b 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a000a1390 con 0x7f6a0013d4a0 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: _finish_auth 0 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: _check_auth_tickets 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:41.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 5 monclient: authenticate success, global_id 5279 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f69f80018d0 con 0x7f6a0013d4a0 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: dump: 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:41.448 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:41.449 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:41.449 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:41.449 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:41.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:41.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69f8001b30 con 0x7f6a0013d4a0 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 1 -- 172.21.15.63:0/1030329225 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f69f8001e80 con 0x7f6a0013d4a0 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 4 set_mon_vals no callback set 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f69fd59a640 10 monclient: dump: 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.450 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 10 monclient: shutdown 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- 172.21.15.63:0/1030329225 >> v1:172.21.15.6:6790/0 conn(0x7f6a0013d4a0 legacy=0x7f6a0013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- 172.21.15.63:0/1030329225 shutdown_connections 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- 172.21.15.63:0/1030329225 >> 172.21.15.63:0/1030329225 conn(0x7f6a00075470 msgr2=0x7f6a00075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- 172.21.15.63:0/1030329225 shutdown_connections 2024-09-16T05:37:41.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.438+0000 7f6a04845640 1 -- 172.21.15.63:0/1030329225 wait complete. 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: build_initial_monmap 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: monmap: 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:41.452 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:41.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 1 Processor -- start 2024-09-16T05:37:41.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 1 -- start start 2024-09-16T05:37:41.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: init 2024-09-16T05:37:41.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:41.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient(hunting): picked mon.noname-b con 0x7f6a000a3d80 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient(hunting): picked mon.noname-a con 0x7f6a000a4870 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0013d4a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a00138e80 con 0x7f6a000a4870 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a000a0570 con 0x7f6a000a3d80 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a00139160 con 0x7f6a0013d4a0 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient(hunting): authenticate will time out at 1118.221573s 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69fe59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6a000a3d80 legacy=0x7f6a001cb880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69fdd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a000a4870 0x7f6a001d0860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53454/0 (socket says 172.21.15.63:53454) 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69fdd9b640 1 -- 172.21.15.63:0/1061389001 learned_addr learned my addr 172.21.15.63:0/1061389001 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277714784 0 0) 0x7f6a00139160 con 0x7f6a0013d4a0 2024-09-16T05:37:41.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): my global_id is 5282 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69e0003530 con 0x7f6a0013d4a0 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2954429604 0 0) 0x7f6a00138e80 con 0x7f6a000a4870 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): my global_id is 5295 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a00139160 con 0x7f6a000a4870 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2227175328 0 0) 0x7f69e0003530 con 0x7f6a0013d4a0 2024-09-16T05:37:41.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a00138e80 con 0x7f6a0013d4a0 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f69f80017d0 con 0x7f6a0013d4a0 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): dump: 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:41.459 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 672584833 0 0) 0x7f6a00139160 con 0x7f6a000a4870 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69e0003530 con 0x7f6a000a4870 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f69f4001900 con 0x7f6a000a4870 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient(hunting): dump: 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:41.460 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 112402027 0 0) 0x7f6a00138e80 con 0x7f6a0013d4a0 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 >> v1:172.21.15.63:6789/0 conn(0x7f6a000a3d80 legacy=0x7f6a001cb880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 >> v1:172.21.15.6:6789/0 conn(0x7f6a000a4870 legacy=0x7f6a001d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:41.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 monclient: found mon.c 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a00139440 con 0x7f6a0013d4a0 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 5 monclient: authenticate success, global_id 5282 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: _renew_subs 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a0013cbe0 con 0x7f6a0013d4a0 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: _renew_subs 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:41.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a000710c0 con 0x7f6a0013d4a0 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69f8001b30 con 0x7f6a0013d4a0 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f69f8004fc0 con 0x7f6a0013d4a0 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69eeffd640 10 monclient: dump: 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:41.463 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.442+0000 7f69fcd99640 4 set_mon_vals no callback set 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.446+0000 7f6a04845640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.446+0000 7f6a04845640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.446+0000 7f6a04845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.446+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e4005090 con 0x7f6a0013d4a0 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.446+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f69f8005160 con 0x7f6a0013d4a0 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.446+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(92..92 src has 1..92) ==== 5076+0+0 (unknown 3483656642 0 0) 0x7f69f8044a60 con 0x7f6a0013d4a0 2024-09-16T05:37:41.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.446+0000 7f69fe59c640 10 monclient: get_auth_request con 0x7f69e0036930 auth_method 0 2024-09-16T05:37:41.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.458+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f69f8015700 con 0x7f6a0013d4a0 2024-09-16T05:37:41.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.458+0000 7f69eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:41.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.458+0000 7f69eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:41.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.626+0000 7f6a04845640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:41.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.626+0000 7f6a04845640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:41.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.626+0000 7f6a04845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:41.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:41.626+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7f69e4005380 con 0x7f6a0013d4a0 2024-09-16T05:37:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.206+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v93) ==== 120+0+0 (unknown 3421688554 0 0) 0x7f69f8019350 con 0x7f6a0013d4a0 2024-09-16T05:37:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.206+0000 7f69eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:42.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.206+0000 7f69eeffd640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-16T05:37:42.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.310+0000 7f6a04845640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:42.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.314+0000 7f6a04845640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:42.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.314+0000 7f6a04845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:42.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.314+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7f69e4002220 con 0x7f6a0013d4a0 2024-09-16T05:37:42.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.314+0000 7f69eeffd640 1 -- 172.21.15.63:0/1061389001 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v93) ==== 120+0+0 (unknown 3941489345 0 0) 0x7f69f801dc10 con 0x7f6a0013d4a0 2024-09-16T05:37:42.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.314+0000 7f69eeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:42.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.314+0000 7f69eeffd640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-16T05:37:42.319 INFO:tasks.workunit.client.0.smithi063.stderr:device 'foo' does not appear in the crush map 2024-09-16T05:37:42.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.318+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 >> v1:172.21.15.63:6812/3643207211 conn(0x7f69e0036930 legacy=0x7f69e0038df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:42.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.318+0000 7f6a04845640 10 monclient: shutdown 2024-09-16T05:37:42.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.318+0000 7f6a04845640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:42.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.318+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 >> v1:172.21.15.6:6790/0 conn(0x7f6a0013d4a0 legacy=0x7f6a001d3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:42.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.318+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 shutdown_connections 2024-09-16T05:37:42.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.318+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 >> 172.21.15.63:0/1061389001 conn(0x7f6a00075470 msgr2=0x7f6a000a71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:42.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.318+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 shutdown_connections 2024-09-16T05:37:42.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.318+0000 7f6a04845640 1 -- 172.21.15.63:0/1061389001 wait complete. 2024-09-16T05:37:42.337 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm osd.9 host2 2024-09-16T05:37:42.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient: get_monmap_and_config 2024-09-16T05:37:42.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient: build_initial_monmap 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient: monmap: 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:42.457870+0000 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:42.457870+0000 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:42.459 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:42.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 1 Processor -- start 2024-09-16T05:37:42.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 1 -- start start 2024-09-16T05:37:42.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient: init 2024-09-16T05:37:42.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient(hunting): picked mon.noname-b con 0x7f9c60136960 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient(hunting): picked mon.noname-a con 0x7f9c600a1c40 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient(hunting): picked mon.noname-c con 0x7f9c60137450 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c600a09c0 con 0x7f9c600a1c40 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c600a0f80 con 0x7f9c60137450 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c600a4bb0 con 0x7f9c60136960 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c64a2d640 10 monclient(hunting): authenticate will time out at 1119.235048s 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5ed9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9c60137450 legacy=0x7f9c6013f4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:42.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9c600a1c40 0x7f9c6013bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53466/0 (socket says 172.21.15.63:53466) 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5e59c640 1 --1- 172.21.15.63:0/3924240864 >> v1:172.21.15.6:6790/0 conn(0x7f9c60136960 0x7f9c60136d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48604/0 (socket says 172.21.15.63:48604) 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5dd9b640 1 -- 172.21.15.63:0/3924240864 learned_addr learned my addr 172.21.15.63:0/3924240864 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954823115 0 0) 0x7f9c600a4bb0 con 0x7f9c60136960 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 10 monclient(hunting): my global_id is 5285 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c4c003530 con 0x7f9c60136960 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 656351807 0 0) 0x7f9c600a09c0 con 0x7f9c600a1c40 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.454+0000 7f9c5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:42.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient(hunting): my global_id is 5298 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c600a4bb0 con 0x7f9c600a1c40 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 140412077 0 0) 0x7f9c4c003530 con 0x7f9c60136960 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 >> v1:172.21.15.63:6789/0 conn(0x7f9c60137450 legacy=0x7f9c6013f4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 >> v1:172.21.15.6:6789/0 conn(0x7f9c600a1c40 legacy=0x7f9c6013bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 monclient: found mon.noname-b 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c600a4e60 con 0x7f9c60136960 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: _finish_auth 0 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:42.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9c580019a0 con 0x7f9c60136960 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: dump: 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:42.853 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 5 monclient: authenticate success, global_id 5285 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c58001c00 con 0x7f9c60136960 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 1 -- 172.21.15.63:0/3924240864 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9c58001f50 con 0x7f9c60136960 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5d59a640 10 monclient: dump: 2024-09-16T05:37:42.854 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 4 set_mon_vals no callback set 2024-09-16T05:37:42.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient: shutdown 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- 172.21.15.63:0/3924240864 >> v1:172.21.15.6:6790/0 conn(0x7f9c60136960 legacy=0x7f9c60136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- 172.21.15.63:0/3924240864 shutdown_connections 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- 172.21.15.63:0/3924240864 >> 172.21.15.63:0/3924240864 conn(0x7f9c600a60a0 msgr2=0x7f9c600a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- 172.21.15.63:0/3924240864 shutdown_connections 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- 172.21.15.63:0/3924240864 wait complete. 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient: build_initial_monmap 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient: monmap: 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:42.856 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:42.857 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:42.857 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:42.857 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:42.857 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:42.857 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:42.857 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:42.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 Processor -- start 2024-09-16T05:37:42.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- start start 2024-09-16T05:37:42.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient: init 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient(hunting): picked mon.noname-c con 0x7f9c600a1c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient(hunting): picked mon.noname-b con 0x7f9c60136960 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient(hunting): picked mon.noname-a con 0x7f9c60137450 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c60121990 con 0x7f9c60137450 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c600a04c0 con 0x7f9c60136960 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5dd9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9c60136960 legacy=0x7f9c601d0670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c600a7940 con 0x7f9c600a1c40 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c64a2d640 10 monclient(hunting): authenticate will time out at 1119.239025s 2024-09-16T05:37:42.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5e59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9c600a1c40 0x7f9c60135060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48614/0 (socket says 172.21.15.63:48614) 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c5e59c640 1 -- 172.21.15.63:0/2256513831 learned_addr learned my addr 172.21.15.63:0/2256513831 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420773859 0 0) 0x7f9c600a7940 con 0x7f9c600a1c40 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 10 monclient(hunting): my global_id is 5288 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c40003530 con 0x7f9c600a1c40 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2283327591 0 0) 0x7f9c60121990 con 0x7f9c60137450 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:42.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 10 monclient(hunting): my global_id is 5301 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.458+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c600a7940 con 0x7f9c60137450 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1785686832 0 0) 0x7f9c40003530 con 0x7f9c600a1c40 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c60121990 con 0x7f9c600a1c40 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9c580018a0 con 0x7f9c600a1c40 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:42.868 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:42.869 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:42.869 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:42.869 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:42.869 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:42.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:42.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1018245826 0 0) 0x7f9c600a7940 con 0x7f9c60137450 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c40003530 con 0x7f9c60137450 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9c50003fc0 con 0x7f9c60137450 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.870 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2311797289 0 0) 0x7f9c60121990 con 0x7f9c600a1c40 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 >> v1:172.21.15.63:6789/0 conn(0x7f9c60136960 legacy=0x7f9c601d0670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 >> v1:172.21.15.6:6789/0 conn(0x7f9c60137450 legacy=0x7f9c601d3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 monclient: found mon.c 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:42.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c600a7410 con 0x7f9c600a1c40 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 5 monclient: authenticate success, global_id 5288 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 10 monclient: _renew_subs 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9c60122a20 con 0x7f9c600a1c40 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 10 monclient: _renew_subs 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9c601cb7b0 con 0x7f9c600a1c40 2024-09-16T05:37:42.872 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c58001bc0 con 0x7f9c600a1c40 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9c58004fc0 con 0x7f9c600a1c40 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 10 monclient: dump: 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:42.873 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c5cd99640 4 set_mon_vals no callback set 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c44005090 con 0x7f9c600a1c40 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9c5800fd60 con 0x7f9c600a1c40 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(93..93 src has 1..93) ==== 4947+0+0 (unknown 3716195268 0 0) 0x7f9c58044a40 con 0x7f9c600a1c40 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.462+0000 7f9c5dd9b640 10 monclient: get_auth_request con 0x7f9c40036980 auth_method 0 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.474+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9c58015760 con 0x7f9c600a1c40 2024-09-16T05:37:42.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.474+0000 7f9c56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:42.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.474+0000 7f9c56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:42.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.646+0000 7f9c64a2d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T05:37:42.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.646+0000 7f9c64a2d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T05:37:42.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.646+0000 7f9c64a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:42.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:42.646+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) -- 0x7f9c44005b90 con 0x7f9c600a1c40 2024-09-16T05:37:43.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.206+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 removed item id 9 name 'osd.9' from crush map v94) ==== 143+0+0 (unknown 3001319331 0 0) 0x7f9c580193b0 con 0x7f9c600a1c40 2024-09-16T05:37:43.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.206+0000 7f9c56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T05:37:43.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.206+0000 7f9c56ffd640 10 monclient: _finish_command 2 = system:0 removed item id 9 name 'osd.9' from crush map 2024-09-16T05:37:43.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.310+0000 7f9c64a2d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T05:37:43.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.310+0000 7f9c64a2d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T05:37:43.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.310+0000 7f9c64a2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:43.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.310+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) -- 0x7f9c440010a0 con 0x7f9c600a1c40 2024-09-16T05:37:43.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c56ffd640 1 -- 172.21.15.63:0/2256513831 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 device 'osd.9' does not appear in the crush map v94) ==== 145+0+0 (unknown 1602828898 0 0) 0x7f9c5801dc70 con 0x7f9c600a1c40 2024-09-16T05:37:43.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c56ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T05:37:43.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c56ffd640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T05:37:43.318 INFO:tasks.workunit.client.0.smithi063.stderr:device 'osd.9' does not appear in the crush map 2024-09-16T05:37:43.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9c40036980 legacy=0x7f9c40038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:43.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c64a2d640 10 monclient: shutdown 2024-09-16T05:37:43.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c64a2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:43.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 >> v1:172.21.15.6:6790/0 conn(0x7f9c600a1c40 legacy=0x7f9c60135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:43.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 shutdown_connections 2024-09-16T05:37:43.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 >> 172.21.15.63:0/2256513831 conn(0x7f9c600a60a0 msgr2=0x7f9c600a7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:43.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 shutdown_connections 2024-09-16T05:37:43.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.314+0000 7f9c64a2d640 1 -- 172.21.15.63:0/2256513831 wait complete. 2024-09-16T05:37:43.335 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm host2 2024-09-16T05:37:43.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient: get_monmap_and_config 2024-09-16T05:37:43.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient: build_initial_monmap 2024-09-16T05:37:43.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient: monmap: 2024-09-16T05:37:43.458 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:43.458 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:43.458 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:43.457558+0000 2024-09-16T05:37:43.458 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:43.457558+0000 2024-09-16T05:37:43.459 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:43.459 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:43.459 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:43.459 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:43.459 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:43.459 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:43.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 1 Processor -- start 2024-09-16T05:37:43.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 1 -- start start 2024-09-16T05:37:43.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient: init 2024-09-16T05:37:43.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:43.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:43.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient(hunting): picked mon.noname-a con 0x7faa1c136ac0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient(hunting): picked mon.noname-c con 0x7faa1c09fd80 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient(hunting): picked mon.noname-b con 0x7faa1c13d0b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1c072cb0 con 0x7faa1c136ac0 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1c136670 con 0x7faa1c09fd80 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1c0a14f0 con 0x7faa1c13d0b0 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 10 monclient(hunting): authenticate will time out at 1120.234497s 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7faa1c09fd80 legacy=0x7faa1c13b990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa20c07640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa1c136ac0 0x7faa1c136ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53480/0 (socket says 172.21.15.63:53480) 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa21408640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7faa1c13d0b0 0x7faa1c13f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48620/0 (socket says 172.21.15.63:48620) 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa20c07640 1 -- 172.21.15.63:0/1406819620 learned_addr learned my addr 172.21.15.63:0/1406819620 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2208994909 0 0) 0x7faa10001120 con 0x7faa1c136ac0 2024-09-16T05:37:43.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient(hunting): my global_id is 5304 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa0c003530 con 0x7faa1c136ac0 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3852180853 0 0) 0x7faa1c0a14f0 con 0x7faa1c13d0b0 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient(hunting): my global_id is 5291 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa10001120 con 0x7faa1c13d0b0 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2631360098 0 0) 0x7faa0c003530 con 0x7faa1c136ac0 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 >> v1:172.21.15.6:6790/0 conn(0x7faa1c13d0b0 legacy=0x7faa1c13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:43.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 >> v1:172.21.15.63:6789/0 conn(0x7faa1c09fd80 legacy=0x7faa1c13b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 monclient: found mon.noname-a 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1c0a0630 con 0x7faa1c136ac0 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa100017f0 con 0x7faa1c136ac0 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 5 monclient: authenticate success, global_id 5304 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa22e6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:43.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.454+0000 7faa1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa1b7fe640 10 monclient: dump: 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:43.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa10001a50 con 0x7faa1c136ac0 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa1b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa1b7fe640 1 -- 172.21.15.63:0/1406819620 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa100048b0 con 0x7faa1c136ac0 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 4 set_mon_vals no callback set 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa1b7fe640 10 monclient: dump: 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:43.466 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:43.467 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:43.467 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:43.467 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:43.467 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:43.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient: shutdown 2024-09-16T05:37:43.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:43.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- 172.21.15.63:0/1406819620 >> v1:172.21.15.6:6789/0 conn(0x7faa1c136ac0 legacy=0x7faa1c136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- 172.21.15.63:0/1406819620 shutdown_connections 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- 172.21.15.63:0/1406819620 >> 172.21.15.63:0/1406819620 conn(0x7faa1c0a1c90 msgr2=0x7faa1c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- 172.21.15.63:0/1406819620 shutdown_connections 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- 172.21.15.63:0/1406819620 wait complete. 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient: build_initial_monmap 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient: monmap: 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:43.472 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:43.473 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:43.473 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:43.473 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:43.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 Processor -- start 2024-09-16T05:37:43.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- start start 2024-09-16T05:37:43.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient: init 2024-09-16T05:37:43.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:43.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:43.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient(hunting): picked mon.noname-a con 0x7faa1c09fd80 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:43.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient(hunting): picked mon.noname-c con 0x7faa1c136ac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:43.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient(hunting): picked mon.noname-b con 0x7faa1c13d0b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:43.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1c11e6e0 con 0x7faa1c09fd80 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1c11d580 con 0x7faa1c13d0b0 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1c11e9c0 con 0x7faa1c136ac0 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa22e6b640 10 monclient(hunting): authenticate will time out at 1120.238278s 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa21408640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7faa1c13d0b0 legacy=0x7faa1c1d3db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa20c07640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa1c09fd80 0x7faa1c132c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53488/0 (socket says 172.21.15.63:53488) 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa20c07640 1 -- 172.21.15.63:0/1565909798 learned_addr learned my addr 172.21.15.63:0/1565909798 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2222841518 0 0) 0x7faa1c11e6e0 con 0x7faa1c09fd80 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): my global_id is 5307 2024-09-16T05:37:43.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa00003530 con 0x7faa1c09fd80 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635340202 0 0) 0x7faa1c11e9c0 con 0x7faa1c136ac0 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): my global_id is 5294 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa1c11e6e0 con 0x7faa1c136ac0 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2972943324 0 0) 0x7faa00003530 con 0x7faa1c09fd80 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa1c11e9c0 con 0x7faa1c09fd80 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa10000ee0 con 0x7faa1c09fd80 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): dump: 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:43.479 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2966001083 0 0) 0x7faa1c11e6e0 con 0x7faa1c136ac0 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa00003530 con 0x7faa1c136ac0 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa080040a0 con 0x7faa1c136ac0 2024-09-16T05:37:43.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.458+0000 7faa197fa640 10 monclient(hunting): dump: 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3979416780 0 0) 0x7faa1c11e9c0 con 0x7faa1c09fd80 2024-09-16T05:37:43.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 >> v1:172.21.15.6:6790/0 conn(0x7faa1c136ac0 legacy=0x7faa1c1d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 >> v1:172.21.15.63:6789/0 conn(0x7faa1c13d0b0 legacy=0x7faa1c1d3db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 monclient: found mon.a 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1c09d840 con 0x7faa1c09fd80 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 5 monclient: authenticate success, global_id 5307 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 10 monclient: _renew_subs 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa1c09d310 con 0x7faa1c09fd80 2024-09-16T05:37:43.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 10 monclient: _renew_subs 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faa1c11d300 con 0x7faa1c09fd80 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa10001a50 con 0x7faa1c09fd80 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa10004e70 con 0x7faa1c09fd80 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 10 monclient: dump: 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:43.483 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa1affd640 4 set_mon_vals no callback set 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9fc005090 con 0x7faa1c09fd80 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7faa1000fc10 con 0x7faa1c09fd80 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(94..94 src has 1..94) ==== 4926+0+0 (unknown 2411216767 0 0) 0x7faa100446f0 con 0x7faa1c09fd80 2024-09-16T05:37:43.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.462+0000 7faa1bfff640 10 monclient: get_auth_request con 0x7faa00036990 auth_method 0 2024-09-16T05:37:43.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.474+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7faa1c072cb0 con 0x7faa1c09fd80 2024-09-16T05:37:43.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.474+0000 7faa197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:43.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.474+0000 7faa197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:43.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.642+0000 7faa22e6b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T05:37:43.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.642+0000 7faa22e6b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T05:37:43.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.642+0000 7faa22e6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:43.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:43.642+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) -- 0x7fa9fc005380 con 0x7faa1c09fd80 2024-09-16T05:37:44.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.218+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v95) ==== 124+0+0 (unknown 2548565269 0 0) 0x7faa10015550 con 0x7faa1c09fd80 2024-09-16T05:37:44.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.218+0000 7faa197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T05:37:44.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.218+0000 7faa197fa640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T05:37:44.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.322+0000 7faa22e6b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T05:37:44.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.322+0000 7faa22e6b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T05:37:44.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.322+0000 7faa22e6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:44.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.322+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) -- 0x7fa9fc0010a0 con 0x7faa1c09fd80 2024-09-16T05:37:44.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa197fa640 1 -- 172.21.15.63:0/1565909798 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 device 'host2' does not appear in the crush map v95) ==== 124+0+0 (unknown 2269371625 0 0) 0x7faa10014e50 con 0x7faa1c09fd80 2024-09-16T05:37:44.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T05:37:44.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa197fa640 10 monclient: _finish_command 3 = system:0 device 'host2' does not appear in the crush map 2024-09-16T05:37:44.329 INFO:tasks.workunit.client.0.smithi063.stderr:device 'host2' does not appear in the crush map 2024-09-16T05:37:44.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 >> v1:172.21.15.63:6812/3643207211 conn(0x7faa00036990 legacy=0x7faa00038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:44.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa22e6b640 10 monclient: shutdown 2024-09-16T05:37:44.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa22e6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:44.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 >> v1:172.21.15.6:6789/0 conn(0x7faa1c09fd80 legacy=0x7faa1c132c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:44.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 shutdown_connections 2024-09-16T05:37:44.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 >> 172.21.15.63:0/1565909798 conn(0x7faa1c0a1c90 msgr2=0x7faa1c09cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:44.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 shutdown_connections 2024-09-16T05:37:44.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.326+0000 7faa22e6b640 1 -- 172.21.15.63:0/1565909798 wait complete. 2024-09-16T05:37:44.346 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket foo host 2024-09-16T05:37:44.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.462+0000 7fc7a292d640 10 monclient: get_monmap_and_config 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.462+0000 7fc7a292d640 10 monclient: build_initial_monmap 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.462+0000 7fc7a292d640 10 monclient: monmap: 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:44.468093+0000 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:44.468093+0000 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:44.469 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:44.470 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:44.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 Processor -- start 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- start start 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient: init 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient(hunting): picked mon.noname-a con 0x7fc79c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient(hunting): picked mon.noname-c con 0x7fc79c0a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient(hunting): picked mon.noname-b con 0x7fc79c09b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79c074200 con 0x7fc79c09ac90 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79c09f900 con 0x7fc79c0a0280 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79c09e870 con 0x7fc79c09b780 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:44.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient(hunting): authenticate will time out at 1121.245016s 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc793fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc79c0a0280 legacy=0x7fc79c1398b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc79c09ac90 0x7fc79c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33490/0 (socket says 172.21.15.63:33490) 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79bfff640 1 -- 172.21.15.63:0/2800870528 learned_addr learned my addr 172.21.15.63:0/2800870528 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2071768033 0 0) 0x7fc79c074200 con 0x7fc79c09ac90 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient(hunting): my global_id is 5310 2024-09-16T05:37:44.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc788003530 con 0x7fc79c09ac90 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1407832943 0 0) 0x7fc79c09e870 con 0x7fc79c09b780 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient(hunting): my global_id is 5297 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc79c074200 con 0x7fc79c09b780 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3121788136 0 0) 0x7fc79c074200 con 0x7fc79c09b780 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 >> v1:172.21.15.63:6789/0 conn(0x7fc79c0a0280 legacy=0x7fc79c1398b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 >> v1:172.21.15.6:6789/0 conn(0x7fc79c09ac90 legacy=0x7fc79c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:37:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c141730 con 0x7fc79c09b780 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc794001980 con 0x7fc79c09b780 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: dump: 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:44.476 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 5 monclient: authenticate success, global_id 5297 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc794001c70 con 0x7fc79c09b780 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 4 set_mon_vals no callback set 2024-09-16T05:37:44.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 1 -- 172.21.15.63:0/2800870528 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc794002080 con 0x7fc79c09b780 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc79b7fe640 10 monclient: dump: 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:44.478 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 10 monclient: shutdown 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- 172.21.15.63:0/2800870528 >> v1:172.21.15.6:6790/0 conn(0x7fc79c09b780 legacy=0x7fc79c13cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- 172.21.15.63:0/2800870528 shutdown_connections 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- 172.21.15.63:0/2800870528 >> 172.21.15.63:0/2800870528 conn(0x7fc79c0a5290 msgr2=0x7fc79c0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- 172.21.15.63:0/2800870528 shutdown_connections 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.466+0000 7fc7a292d640 1 -- 172.21.15.63:0/2800870528 wait complete. 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: build_initial_monmap 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: monmap: 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:44.479 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:44.480 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:44.480 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:44.480 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:44.480 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:44.480 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:44.480 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:44.480 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:44.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 1 Processor -- start 2024-09-16T05:37:44.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 1 -- start start 2024-09-16T05:37:44.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: init 2024-09-16T05:37:44.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:44.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:44.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient(hunting): picked mon.noname-b con 0x7fc79c09ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient(hunting): picked mon.noname-a con 0x7fc79c09b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient(hunting): picked mon.noname-c con 0x7fc79c0a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79c13f2a0 con 0x7fc79c09b780 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79c09d250 con 0x7fc79c09ac90 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79c13f580 con 0x7fc79c0a0280 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc79bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc79c09ac90 legacy=0x7fc79c09c430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient(hunting): authenticate will time out at 1121.249197s 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a0eca640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc79c0a0280 0x7fc79c1d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38610/0 (socket says 172.21.15.63:38610) 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a0eca640 1 -- 172.21.15.63:0/2910098772 learned_addr learned my addr 172.21.15.63:0/2910098772 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2909529311 0 0) 0x7fc79c13f2a0 con 0x7fc79c09b780 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:44.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): my global_id is 5313 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc77c003530 con 0x7fc79c09b780 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211386200 0 0) 0x7fc79c13f580 con 0x7fc79c0a0280 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): my global_id is 5300 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc79c13f2a0 con 0x7fc79c0a0280 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1309247779 0 0) 0x7fc77c003530 con 0x7fc79c09b780 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc79c13f580 con 0x7fc79c09b780 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3591965578 0 0) 0x7fc79c13f2a0 con 0x7fc79c0a0280 2024-09-16T05:37:44.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc77c003530 con 0x7fc79c0a0280 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc784004110 con 0x7fc79c09b780 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): dump: 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:44.487 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc794001940 con 0x7fc79c0a0280 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient(hunting): dump: 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:44.488 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 325538242 0 0) 0x7fc79c13f580 con 0x7fc79c09b780 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 >> v1:172.21.15.6:6790/0 conn(0x7fc79c0a0280 legacy=0x7fc79c1d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 >> v1:172.21.15.63:6789/0 conn(0x7fc79c09ac90 legacy=0x7fc79c09c430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 monclient: found mon.a 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c06e0b0 con 0x7fc79c09b780 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:44.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:44.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:44.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 5 monclient: authenticate success, global_id 5313 2024-09-16T05:37:44.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: _renew_subs 2024-09-16T05:37:44.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:44.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc79c06db80 con 0x7fc79c09b780 2024-09-16T05:37:44.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: _renew_subs 2024-09-16T05:37:44.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc79c141420 con 0x7fc79c09b780 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc784004690 con 0x7fc79c09b780 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc7840049e0 con 0x7fc79c09b780 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc79affd640 4 set_mon_vals no callback set 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7997fa640 10 monclient: dump: 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:44.491 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.470+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc780005090 con 0x7fc79c09b780 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.474+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc784004c00 con 0x7fc79c09b780 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.474+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(95..95 src has 1..95) ==== 4791+0+0 (unknown 252731758 0 0) 0x7fc784044510 con 0x7fc79c09b780 2024-09-16T05:37:44.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.474+0000 7fc79bfff640 10 monclient: get_auth_request con 0x7fc77c036a10 auth_method 0 2024-09-16T05:37:44.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.486+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc784015330 con 0x7fc79c09b780 2024-09-16T05:37:44.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.486+0000 7fc7997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:44.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.486+0000 7fc7997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:44.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.650+0000 7fc7a292d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T05:37:44.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.650+0000 7fc7a292d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T05:37:44.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.650+0000 7fc7a292d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:44.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:44.650+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) -- 0x7fc780005b90 con 0x7fc79c09b780 2024-09-16T05:37:45.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.222+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 added bucket foo type host to crush map v96) ==== 138+0+0 (unknown 1532112770 0 0) 0x7fc784014c30 con 0x7fc79c09b780 2024-09-16T05:37:45.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.222+0000 7fc7997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T05:37:45.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.222+0000 7fc7997fa640 10 monclient: _finish_command 2 = system:0 added bucket foo type host to crush map 2024-09-16T05:37:45.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7a292d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T05:37:45.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7a292d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T05:37:45.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7a292d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:45.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) -- 0x7fc780005610 con 0x7fc79c09b780 2024-09-16T05:37:45.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7997fa640 1 -- 172.21.15.63:0/2910098772 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 bucket 'foo' already exists v96) ==== 126+0+0 (unknown 774400408 0 0) 0x7fc78401d7e0 con 0x7fc79c09b780 2024-09-16T05:37:45.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7997fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T05:37:45.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7997fa640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-16T05:37:45.330 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'foo' already exists 2024-09-16T05:37:45.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc77c036a10 legacy=0x7fc77c038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:45.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7a292d640 10 monclient: shutdown 2024-09-16T05:37:45.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7a292d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:45.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.326+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 >> v1:172.21.15.6:6789/0 conn(0x7fc79c09b780 legacy=0x7fc79c09cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:45.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.330+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 shutdown_connections 2024-09-16T05:37:45.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.330+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 >> 172.21.15.63:0/2910098772 conn(0x7fc79c0a5290 msgr2=0x7fc79c0a4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:45.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.330+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 shutdown_connections 2024-09-16T05:37:45.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.330+0000 7fc7a292d640 1 -- 172.21.15.63:0/2910098772 wait complete. 2024-09-16T05:37:45.348 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush move foo root=default rack=localrack 2024-09-16T05:37:45.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient: get_monmap_and_config 2024-09-16T05:37:45.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient: build_initial_monmap 2024-09-16T05:37:45.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient: monmap: 2024-09-16T05:37:45.473 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:45.473 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:45.473 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:45.472635+0000 2024-09-16T05:37:45.473 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:45.472635+0000 2024-09-16T05:37:45.474 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:45.474 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:45.474 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:45.474 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:45.474 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:45.474 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 1 Processor -- start 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 1 -- start start 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient: init 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient(hunting): picked mon.noname-c con 0x7f55740a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient(hunting): picked mon.noname-b con 0x7f55740a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient(hunting): picked mon.noname-a con 0x7f55740a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55740a51e0 con 0x7f55740a4790 2024-09-16T05:37:45.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55740a0d60 con 0x7f55740a3ca0 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55740a1040 con 0x7f55740a4d90 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient(hunting): authenticate will time out at 1122.249669s 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5579a74640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f55740a3ca0 legacy=0x7f55740a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5579273640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f55740a4d90 0x7f557413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38616/0 (socket says 172.21.15.63:38616) 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5579273640 1 -- 172.21.15.63:0/3587890018 learned_addr learned my addr 172.21.15.63:0/3587890018 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557a275640 1 --1- 172.21.15.63:0/3587890018 >> v1:172.21.15.6:6789/0 conn(0x7f55740a4790 0x7f557413f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33510/0 (socket says 172.21.15.63:33510) 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 842389682 0 0) 0x7f55740a51e0 con 0x7f55740a4790 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient(hunting): _negotiate 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:45.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient(hunting): my global_id is 5316 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5560003530 con 0x7f55740a4790 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749737046 0 0) 0x7f55740a1040 con 0x7f55740a4d90 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient(hunting): _negotiate 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient(hunting): my global_id is 5303 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55740a51e0 con 0x7f55740a4d90 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3008624313 0 0) 0x7f5560003530 con 0x7f55740a4790 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 >> v1:172.21.15.6:6790/0 conn(0x7f55740a4d90 legacy=0x7f557413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 >> v1:172.21.15.63:6789/0 conn(0x7f55740a3ca0 legacy=0x7f55740a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: _finish_hunting 0 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 monclient: found mon.noname-a 2024-09-16T05:37:45.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55740a0810 con 0x7f55740a4790 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: _finish_auth 0 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: _check_auth_tickets 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f556c001620 con 0x7f55740a4790 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 5 monclient: authenticate success, global_id 5316 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: dump: 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:45.482 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f556c0018d0 con 0x7f55740a4790 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f557bcd8640 4 set_mon_vals no callback set 2024-09-16T05:37:45.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 1 -- 172.21.15.63:0/3587890018 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f556c001ca0 con 0x7f55740a4790 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.470+0000 7f5578a72640 10 monclient: dump: 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:45.484 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: shutdown 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- 172.21.15.63:0/3587890018 >> v1:172.21.15.6:6789/0 conn(0x7f55740a4790 legacy=0x7f557413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- 172.21.15.63:0/3587890018 shutdown_connections 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- 172.21.15.63:0/3587890018 >> 172.21.15.63:0/3587890018 conn(0x7f5574075470 msgr2=0x7f5574075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- 172.21.15.63:0/3587890018 shutdown_connections 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- 172.21.15.63:0/3587890018 wait complete. 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: build_initial_monmap 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: monmap: 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:45.485 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 Processor -- start 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- start start 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: init 2024-09-16T05:37:45.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient(hunting): picked mon.noname-a con 0x7f55740a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient(hunting): picked mon.noname-c con 0x7f55740a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient(hunting): picked mon.noname-b con 0x7f55740a4d90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55740a0500 con 0x7f55740a3ca0 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55741383f0 con 0x7f55740a4d90 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5574136de0 con 0x7f55740a4790 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient(hunting): authenticate will time out at 1122.253650s 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557a275640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f55740a4d90 legacy=0x7f55741d39d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f5579a74640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f55740a3ca0 0x7f557413f870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33514/0 (socket says 172.21.15.63:33514) 2024-09-16T05:37:45.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f5579a74640 1 -- 172.21.15.63:0/4199614935 learned_addr learned my addr 172.21.15.63:0/4199614935 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211511926 0 0) 0x7f5574136de0 con 0x7f55740a4790 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): my global_id is 5306 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5558003530 con 0x7f55740a4790 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636551392 0 0) 0x7f5570003e20 con 0x7f55740a3ca0 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:45.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): my global_id is 5319 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5574136de0 con 0x7f55740a3ca0 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 190640575 0 0) 0x7f5574136de0 con 0x7f55740a3ca0 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5570003e20 con 0x7f55740a3ca0 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3166613177 0 0) 0x7f5558003530 con 0x7f55740a4790 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5574136de0 con 0x7f55740a4790 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f55700046b0 con 0x7f55740a3ca0 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): dump: 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:45.491 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f55680018c0 con 0x7f55740a4790 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:45.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient(hunting): dump: 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3044875105 0 0) 0x7f5570003e20 con 0x7f55740a3ca0 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 >> v1:172.21.15.6:6790/0 conn(0x7f55740a4790 legacy=0x7f55741d02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:45.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 >> v1:172.21.15.63:6789/0 conn(0x7f55740a4d90 legacy=0x7f55741d39d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 monclient: found mon.a 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5574139010 con 0x7f55740a3ca0 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 5 monclient: authenticate success, global_id 5319 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: _renew_subs 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:45.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f55741392f0 con 0x7f55740a3ca0 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: _renew_subs 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f557413ff80 con 0x7f55740a3ca0 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5570004bd0 con 0x7f55740a3ca0 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5570004fa0 con 0x7f55740a3ca0 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f5567fff640 4 set_mon_vals no callback set 2024-09-16T05:37:45.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.474+0000 7f55667fc640 10 monclient: dump: 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:45.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.478+0000 7f557bcd8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:45.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.478+0000 7f557bcd8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:45.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.478+0000 7f557bcd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:45.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.478+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f555c005090 con 0x7f55740a3ca0 2024-09-16T05:37:45.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.478+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f55700051c0 con 0x7f55740a3ca0 2024-09-16T05:37:45.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.478+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(96..96 src has 1..96) ==== 4818+0+0 (unknown 190231348 0 0) 0x7f5570044bb0 con 0x7f55740a3ca0 2024-09-16T05:37:45.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.478+0000 7f5579273640 10 monclient: get_auth_request con 0x7f5558036a50 auth_method 0 2024-09-16T05:37:45.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.490+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5570044e20 con 0x7f55740a3ca0 2024-09-16T05:37:45.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.490+0000 7f55667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:45.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.490+0000 7f55667fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:45.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.650+0000 7f557bcd8640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T05:37:45.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.650+0000 7f557bcd8640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T05:37:45.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.650+0000 7f557bcd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:45.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.650+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) -- 0x7f555c005380 con 0x7f55740a3ca0 2024-09-16T05:37:45.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.994+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v97) ==== 204+0+0 (unknown 1276283469 0 0) 0x7f55740a0500 con 0x7f55740a3ca0 2024-09-16T05:37:45.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.994+0000 7f55667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T05:37:45.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:45.994+0000 7f55667fc640 10 monclient: _finish_command 2 = system:0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T05:37:46.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.094+0000 7f557bcd8640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T05:37:46.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.094+0000 7f557bcd8640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T05:37:46.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.094+0000 7f557bcd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:46.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.094+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) -- 0x7f555c005ac0 con 0x7f55740a3ca0 2024-09-16T05:37:46.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.094+0000 7f55667fc640 1 -- 172.21.15.63:0/4199614935 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v97) ==== 214+0+0 (unknown 3056825855 0 0) 0x7f5570015950 con 0x7f55740a3ca0 2024-09-16T05:37:46.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.094+0000 7f55667fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T05:37:46.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.094+0000 7f55667fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T05:37:46.100 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T05:37:46.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.098+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5558036a50 legacy=0x7f5558038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:46.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.098+0000 7f557bcd8640 10 monclient: shutdown 2024-09-16T05:37:46.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.098+0000 7f557bcd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:46.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.098+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 >> v1:172.21.15.6:6789/0 conn(0x7f55740a3ca0 legacy=0x7f557413f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:46.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.098+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 shutdown_connections 2024-09-16T05:37:46.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.098+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 >> 172.21.15.63:0/4199614935 conn(0x7f5574075470 msgr2=0x7f55740a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:46.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.098+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 shutdown_connections 2024-09-16T05:37:46.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.098+0000 7f557bcd8640 1 -- 172.21.15.63:0/4199614935 wait complete. 2024-09-16T05:37:46.116 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush create-or-move osd.7 1.0 root=default 2024-09-16T05:37:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.234+0000 7f0cb2fcb640 10 monclient: get_monmap_and_config 2024-09-16T05:37:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.234+0000 7f0cb2fcb640 10 monclient: build_initial_monmap 2024-09-16T05:37:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.234+0000 7f0cb2fcb640 10 monclient: monmap: 2024-09-16T05:37:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:46.240 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:46.239572+0000 2024-09-16T05:37:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:46.239572+0000 2024-09-16T05:37:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:46.241 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:46.241 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.234+0000 7f0cb2fcb640 1 Processor -- start 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.234+0000 7f0cb2fcb640 1 -- start start 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.234+0000 7f0cb2fcb640 10 monclient: init 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 10 monclient(hunting): picked mon.noname-b con 0x7f0cac0a22a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 10 monclient(hunting): picked mon.noname-a con 0x7f0cac0731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 10 monclient(hunting): picked mon.noname-c con 0x7f0cac1395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cac09c580 con 0x7f0cac0731d0 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cac0a1300 con 0x7f0cac1395a0 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cac09d0f0 con 0x7f0cac0a22a0 2024-09-16T05:37:46.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:46.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb2fcb640 10 monclient(hunting): authenticate will time out at 1123.016729s 2024-09-16T05:37:46.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb1568640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0cac1395a0 legacy=0x7f0cac13b990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:46.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0ca3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0cac0731d0 0x7f0cac09fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33530/0 (socket says 172.21.15.63:33530) 2024-09-16T05:37:46.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0cb0d67640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0cac0a22a0 0x7f0cac09f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38632/0 (socket says 172.21.15.63:38632) 2024-09-16T05:37:46.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.238+0000 7f0ca3fff640 1 -- 172.21.15.63:0/2505689162 learned_addr learned my addr 172.21.15.63:0/2505689162 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:46.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750455816 0 0) 0x7f0cac09c580 con 0x7f0cac0731d0 2024-09-16T05:37:46.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:46.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:46.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:46.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient(hunting): my global_id is 5322 2024-09-16T05:37:46.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c94003530 con 0x7f0cac0731d0 2024-09-16T05:37:46.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4208608148 0 0) 0x7f0cac09d0f0 con 0x7f0cac0a22a0 2024-09-16T05:37:46.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:46.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:46.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:46.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient(hunting): my global_id is 5309 2024-09-16T05:37:46.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cac09c580 con 0x7f0cac0a22a0 2024-09-16T05:37:46.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2207733109 0 0) 0x7f0c94003530 con 0x7f0cac0731d0 2024-09-16T05:37:46.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 >> v1:172.21.15.6:6790/0 conn(0x7f0cac0a22a0 legacy=0x7f0cac09f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:46.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 >> v1:172.21.15.63:6789/0 conn(0x7f0cac1395a0 legacy=0x7f0cac13b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:46.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:46.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 monclient: found mon.noname-a 2024-09-16T05:37:46.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:46.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:46.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac09d3d0 con 0x7f0cac0731d0 2024-09-16T05:37:46.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:46.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:46.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:46.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c9c001550 con 0x7f0cac0731d0 2024-09-16T05:37:46.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:46.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:46.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: dump: 2024-09-16T05:37:46.263 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:46.263 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:46.263 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.263 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.263 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:46.263 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 5 monclient: authenticate success, global_id 5322 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c9c0017b0 con 0x7f0cac0731d0 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 1 -- 172.21.15.63:0/2505689162 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c9c001b80 con 0x7f0cac0731d0 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:46.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0ca37fe640 10 monclient: dump: 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:46.265 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 4 set_mon_vals no callback set 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 10 monclient: shutdown 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2505689162 >> v1:172.21.15.6:6789/0 conn(0x7f0cac0731d0 legacy=0x7f0cac09fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2505689162 shutdown_connections 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2505689162 >> 172.21.15.63:0/2505689162 conn(0x7f0cac0a1e90 msgr2=0x7f0cac075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2505689162 shutdown_connections 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.242+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2505689162 wait complete. 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: build_initial_monmap 2024-09-16T05:37:46.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: monmap: 2024-09-16T05:37:46.267 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:46.267 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:46.267 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:46.267 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:46.267 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:46.267 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:46.268 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:46.268 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:46.268 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:46.268 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:46.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 1 Processor -- start 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 1 -- start start 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: init 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient(hunting): picked mon.noname-a con 0x7f0cac0731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient(hunting): picked mon.noname-b con 0x7f0cac0a22a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient(hunting): picked mon.noname-c con 0x7f0cac1395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cac0a42e0 con 0x7f0cac0731d0 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cac11c220 con 0x7f0cac0a22a0 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cac072a00 con 0x7f0cac1395a0 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:46.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient(hunting): authenticate will time out at 1123.025024s 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0cac0a22a0 legacy=0x7f0cac131b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb0d67640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0cac0731d0 0x7f0cac1c6c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33534/0 (socket says 172.21.15.63:33534) 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb0d67640 1 -- 172.21.15.63:0/2661876803 learned_addr learned my addr 172.21.15.63:0/2661876803 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3326344274 0 0) 0x7f0cac072a00 con 0x7f0cac1395a0 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): my global_id is 5312 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c98003530 con 0x7f0cac1395a0 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3751796107 0 0) 0x7f0cac09c580 con 0x7f0cac0731d0 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:46.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): my global_id is 5325 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cac072a00 con 0x7f0cac0731d0 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 648349279 0 0) 0x7f0c98003530 con 0x7f0cac1395a0 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0cac09c580 con 0x7f0cac1395a0 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4243127409 0 0) 0x7f0cac072a00 con 0x7f0cac0731d0 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c98003530 con 0x7f0cac0731d0 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ca4004340 con 0x7f0cac1395a0 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): dump: 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:46.271 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ca80018a0 con 0x7f0cac0731d0 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:46.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient(hunting): dump: 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:46.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 970202242 0 0) 0x7f0cac09c580 con 0x7f0cac1395a0 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 >> v1:172.21.15.63:6789/0 conn(0x7f0cac0a22a0 legacy=0x7f0cac131b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 >> v1:172.21.15.6:6789/0 conn(0x7f0cac0731d0 legacy=0x7f0cac1c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 monclient: found mon.c 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac071c80 con 0x7f0cac1395a0 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 5 monclient: authenticate success, global_id 5312 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: _renew_subs 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:46.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0cac11bc60 con 0x7f0cac1395a0 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: _renew_subs 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0cac132430 con 0x7f0cac1395a0 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ca40045a0 con 0x7f0cac1395a0 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ca4004970 con 0x7f0cac1395a0 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca17fa640 10 monclient: dump: 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.275 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:46.276 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:46.276 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:46.276 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:46.276 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:46.276 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:46.276 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:46.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.246+0000 7f0ca2ffd640 4 set_mon_vals no callback set 2024-09-16T05:37:46.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.250+0000 7f0cb2fcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:46.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.250+0000 7f0cb2fcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:46.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.250+0000 7f0cb2fcb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:46.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.250+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c8c005090 con 0x7f0cac1395a0 2024-09-16T05:37:46.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.250+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0ca4004b90 con 0x7f0cac1395a0 2024-09-16T05:37:46.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.250+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(97..97 src has 1..97) ==== 5115+0+0 (unknown 1302083681 0 0) 0x7f0ca40449d0 con 0x7f0cac1395a0 2024-09-16T05:37:46.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.250+0000 7f0cb0d67640 10 monclient: get_auth_request con 0x7f0c98036a10 auth_method 0 2024-09-16T05:37:46.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.262+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0ca40156b0 con 0x7f0cac1395a0 2024-09-16T05:37:46.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.262+0000 7f0ca17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:46.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.262+0000 7f0ca17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:46.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.434+0000 7f0cb2fcb640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T05:37:46.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.434+0000 7f0cb2fcb640 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T05:37:46.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.434+0000 7f0cb2fcb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:46.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:46.434+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) -- 0x7f0c8c005380 con 0x7f0cac1395a0 2024-09-16T05:37:47.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.246+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map v98) ==== 212+0+0 (unknown 3767814759 0 0) 0x7f0ca4014fb0 con 0x7f0cac1395a0 2024-09-16T05:37:47.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.246+0000 7f0ca17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T05:37:47.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.246+0000 7f0ca17fa640 10 monclient: _finish_command 2 = system:0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T05:37:47.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.358+0000 7f0cb2fcb640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T05:37:47.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.358+0000 7f0cb2fcb640 10 monclient: _send_command 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T05:37:47.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.358+0000 7f0cb2fcb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:47.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.358+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) -- 0x7f0c8c005c50 con 0x7f0cac1395a0 2024-09-16T05:37:47.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.358+0000 7f0ca17fa640 1 -- 172.21.15.63:0/2661876803 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map v98) ==== 211+0+0 (unknown 592684754 0 0) 0x7f0ca401db60 con 0x7f0cac1395a0 2024-09-16T05:37:47.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.358+0000 7f0ca17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T05:37:47.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.358+0000 7f0ca17fa640 10 monclient: _finish_command 3 = system:0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T05:37:47.364 INFO:tasks.workunit.client.0.smithi063.stderr:create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T05:37:47.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.362+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0c98036a10 legacy=0x7f0c98038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:47.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.362+0000 7f0cb2fcb640 10 monclient: shutdown 2024-09-16T05:37:47.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.362+0000 7f0cb2fcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:47.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.362+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 >> v1:172.21.15.6:6790/0 conn(0x7f0cac1395a0 legacy=0x7f0cac1cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:47.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.362+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 shutdown_connections 2024-09-16T05:37:47.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.362+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 >> 172.21.15.63:0/2661876803 conn(0x7f0cac0a1e90 msgr2=0x7f0cac0a2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:47.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.362+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 shutdown_connections 2024-09-16T05:37:47.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.362+0000 7f0cb2fcb640 1 -- 172.21.15.63:0/2661876803 wait complete. 2024-09-16T05:37:47.381 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush move osd.7 host=foo 2024-09-16T05:37:47.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient: get_monmap_and_config 2024-09-16T05:37:47.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient: build_initial_monmap 2024-09-16T05:37:47.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient: monmap: 2024-09-16T05:37:47.505 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:47.505 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:47.506 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:47.504668+0000 2024-09-16T05:37:47.506 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:47.504668+0000 2024-09-16T05:37:47.506 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:47.506 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:47.506 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:47.506 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:47.506 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:47.506 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:47.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 1 Processor -- start 2024-09-16T05:37:47.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 1 -- start start 2024-09-16T05:37:47.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient: init 2024-09-16T05:37:47.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:47.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:47.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient(hunting): picked mon.noname-c con 0x7f03e00a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:47.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient(hunting): picked mon.noname-a con 0x7f03e00a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:47.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient(hunting): picked mon.noname-b con 0x7f03e013d3c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03e00a4dd0 con 0x7f03e00a4790 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03e00a1870 con 0x7f03e00a3ca0 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03e00a5210 con 0x7f03e013d3c0 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient(hunting): authenticate will time out at 1124.281807s 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03de59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f03e00a3ca0 legacy=0x7f03e00a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03ded9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f03e013d3c0 0x7f03e013f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38642/0 (socket says 172.21.15.63:38642) 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03ddd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03e00a4790 0x7f03e013bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33536/0 (socket says 172.21.15.63:33536) 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03ded9d640 1 -- 172.21.15.63:0/3554459759 learned_addr learned my addr 172.21.15.63:0/3554459759 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205417176 0 0) 0x7f03e00a5210 con 0x7f03e013d3c0 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:47.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient(hunting): my global_id is 5315 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03cc003530 con 0x7f03e013d3c0 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 820472282 0 0) 0x7f03e00a4dd0 con 0x7f03e00a4790 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient(hunting): my global_id is 5328 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e00a5210 con 0x7f03e00a4790 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1228565550 0 0) 0x7f03cc003530 con 0x7f03e013d3c0 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 >> v1:172.21.15.63:6789/0 conn(0x7f03e00a3ca0 legacy=0x7f03e00a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 >> v1:172.21.15.6:6789/0 conn(0x7f03e00a4790 legacy=0x7f03e013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:47.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: _finish_hunting 0 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 monclient: found mon.noname-b 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e00a0fd0 con 0x7f03e013d3c0 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: _finish_auth 0 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: _check_auth_tickets 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f03d8001620 con 0x7f03e013d3c0 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: dump: 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:47.511 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 5 monclient: authenticate success, global_id 5315 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f03d80018d0 con 0x7f03e013d3c0 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:47.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 1 -- 172.21.15.63:0/3554459759 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f03d8001ca0 con 0x7f03e013d3c0 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03dd59a640 10 monclient: dump: 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:47.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:47.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 4 set_mon_vals no callback set 2024-09-16T05:37:47.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 10 monclient: shutdown 2024-09-16T05:37:47.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:47.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 1 -- 172.21.15.63:0/3554459759 >> v1:172.21.15.6:6790/0 conn(0x7f03e013d3c0 legacy=0x7f03e013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:47.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 1 -- 172.21.15.63:0/3554459759 shutdown_connections 2024-09-16T05:37:47.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.502+0000 7f03e4db2640 1 -- 172.21.15.63:0/3554459759 >> 172.21.15.63:0/3554459759 conn(0x7f03e00a6650 msgr2=0x7f03e0075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:47.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- 172.21.15.63:0/3554459759 shutdown_connections 2024-09-16T05:37:47.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- 172.21.15.63:0/3554459759 wait complete. 2024-09-16T05:37:47.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: build_initial_monmap 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: monmap: 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 Processor -- start 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- start start 2024-09-16T05:37:47.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: init 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient(hunting): picked mon.noname-c con 0x7f03e00a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient(hunting): picked mon.noname-a con 0x7f03e00a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient(hunting): picked mon.noname-b con 0x7f03e013d3c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03e0125e10 con 0x7f03e00a4790 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03e01272c0 con 0x7f03e013d3c0 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03e00a88d0 con 0x7f03e00a3ca0 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient(hunting): authenticate will time out at 1124.285524s 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03ded9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f03e013d3c0 legacy=0x7f03e01d3eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:47.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03ddd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03e00a4790 0x7f03e01d0780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33538/0 (socket says 172.21.15.63:33538) 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03ddd9b640 1 -- 172.21.15.63:0/202712417 learned_addr learned my addr 172.21.15.63:0/202712417 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093748050 0 0) 0x7f03e00a88d0 con 0x7f03e00a3ca0 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): my global_id is 5318 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03c0003560 con 0x7f03e00a3ca0 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 790458829 0 0) 0x7f03e0125e10 con 0x7f03e00a4790 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:47.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): my global_id is 5331 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e00a88d0 con 0x7f03e00a4790 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2307619482 0 0) 0x7f03c0003560 con 0x7f03e00a3ca0 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03e0125e10 con 0x7f03e00a3ca0 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f03d0004450 con 0x7f03e00a3ca0 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:47.522 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2794097064 0 0) 0x7f03e00a88d0 con 0x7f03e00a4790 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03c0003560 con 0x7f03e00a4790 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f03c80018c0 con 0x7f03e00a4790 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:47.523 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3050079285 0 0) 0x7f03e0125e10 con 0x7f03e00a3ca0 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 >> v1:172.21.15.63:6789/0 conn(0x7f03e013d3c0 legacy=0x7f03e01d3eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 >> v1:172.21.15.6:6789/0 conn(0x7f03e00a4790 legacy=0x7f03e01d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:47.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 monclient: found mon.c 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e0126fe0 con 0x7f03e00a3ca0 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 5 monclient: authenticate success, global_id 5318 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: _renew_subs 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f03e0125b90 con 0x7f03e00a3ca0 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: _renew_subs 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:47.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f03e0124670 con 0x7f03e00a3ca0 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f03d0004700 con 0x7f03e00a3ca0 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f03d0004ad0 con 0x7f03e00a3ca0 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03dcd99640 4 set_mon_vals no callback set 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03d6ffd640 10 monclient: dump: 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:47.526 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.506+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c4005090 con 0x7f03e00a3ca0 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.510+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f03d0004cf0 con 0x7f03e00a3ca0 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.510+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(98..98 src has 1..98) ==== 5136+0+0 (unknown 1693940265 0 0) 0x7f03d0044970 con 0x7f03e00a3ca0 2024-09-16T05:37:47.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.510+0000 7f03ddd9b640 10 monclient: get_auth_request con 0x7f03c0036a50 auth_method 0 2024-09-16T05:37:47.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.522+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f03d00156c0 con 0x7f03e00a3ca0 2024-09-16T05:37:47.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.522+0000 7f03d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:47.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.522+0000 7f03d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:47.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.694+0000 7f03e4db2640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T05:37:47.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.694+0000 7f03e4db2640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T05:37:47.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.694+0000 7f03e4db2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:47.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:47.694+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) -- 0x7f03c4005b90 con 0x7f03e00a3ca0 2024-09-16T05:37:48.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.246+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 moved item id 7 name 'osd.7' to location {host=foo} in crush map v99) ==== 165+0+0 (unknown 3392640570 0 0) 0x7f03d0019370 con 0x7f03e00a3ca0 2024-09-16T05:37:48.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.246+0000 7f03d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T05:37:48.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.246+0000 7f03d6ffd640 10 monclient: _finish_command 2 = system:0 moved item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T05:37:48.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.350+0000 7f03e4db2640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T05:37:48.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.350+0000 7f03e4db2640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T05:37:48.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.350+0000 7f03e4db2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:48.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.350+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) -- 0x7f03c4005610 con 0x7f03e00a3ca0 2024-09-16T05:37:48.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.354+0000 7f03d6ffd640 1 -- 172.21.15.63:0/202712417 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map v99) ==== 175+0+0 (unknown 3725501166 0 0) 0x7f03d001dc30 con 0x7f03e00a3ca0 2024-09-16T05:37:48.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.354+0000 7f03d6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T05:37:48.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.354+0000 7f03d6ffd640 10 monclient: _finish_command 3 = system:0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T05:37:48.359 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T05:37:48.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.354+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 >> v1:172.21.15.63:6812/3643207211 conn(0x7f03c0036a50 legacy=0x7f03c0038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.354+0000 7f03e4db2640 10 monclient: shutdown 2024-09-16T05:37:48.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.354+0000 7f03e4db2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:48.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.354+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 >> v1:172.21.15.6:6790/0 conn(0x7f03e00a3ca0 legacy=0x7f03e01cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.354+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 shutdown_connections 2024-09-16T05:37:48.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.358+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 >> 172.21.15.63:0/202712417 conn(0x7f03e00a6650 msgr2=0x7f03e0072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:48.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.358+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 shutdown_connections 2024-09-16T05:37:48.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.358+0000 7f03e4db2640 1 -- 172.21.15.63:0/202712417 wait complete. 2024-09-16T05:37:48.376 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd find osd.7 2024-09-16T05:37:48.376 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep host 2024-09-16T05:37:48.376 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep foo 2024-09-16T05:37:48.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.490+0000 7f3da30be640 10 monclient: get_monmap_and_config 2024-09-16T05:37:48.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.490+0000 7f3da30be640 10 monclient: build_initial_monmap 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.490+0000 7f3da30be640 10 monclient: monmap: 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:48.496073+0000 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:48.496073+0000 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:48.497 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 Processor -- start 2024-09-16T05:37:48.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- start start 2024-09-16T05:37:48.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient: init 2024-09-16T05:37:48.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:48.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:48.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient(hunting): picked mon.noname-b con 0x7f3d9c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:48.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient(hunting): picked mon.noname-c con 0x7f3d9c09b070 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:48.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient(hunting): picked mon.noname-a con 0x7f3d9c13a9a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:48.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d9c0722a0 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d9c0a0510 con 0x7f3d9c09b070 2024-09-16T05:37:48.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d9c09e660 con 0x7f3d9c09ac90 2024-09-16T05:37:48.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient(hunting): authenticate will time out at 1125.273014s 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d93fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3d9c09b070 legacy=0x7f3d9c09b480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da165b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3d9c13a9a0 0x7f3d9c13cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33546/0 (socket says 172.21.15.63:33546) 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da165b640 1 -- 172.21.15.63:0/2285090102 learned_addr learned my addr 172.21.15.63:0/2285090102 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da0e5a640 1 --1- 172.21.15.63:0/2285090102 >> v1:172.21.15.6:6790/0 conn(0x7f3d9c09ac90 0x7f3d9c09f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38668/0 (socket says 172.21.15.63:38668) 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225688667 0 0) 0x7f3d9c09e660 con 0x7f3d9c09ac90 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:48.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient(hunting): my global_id is 5321 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d84003530 con 0x7f3d9c09ac90 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1627174821 0 0) 0x7f3d9c0722a0 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient(hunting): my global_id is 5334 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d9c09e660 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3373286649 0 0) 0x7f3d84003530 con 0x7f3d9c09ac90 2024-09-16T05:37:48.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 >> v1:172.21.15.63:6789/0 conn(0x7f3d9c09b070 legacy=0x7f3d9c09b480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 >> v1:172.21.15.6:6789/0 conn(0x7f3d9c13a9a0 legacy=0x7f3d9c13cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 monclient: found mon.noname-b 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9c1414e0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d98001620 con 0x7f3d9c09ac90 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 5 monclient: authenticate success, global_id 5321 2024-09-16T05:37:48.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: dump: 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.510 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d980018d0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 1 -- 172.21.15.63:0/2285090102 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d98001ca0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3d937fe640 10 monclient: dump: 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.511 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 4 set_mon_vals no callback set 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient: shutdown 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- 172.21.15.63:0/2285090102 >> v1:172.21.15.6:6790/0 conn(0x7f3d9c09ac90 legacy=0x7f3d9c09f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- 172.21.15.63:0/2285090102 shutdown_connections 2024-09-16T05:37:48.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- 172.21.15.63:0/2285090102 >> 172.21.15.63:0/2285090102 conn(0x7f3d9c0a1bf0 msgr2=0x7f3d9c0a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- 172.21.15.63:0/2285090102 shutdown_connections 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 1 -- 172.21.15.63:0/2285090102 wait complete. 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient: build_initial_monmap 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.494+0000 7f3da30be640 10 monclient: monmap: 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:48.513 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 1 Processor -- start 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 1 -- start start 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: init 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient(hunting): picked mon.noname-a con 0x7f3d9c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient(hunting): picked mon.noname-b con 0x7f3d9c09b070 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient(hunting): picked mon.noname-c con 0x7f3d9c13a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:48.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d9c1ca210 con 0x7f3d9c09ac90 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d9c13e060 con 0x7f3d9c09b070 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d9c1384c0 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d93fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3d9c09b070 legacy=0x7f3d9c1c9b00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient(hunting): authenticate will time out at 1125.276687s 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da0e5a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3d9c09ac90 0x7f3d9c1c93f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33558/0 (socket says 172.21.15.63:33558) 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da0e5a640 1 -- 172.21.15.63:0/547996387 learned_addr learned my addr 172.21.15.63:0/547996387 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2118955941 0 0) 0x7f3d9c1ca210 con 0x7f3d9c09ac90 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): my global_id is 5337 2024-09-16T05:37:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d88003530 con 0x7f3d9c09ac90 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 509660704 0 0) 0x7f3d9c1384c0 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): my global_id is 5324 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d9c1ca210 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2485962766 0 0) 0x7f3d88003530 con 0x7f3d9c09ac90 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d9c1384c0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d98001c20 con 0x7f3d9c09ac90 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:48.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): dump: 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3836339969 0 0) 0x7f3d9c1ca210 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d88003530 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d94001c40 con 0x7f3d9c13a9a0 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient(hunting): dump: 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.518 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3466849321 0 0) 0x7f3d9c1384c0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 >> v1:172.21.15.6:6790/0 conn(0x7f3d9c13a9a0 legacy=0x7f3d9c1d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 >> v1:172.21.15.63:6789/0 conn(0x7f3d9c09b070 legacy=0x7f3d9c1c9b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 monclient: found mon.a 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9c138910 con 0x7f3d9c09ac90 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:48.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 5 monclient: authenticate success, global_id 5337 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: _renew_subs 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9c138bf0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: _renew_subs 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d9c137770 con 0x7f3d9c09ac90 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d980018d0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d92ffd640 4 set_mon_vals no callback set 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3d98004b90 con 0x7f3d9c09ac90 2024-09-16T05:37:48.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 10 monclient: dump: 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:48.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:48.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d7c005090 con 0x7f3d9c09ac90 2024-09-16T05:37:48.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.498+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3d98004df0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.502+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(99..99 src has 1..99) ==== 5310+0+0 (unknown 2152251222 0 0) 0x7f3d980448e0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.502+0000 7f3d93fff640 10 monclient: get_auth_request con 0x7f3d88036a50 auth_method 0 2024-09-16T05:37:48.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.514+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3d98015500 con 0x7f3d9c09ac90 2024-09-16T05:37:48.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.514+0000 7f3d917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:48.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.514+0000 7f3d917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:48.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.686+0000 7f3da30be640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-16T05:37:48.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.686+0000 7f3da30be640 10 monclient: _send_command 2 [{"prefix": "osd find", "id": 7}] 2024-09-16T05:37:48.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.686+0000 7f3da30be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.686+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd find", "id": 7} v 0) -- 0x7f3d7c005380 con 0x7f3d9c09ac90 2024-09-16T05:37:48.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.686+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v99) ==== 65+0+226 (unknown 445245409 0 2157906930) 0x7f3d9801d9b0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.686+0000 7f3d917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd find", "id": 7}] 2024-09-16T05:37:48.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.686+0000 7f3d917fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:48.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.782+0000 7f3da30be640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-16T05:37:48.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.782+0000 7f3da30be640 10 monclient: _send_command 3 [{"prefix": "osd find", "id": 7}] 2024-09-16T05:37:48.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.782+0000 7f3da30be640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.782+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd find", "id": 7} v 0) -- 0x7f3d7c005ac0 con 0x7f3d9c09ac90 2024-09-16T05:37:48.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.782+0000 7f3d917fa640 1 -- 172.21.15.63:0/547996387 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v99) ==== 65+0+226 (unknown 445245409 0 2157906930) 0x7f3d98040530 con 0x7f3d9c09ac90 2024-09-16T05:37:48.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.782+0000 7f3d917fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd find", "id": 7}] 2024-09-16T05:37:48.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.782+0000 7f3d917fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:48.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.786+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3d88036a50 legacy=0x7f3d88038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.786+0000 7f3da30be640 10 monclient: shutdown 2024-09-16T05:37:48.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.786+0000 7f3da30be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:48.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.786+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 >> v1:172.21.15.6:6789/0 conn(0x7f3d9c09ac90 legacy=0x7f3d9c1c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.786+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 shutdown_connections 2024-09-16T05:37:48.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.786+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 >> 172.21.15.63:0/547996387 conn(0x7f3d9c0a1bf0 msgr2=0x7f3d9c140ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:48.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.786+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 shutdown_connections 2024-09-16T05:37:48.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.786+0000 7f3da30be640 1 -- 172.21.15.63:0/547996387 wait complete. 2024-09-16T05:37:48.805 INFO:tasks.workunit.client.0.smithi063.stdout: "host": "foo", 2024-09-16T05:37:48.805 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm osd.7 2024-09-16T05:37:48.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient: get_monmap_and_config 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient: build_initial_monmap 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient: monmap: 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:48.922619+0000 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:48.922619+0000 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:48.924 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 1 Processor -- start 2024-09-16T05:37:48.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 1 -- start start 2024-09-16T05:37:48.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient: init 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient(hunting): picked mon.noname-b con 0x7f40c00a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient(hunting): picked mon.noname-c con 0x7f40c009b450 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient(hunting): picked mon.noname-a con 0x7f40c009b8a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c009e970 con 0x7f40c009b8a0 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c009c370 con 0x7f40c009b450 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c009d620 con 0x7f40c00a08a0 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c69ca640 10 monclient(hunting): authenticate will time out at 1125.699418s 2024-09-16T05:37:48.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40bf7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f40c009b450 legacy=0x7f40c01389b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40bffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f40c00a08a0 0x7f40c009ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38692/0 (socket says 172.21.15.63:38692) 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40c4f67640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f40c009b8a0 0x7f40c013c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33570/0 (socket says 172.21.15.63:33570) 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.918+0000 7f40bffff640 1 -- 172.21.15.63:0/183816937 learned_addr learned my addr 172.21.15.63:0/183816937 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 428086958 0 0) 0x7f40c009d620 con 0x7f40c00a08a0 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient(hunting): my global_id is 5327 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40a8003530 con 0x7f40c00a08a0 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3580986324 0 0) 0x7f40c009e970 con 0x7f40c009b8a0 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:48.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient(hunting): my global_id is 5340 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40c009d620 con 0x7f40c009b8a0 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 486489310 0 0) 0x7f40c009d620 con 0x7f40c009b8a0 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 >> v1:172.21.15.6:6790/0 conn(0x7f40c00a08a0 legacy=0x7f40c009ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 >> v1:172.21.15.63:6789/0 conn(0x7f40c009b450 legacy=0x7f40c01389b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 monclient: found mon.noname-a 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c009d900 con 0x7f40c009b8a0 2024-09-16T05:37:48.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f40b8001900 con 0x7f40c009b8a0 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 5 monclient: authenticate success, global_id 5340 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: dump: 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.934 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40b8001b60 con 0x7f40c009b8a0 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 1 -- 172.21.15.63:0/183816937 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f40b8001f30 con 0x7f40c009b8a0 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 4 set_mon_vals no callback set 2024-09-16T05:37:48.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40beffd640 10 monclient: dump: 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.936 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient: shutdown 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- 172.21.15.63:0/183816937 >> v1:172.21.15.6:6789/0 conn(0x7f40c009b8a0 legacy=0x7f40c013c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- 172.21.15.63:0/183816937 shutdown_connections 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- 172.21.15.63:0/183816937 >> 172.21.15.63:0/183816937 conn(0x7f40c00a1c90 msgr2=0x7f40c00a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- 172.21.15.63:0/183816937 shutdown_connections 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- 172.21.15.63:0/183816937 wait complete. 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient: build_initial_monmap 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient: monmap: 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:48.937 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 Processor -- start 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- start start 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient: init 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:48.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient(hunting): picked mon.noname-b con 0x7f40c009b450 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient(hunting): picked mon.noname-c con 0x7f40c009b8a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient(hunting): picked mon.noname-a con 0x7f40c00a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c0120630 con 0x7f40c00a08a0 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c0122db0 con 0x7f40c009b450 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c011f8f0 con 0x7f40c009b8a0 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c69ca640 10 monclient(hunting): authenticate will time out at 1125.703440s 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40bffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f40c009b450 legacy=0x7f40c011ead0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c4f67640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f40c00a08a0 0x7f40c01cf2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33582/0 (socket says 172.21.15.63:33582) 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40c4f67640 1 -- 172.21.15.63:0/2423999053 learned_addr learned my addr 172.21.15.63:0/2423999053 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.922+0000 7f40bf7fe640 1 --1- 172.21.15.63:0/2423999053 >> v1:172.21.15.6:6790/0 conn(0x7f40c009b8a0 0x7f40c011f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38708/0 (socket says 172.21.15.63:38708) 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866419877 0 0) 0x7f40c0120630 con 0x7f40c00a08a0 2024-09-16T05:37:48.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): my global_id is 5343 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40ac003560 con 0x7f40c00a08a0 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1654104886 0 0) 0x7f40c011f8f0 con 0x7f40c009b8a0 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): my global_id is 5330 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40c0120630 con 0x7f40c009b8a0 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3424037086 0 0) 0x7f40ac003560 con 0x7f40c00a08a0 2024-09-16T05:37:48.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40c011f8f0 con 0x7f40c00a08a0 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f40b8001840 con 0x7f40c00a08a0 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): dump: 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.941 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 188460029 0 0) 0x7f40c0120630 con 0x7f40c009b8a0 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40ac003560 con 0x7f40c009b8a0 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f40b4004050 con 0x7f40c009b8a0 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient(hunting): dump: 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.942 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1405239440 0 0) 0x7f40c011f8f0 con 0x7f40c00a08a0 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 >> v1:172.21.15.6:6790/0 conn(0x7f40c009b8a0 legacy=0x7f40c011f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 >> v1:172.21.15.63:6789/0 conn(0x7f40c009b450 legacy=0x7f40c011ead0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient: _finish_hunting 0 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 monclient: found mon.a 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c009c0f0 con 0x7f40c00a08a0 2024-09-16T05:37:48.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient: _finish_auth 0 2024-09-16T05:37:48.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient: _check_auth_tickets 2024-09-16T05:37:48.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:48.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 5 monclient: authenticate success, global_id 5343 2024-09-16T05:37:48.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 10 monclient: _renew_subs 2024-09-16T05:37:48.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40c00a40f0 con 0x7f40c00a08a0 2024-09-16T05:37:48.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 10 monclient: _renew_subs 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f40c01226b0 con 0x7f40c00a08a0 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40b8001aa0 con 0x7f40c00a08a0 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bdffb640 4 set_mon_vals no callback set 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f40b8004e20 con 0x7f40c00a08a0 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 10 monclient: dump: 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.946 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:48.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a0005090 con 0x7f40c00a08a0 2024-09-16T05:37:48.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f40b8005040 con 0x7f40c00a08a0 2024-09-16T05:37:48.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(99..99 src has 1..99) ==== 5310+0+0 (unknown 2152251222 0 0) 0x7f40b8044ac0 con 0x7f40c00a08a0 2024-09-16T05:37:48.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.926+0000 7f40bffff640 10 monclient: get_auth_request con 0x7f40ac036aa0 auth_method 0 2024-09-16T05:37:48.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.938+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f40b80156e0 con 0x7f40c00a08a0 2024-09-16T05:37:48.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.938+0000 7f40bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:48.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:48.938+0000 7f40bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:49.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.106+0000 7f40c69ca640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T05:37:49.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.106+0000 7f40c69ca640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T05:37:49.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.106+0000 7f40c69ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.106+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) -- 0x7f40a0005380 con 0x7f40c00a08a0 2024-09-16T05:37:49.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.246+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 removed item id 7 name 'osd.7' from crush map v100) ==== 122+0+0 (unknown 1205123795 0 0) 0x7f40b8019330 con 0x7f40c00a08a0 2024-09-16T05:37:49.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.246+0000 7f40bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T05:37:49.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.246+0000 7f40bcff9640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-16T05:37:49.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.354+0000 7f40c69ca640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T05:37:49.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.354+0000 7f40c69ca640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T05:37:49.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.354+0000 7f40c69ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.354+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) -- 0x7f40a00010a0 con 0x7f40c00a08a0 2024-09-16T05:37:49.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.354+0000 7f40bcff9640 1 -- 172.21.15.63:0/2423999053 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 device 'osd.7' does not appear in the crush map v100) ==== 124+0+0 (unknown 3535862820 0 0) 0x7f40b801dbf0 con 0x7f40c00a08a0 2024-09-16T05:37:49.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.354+0000 7f40bcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T05:37:49.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.354+0000 7f40bcff9640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-16T05:37:49.359 INFO:tasks.workunit.client.0.smithi063.stderr:device 'osd.7' does not appear in the crush map 2024-09-16T05:37:49.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.358+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 >> v1:172.21.15.63:6812/3643207211 conn(0x7f40ac036aa0 legacy=0x7f40ac038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.358+0000 7f40c69ca640 10 monclient: shutdown 2024-09-16T05:37:49.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.358+0000 7f40c69ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:49.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.358+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 >> v1:172.21.15.6:6789/0 conn(0x7f40c00a08a0 legacy=0x7f40c01cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.358+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 shutdown_connections 2024-09-16T05:37:49.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.358+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 >> 172.21.15.63:0/2423999053 conn(0x7f40c00a1c90 msgr2=0x7f40c009f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:49.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.358+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 shutdown_connections 2024-09-16T05:37:49.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.358+0000 7f40c69ca640 1 -- 172.21.15.63:0/2423999053 wait complete. 2024-09-16T05:37:49.376 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm osd.9 2024-09-16T05:37:49.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient: get_monmap_and_config 2024-09-16T05:37:49.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient: build_initial_monmap 2024-09-16T05:37:49.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient: monmap: 2024-09-16T05:37:49.499 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:49.499 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.499 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:49.497897+0000 2024-09-16T05:37:49.499 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:49.497897+0000 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 1 Processor -- start 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 1 -- start start 2024-09-16T05:37:49.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient: init 2024-09-16T05:37:49.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:49.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:49.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f8136ac0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:49.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient(hunting): picked mon.noname-b con 0x7fa6f809fd80 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:49.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f813d0b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:49.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f8072cb0 con 0x7fa6f8136ac0 2024-09-16T05:37:49.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f8136670 con 0x7fa6f813d0b0 2024-09-16T05:37:49.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f80a14f0 con 0x7fa6f809fd80 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa700024640 10 monclient(hunting): authenticate will time out at 1126.274878s 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fe5c1640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa6f813d0b0 legacy=0x7fa6f813f4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fddc0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6f8136ac0 0x7fa6f8136ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33588/0 (socket says 172.21.15.63:33588) 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fddc0640 1 -- 172.21.15.63:0/62276118 learned_addr learned my addr 172.21.15.63:0/62276118 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682701363 0 0) 0x7fa6e8001120 con 0x7fa6f8136ac0 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 10 monclient(hunting): my global_id is 5346 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6e4003530 con 0x7fa6f8136ac0 2024-09-16T05:37:49.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3119560780 0 0) 0x7fa6f80a14f0 con 0x7fa6f809fd80 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 10 monclient(hunting): my global_id is 5333 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.494+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6e8001120 con 0x7fa6f809fd80 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2601898347 0 0) 0x7fa6e4003530 con 0x7fa6f8136ac0 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 >> v1:172.21.15.6:6790/0 conn(0x7fa6f809fd80 legacy=0x7fa6f813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 >> v1:172.21.15.63:6789/0 conn(0x7fa6f813d0b0 legacy=0x7fa6f813f4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 1 monclient: found mon.noname-a 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f80a0630 con 0x7fa6f8136ac0 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: _finish_auth 0 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa6e80017f0 con 0x7fa6f8136ac0 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: dump: 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.510 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 5 monclient: authenticate success, global_id 5346 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6e8001a50 con 0x7fa6f8136ac0 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 1 -- 172.21.15.63:0/62276118 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa6e80048b0 con 0x7fa6f8136ac0 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fcdbe640 10 monclient: dump: 2024-09-16T05:37:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 4 set_mon_vals no callback set 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient: shutdown 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:49.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- 172.21.15.63:0/62276118 >> v1:172.21.15.6:6789/0 conn(0x7fa6f8136ac0 legacy=0x7fa6f8136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- 172.21.15.63:0/62276118 shutdown_connections 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- 172.21.15.63:0/62276118 >> 172.21.15.63:0/62276118 conn(0x7fa6f80a1c90 msgr2=0x7fa6f80a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- 172.21.15.63:0/62276118 shutdown_connections 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- 172.21.15.63:0/62276118 wait complete. 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient: build_initial_monmap 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient: monmap: 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 Processor -- start 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- start start 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient: init 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f809fd80 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient(hunting): picked mon.noname-b con 0x7fa6f8136ac0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f813d0b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f811f8d0 con 0x7fa6f809fd80 2024-09-16T05:37:49.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f8120d80 con 0x7fa6f8136ac0 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f813c7b0 con 0x7fa6f813d0b0 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa700024640 10 monclient(hunting): authenticate will time out at 1126.278686s 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fd5bf640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa6f8136ac0 legacy=0x7fa6f81d0680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fddc0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6f809fd80 0x7fa6f8135290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33592/0 (socket says 172.21.15.63:33592) 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6fddc0640 1 -- 172.21.15.63:0/1396707671 learned_addr learned my addr 172.21.15.63:0/1396707671 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1753566884 0 0) 0x7fa6f813c7b0 con 0x7fa6f813d0b0 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 10 monclient(hunting): my global_id is 5336 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6dc003530 con 0x7fa6f813d0b0 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4233609194 0 0) 0x7fa6f811f8d0 con 0x7fa6f809fd80 2024-09-16T05:37:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 10 monclient(hunting): my global_id is 5349 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.498+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6f813c7b0 con 0x7fa6f809fd80 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1440075764 0 0) 0x7fa6f813c7b0 con 0x7fa6f809fd80 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6f811f8d0 con 0x7fa6f809fd80 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1475060519 0 0) 0x7fa6dc003530 con 0x7fa6f813d0b0 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6f813c7b0 con 0x7fa6f813d0b0 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa6e8001c00 con 0x7fa6f809fd80 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:49.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient(hunting): dump: 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:49.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa6f0004340 con 0x7fa6f813d0b0 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient(hunting): dump: 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3537682416 0 0) 0x7fa6f811f8d0 con 0x7fa6f809fd80 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 >> v1:172.21.15.6:6790/0 conn(0x7fa6f813d0b0 legacy=0x7fa6f81d3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 >> v1:172.21.15.63:6789/0 conn(0x7fa6f8136ac0 legacy=0x7fa6f81d0680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 monclient: found mon.a 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f8120aa0 con 0x7fa6f809fd80 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 5 monclient: authenticate success, global_id 5349 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 10 monclient: _renew_subs 2024-09-16T05:37:49.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f811f650 con 0x7fa6f809fd80 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 10 monclient: _renew_subs 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6f811e0e0 con 0x7fa6f809fd80 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6e8001a50 con 0x7fa6f809fd80 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa6e8004f40 con 0x7fa6f809fd80 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6effff640 4 set_mon_vals no callback set 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 10 monclient: dump: 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.520 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e0005090 con 0x7fa6f809fd80 2024-09-16T05:37:49.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa6e8005160 con 0x7fa6f809fd80 2024-09-16T05:37:49.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(100..100 src has 1..100) ==== 5260+0+0 (unknown 2942924701 0 0) 0x7fa6e8044b60 con 0x7fa6f809fd80 2024-09-16T05:37:49.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.502+0000 7fa6fd5bf640 10 monclient: get_auth_request con 0x7fa6dc036920 auth_method 0 2024-09-16T05:37:49.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.514+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa6f8072cb0 con 0x7fa6f809fd80 2024-09-16T05:37:49.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.514+0000 7fa6ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:49.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.514+0000 7fa6ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.674+0000 7fa700024640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T05:37:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.674+0000 7fa700024640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T05:37:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.674+0000 7fa700024640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.674+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) -- 0x7fa6e0005380 con 0x7fa6f809fd80 2024-09-16T05:37:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.678+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v100) ==== 124+0+0 (unknown 2255526108 0 0) 0x7fa6e80157b0 con 0x7fa6f809fd80 2024-09-16T05:37:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.678+0000 7fa6ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T05:37:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.678+0000 7fa6ee7fc640 10 monclient: _finish_command 2 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T05:37:49.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T05:37:49.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T05:37:49.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:49.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) -- 0x7fa6e0002220 con 0x7fa6f809fd80 2024-09-16T05:37:49.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa6ee7fc640 1 -- 172.21.15.63:0/1396707671 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v100) ==== 124+0+0 (unknown 2255526108 0 0) 0x7fa6e801dc60 con 0x7fa6f809fd80 2024-09-16T05:37:49.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa6ee7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T05:37:49.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa6ee7fc640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T05:37:49.779 INFO:tasks.workunit.client.0.smithi063.stderr:device 'osd.9' does not appear in the crush map 2024-09-16T05:37:49.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa6dc036920 legacy=0x7fa6dc038de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 10 monclient: shutdown 2024-09-16T05:37:49.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:49.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 >> v1:172.21.15.6:6789/0 conn(0x7fa6f809fd80 legacy=0x7fa6f8135290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.774+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 shutdown_connections 2024-09-16T05:37:49.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.778+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 >> 172.21.15.63:0/1396707671 conn(0x7fa6f80a1c90 msgr2=0x7fa6f80a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:49.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.778+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 shutdown_connections 2024-09-16T05:37:49.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.778+0000 7fa700024640 1 -- 172.21.15.63:0/1396707671 wait complete. 2024-09-16T05:37:49.794 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm foo 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient: get_monmap_and_config 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient: build_initial_monmap 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient: monmap: 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:49.914395+0000 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:49.914395+0000 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:49.915 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.916 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:49.916 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:49.916 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:49.916 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 1 Processor -- start 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 1 -- start start 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient: init 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient(hunting): picked mon.noname-c con 0x7ff92c09ead0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient(hunting): picked mon.noname-a con 0x7ff92c09eeb0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient(hunting): picked mon.noname-b con 0x7ff92c09f290 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c0730a0 con 0x7ff92c09eeb0 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c09d5a0 con 0x7ff92c09ead0 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c0a1d00 con 0x7ff92c09f290 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:49.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff932da3640 10 monclient(hunting): authenticate will time out at 1126.691309s 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff930b3f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff92c09ead0 legacy=0x7ff92c136c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff92bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff92c09eeb0 0x7ff92c13baa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33600/0 (socket says 172.21.15.63:33600) 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff931340640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff92c09f290 0x7ff92c13f1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38720/0 (socket says 172.21.15.63:38720) 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.910+0000 7ff92bfff640 1 -- 172.21.15.63:0/4145955025 learned_addr learned my addr 172.21.15.63:0/4145955025 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458536284 0 0) 0x7ff920001120 con 0x7ff92c09eeb0 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient(hunting): my global_id is 5352 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff914003530 con 0x7ff92c09eeb0 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284594092 0 0) 0x7ff92c0a1d00 con 0x7ff92c09f290 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:49.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient(hunting): my global_id is 5339 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff920001120 con 0x7ff92c09f290 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2451397676 0 0) 0x7ff920001120 con 0x7ff92c09f290 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 >> v1:172.21.15.63:6789/0 conn(0x7ff92c09ead0 legacy=0x7ff92c136c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 >> v1:172.21.15.6:6789/0 conn(0x7ff92c09eeb0 legacy=0x7ff92c13baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c075770 con 0x7ff92c09f290 2024-09-16T05:37:49.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff91c001580 con 0x7ff92c09f290 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: dump: 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.922 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 5 monclient: authenticate success, global_id 5339 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff91c001860 con 0x7ff92c09f290 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 1 -- 172.21.15.63:0/4145955025 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff91c001c70 con 0x7ff92c09f290 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff92b7fe640 10 monclient: dump: 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.923 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 4 set_mon_vals no callback set 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient: shutdown 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- 172.21.15.63:0/4145955025 >> v1:172.21.15.6:6790/0 conn(0x7ff92c09f290 legacy=0x7ff92c13f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- 172.21.15.63:0/4145955025 shutdown_connections 2024-09-16T05:37:49.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- 172.21.15.63:0/4145955025 >> 172.21.15.63:0/4145955025 conn(0x7ff92c0a58f0 msgr2=0x7ff92c0a5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:49.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- 172.21.15.63:0/4145955025 shutdown_connections 2024-09-16T05:37:49.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- 172.21.15.63:0/4145955025 wait complete. 2024-09-16T05:37:49.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient: build_initial_monmap 2024-09-16T05:37:49.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient: monmap: 2024-09-16T05:37:49.931 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:49.931 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 Processor -- start 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- start start 2024-09-16T05:37:49.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient: init 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient(hunting): picked mon.noname-a con 0x7ff92c09ead0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient(hunting): picked mon.noname-c con 0x7ff92c09eeb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient(hunting): picked mon.noname-b con 0x7ff92c09f290 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c0a39c0 con 0x7ff92c09ead0 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c094700 con 0x7ff92c09f290 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c08e2b0 con 0x7ff92c09eeb0 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff931340640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff92c09f290 legacy=0x7ff92c1d39b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff932da3640 10 monclient(hunting): authenticate will time out at 1126.695507s 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff930b3f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff92c09ead0 0x7ff92c08d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33604/0 (socket says 172.21.15.63:33604) 2024-09-16T05:37:49.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.914+0000 7ff930b3f640 1 -- 172.21.15.63:0/3015020175 learned_addr learned my addr 172.21.15.63:0/3015020175 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059287493 0 0) 0x7ff92c08e2b0 con 0x7ff92c09eeb0 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): my global_id is 5342 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff918003530 con 0x7ff92c09eeb0 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3935604600 0 0) 0x7ff924003e20 con 0x7ff92c09ead0 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): my global_id is 5355 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff92c08e2b0 con 0x7ff92c09ead0 2024-09-16T05:37:49.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2598194095 0 0) 0x7ff918003530 con 0x7ff92c09eeb0 2024-09-16T05:37:49.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff924003e20 con 0x7ff92c09eeb0 2024-09-16T05:37:49.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff920001590 con 0x7ff92c09eeb0 2024-09-16T05:37:49.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:49.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:49.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): dump: 2024-09-16T05:37:49.935 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.935 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.935 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:49.936 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 903563814 0 0) 0x7ff92c08e2b0 con 0x7ff92c09ead0 2024-09-16T05:37:49.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff918003530 con 0x7ff92c09ead0 2024-09-16T05:37:49.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff9240046b0 con 0x7ff92c09ead0 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient(hunting): dump: 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.938 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 255875395 0 0) 0x7ff924003e20 con 0x7ff92c09eeb0 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 >> v1:172.21.15.63:6789/0 conn(0x7ff92c09f290 legacy=0x7ff92c1d39b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 >> v1:172.21.15.6:6789/0 conn(0x7ff92c09ead0 legacy=0x7ff92c08d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 monclient: found mon.c 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c09c710 con 0x7ff92c09eeb0 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 5 monclient: authenticate success, global_id 5342 2024-09-16T05:37:49.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 10 monclient: _renew_subs 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c08ef30 con 0x7ff92c09eeb0 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 10 monclient: _renew_subs 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff92c08fcb0 con 0x7ff92c09eeb0 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9200017f0 con 0x7ff92c09eeb0 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff920004c40 con 0x7ff92c09eeb0 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 10 monclient: dump: 2024-09-16T05:37:49.940 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:49.941 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:49.941 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.941 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:49.941 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:49.941 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff92affd640 4 set_mon_vals no callback set 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff90c005090 con 0x7ff92c09eeb0 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff92000f9e0 con 0x7ff92c09eeb0 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(100..100 src has 1..100) ==== 5260+0+0 (unknown 2942924701 0 0) 0x7ff920044770 con 0x7ff92c09eeb0 2024-09-16T05:37:49.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.918+0000 7ff930b3f640 10 monclient: get_auth_request con 0x7ff918036a70 auth_method 0 2024-09-16T05:37:49.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.930+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff92c0730a0 con 0x7ff92c09eeb0 2024-09-16T05:37:49.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.930+0000 7ff9297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:49.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:49.930+0000 7ff9297fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:50.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.102+0000 7ff932da3640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:50.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.102+0000 7ff932da3640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:50.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.102+0000 7ff932da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:50.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.102+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7ff90c005380 con 0x7ff92c09eeb0 2024-09-16T05:37:50.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.254+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v101) ==== 120+0+0 (unknown 3542072299 0 0) 0x7ff920015360 con 0x7ff92c09eeb0 2024-09-16T05:37:50.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.254+0000 7ff9297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:50.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.254+0000 7ff9297fa640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-16T05:37:50.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.362+0000 7ff932da3640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:50.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.362+0000 7ff932da3640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:50.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.362+0000 7ff932da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:50.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.362+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7ff90c0010a0 con 0x7ff92c09eeb0 2024-09-16T05:37:50.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff9297fa640 1 -- 172.21.15.63:0/3015020175 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v101) ==== 120+0+0 (unknown 4060296128 0 0) 0x7ff920018fb0 con 0x7ff92c09eeb0 2024-09-16T05:37:50.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff9297fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T05:37:50.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff9297fa640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-16T05:37:50.370 INFO:tasks.workunit.client.0.smithi063.stderr:device 'foo' does not appear in the crush map 2024-09-16T05:37:50.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff918036a70 legacy=0x7ff918038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:50.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff932da3640 10 monclient: shutdown 2024-09-16T05:37:50.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff932da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:50.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 >> v1:172.21.15.6:6790/0 conn(0x7ff92c09eeb0 legacy=0x7ff92c08dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:50.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 shutdown_connections 2024-09-16T05:37:50.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 >> 172.21.15.63:0/3015020175 conn(0x7ff92c0a58f0 msgr2=0x7ff92c0a3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:50.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 shutdown_connections 2024-09-16T05:37:50.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.366+0000 7ff932da3640 1 -- 172.21.15.63:0/3015020175 wait complete. 2024-09-16T05:37:50.385 INFO:tasks.workunit.client.0.smithi063.stderr:++ ceph osd create 2024-09-16T05:37:50.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient: get_monmap_and_config 2024-09-16T05:37:50.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient: build_initial_monmap 2024-09-16T05:37:50.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient: monmap: 2024-09-16T05:37:50.504 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:50.504 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:50.504 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:50.502783+0000 2024-09-16T05:37:50.504 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:50.502783+0000 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 1 Processor -- start 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 1 -- start start 2024-09-16T05:37:50.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient: init 2024-09-16T05:37:50.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:50.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient(hunting): picked mon.noname-c con 0x7fa63009f890 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6300a0a30 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6300a0380 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6300a15e0 con 0x7fa6300a0380 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63009b750 con 0x7fa63009f890 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63009be00 con 0x7fa6300a0a30 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa636b4d640 10 monclient(hunting): authenticate will time out at 1127.279746s 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.498+0000 7fa6348e9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa63009f890 legacy=0x7fa63009fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:50.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa6350ea640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6300a0380 0x7fa630143840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33620/0 (socket says 172.21.15.63:33620) 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa6350ea640 1 -- 172.21.15.63:0/862886523 learned_addr learned my addr 172.21.15.63:0/862886523 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2738706019 0 0) 0x7fa6280014a0 con 0x7fa6300a0380 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient(hunting): my global_id is 5358 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa618003530 con 0x7fa6300a0380 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471968077 0 0) 0x7fa63009be00 con 0x7fa6300a0a30 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient(hunting): my global_id is 5345 2024-09-16T05:37:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6280014a0 con 0x7fa6300a0a30 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1757898133 0 0) 0x7fa6280014a0 con 0x7fa6300a0a30 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 >> v1:172.21.15.63:6789/0 conn(0x7fa63009f890 legacy=0x7fa63009fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 >> v1:172.21.15.6:6789/0 conn(0x7fa6300a0380 legacy=0x7fa630143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 monclient: found mon.noname-b 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63009bb80 con 0x7fa6300a0a30 2024-09-16T05:37:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa624001530 con 0x7fa6300a0a30 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 5 monclient: authenticate success, global_id 5345 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: dump: 2024-09-16T05:37:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa624001790 con 0x7fa6300a0a30 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 4 set_mon_vals no callback set 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 1 -- 172.21.15.63:0/862886523 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa624001b60 con 0x7fa6300a0a30 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62f7fe640 10 monclient: dump: 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.517 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient: shutdown 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- 172.21.15.63:0/862886523 >> v1:172.21.15.6:6790/0 conn(0x7fa6300a0a30 legacy=0x7fa630140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- 172.21.15.63:0/862886523 shutdown_connections 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- 172.21.15.63:0/862886523 >> 172.21.15.63:0/862886523 conn(0x7fa6300a1b60 msgr2=0x7fa6300a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- 172.21.15.63:0/862886523 shutdown_connections 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- 172.21.15.63:0/862886523 wait complete. 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient: build_initial_monmap 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient: monmap: 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 Processor -- start 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- start start 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient: init 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient(hunting): picked mon.noname-a con 0x7fa63009f890 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6300a0380 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6300a0a30 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6300a4360 con 0x7fa63009f890 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63013cf30 con 0x7fa6300a0380 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa63013d210 con 0x7fa6300a0a30 2024-09-16T05:37:50.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa636b4d640 10 monclient(hunting): authenticate will time out at 1127.283905s 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.502+0000 7fa62ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa6300a0380 legacy=0x7fa630139ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa6350ea640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa6300a0a30 0x7fa63013a1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38740/0 (socket says 172.21.15.63:38740) 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa6348e9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa63009f890 0x7fa6301393c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33628/0 (socket says 172.21.15.63:33628) 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa6350ea640 1 -- 172.21.15.63:0/872517811 learned_addr learned my addr 172.21.15.63:0/872517811 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3761739373 0 0) 0x7fa620003a50 con 0x7fa63009f890 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): my global_id is 5361 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61c003530 con 0x7fa63009f890 2024-09-16T05:37:50.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3635595148 0 0) 0x7fa63013d210 con 0x7fa6300a0a30 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): my global_id is 5348 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa620003a50 con 0x7fa6300a0a30 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3333484768 0 0) 0x7fa61c003530 con 0x7fa63009f890 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa63013d210 con 0x7fa63009f890 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa6200042a0 con 0x7fa63009f890 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): dump: 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:50.522 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1214222387 0 0) 0x7fa620003a50 con 0x7fa6300a0a30 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa61c003530 con 0x7fa6300a0a30 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa628001770 con 0x7fa6300a0a30 2024-09-16T05:37:50.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient(hunting): dump: 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:50.524 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2307575314 0 0) 0x7fa63013d210 con 0x7fa63009f890 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 >> v1:172.21.15.6:6790/0 conn(0x7fa6300a0a30 legacy=0x7fa63013a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 >> v1:172.21.15.63:6789/0 conn(0x7fa6300a0380 legacy=0x7fa630139ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 monclient: found mon.a 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63013cc50 con 0x7fa63009f890 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 5 monclient: authenticate success, global_id 5361 2024-09-16T05:37:50.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 10 monclient: _renew_subs 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6301d8e50 con 0x7fa63009f890 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 10 monclient: _renew_subs 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6301d91f0 con 0x7fa63009f890 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa620004500 con 0x7fa63009f890 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa620004850 con 0x7fa63009f890 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62effd640 4 set_mon_vals no callback set 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 10 monclient: dump: 2024-09-16T05:37:50.526 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:50.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa610005090 con 0x7fa63009f890 2024-09-16T05:37:50.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa620004a70 con 0x7fa63009f890 2024-09-16T05:37:50.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(101..101 src has 1..101) ==== 4962+0+0 (unknown 3699737540 0 0) 0x7fa6200447b0 con 0x7fa63009f890 2024-09-16T05:37:50.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.506+0000 7fa62ffff640 10 monclient: get_auth_request con 0x7fa61c036ac0 auth_method 0 2024-09-16T05:37:50.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.518+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa6300a4360 con 0x7fa63009f890 2024-09-16T05:37:50.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.518+0000 7fa62d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:50.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.518+0000 7fa62d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:50.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.686+0000 7fa636b4d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:37:50.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.686+0000 7fa636b4d640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T05:37:50.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.686+0000 7fa636b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:50.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:50.686+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7fa610005380 con 0x7fa63009f890 2024-09-16T05:37:51.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:51.250+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v102) ==== 58+0+3 (unknown 3069545335 0 520826459) 0x7fa6200154c0 con 0x7fa63009f890 2024-09-16T05:37:51.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:51.250+0000 7fa62d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T05:37:51.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:51.250+0000 7fa62d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:51.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:51.362+0000 7fa636b4d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:37:51.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:51.362+0000 7fa636b4d640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T05:37:51.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:51.362+0000 7fa636b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:51.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:51.362+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7fa610005ac0 con 0x7fa63009f890 2024-09-16T05:37:52.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.290+0000 7fa62d7fa640 1 -- 172.21.15.63:0/872517811 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v103) ==== 58+0+3 (unknown 58742147 0 212448812) 0x7fa620014dc0 con 0x7fa63009f890 2024-09-16T05:37:52.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.290+0000 7fa62d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T05:37:52.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.290+0000 7fa62d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:52.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.294+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa61c036ac0 legacy=0x7fa61c038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:52.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.294+0000 7fa636b4d640 10 monclient: shutdown 2024-09-16T05:37:52.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.294+0000 7fa636b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:52.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.294+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 >> v1:172.21.15.6:6789/0 conn(0x7fa63009f890 legacy=0x7fa6301393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:52.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.294+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 shutdown_connections 2024-09-16T05:37:52.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.294+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 >> 172.21.15.63:0/872517811 conn(0x7fa6300a1b60 msgr2=0x7fa6300a2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:52.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.294+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 shutdown_connections 2024-09-16T05:37:52.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.294+0000 7fa636b4d640 1 -- 172.21.15.63:0/872517811 wait complete. 2024-09-16T05:37:52.320 INFO:tasks.workunit.client.0.smithi063.stderr:+ o3=11 2024-09-16T05:37:52.320 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add 11 123 root=default 2024-09-16T05:37:52.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.442+0000 7f6120524640 10 monclient: get_monmap_and_config 2024-09-16T05:37:52.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.442+0000 7f6120524640 10 monclient: build_initial_monmap 2024-09-16T05:37:52.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.442+0000 7f6120524640 10 monclient: monmap: 2024-09-16T05:37:52.447 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:52.447 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:52.447 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:52.446644+0000 2024-09-16T05:37:52.447 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:52.446644+0000 2024-09-16T05:37:52.447 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:52.448 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:52.448 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:52.448 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:52.448 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:52.448 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:52.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.442+0000 7f6120524640 1 Processor -- start 2024-09-16T05:37:52.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.442+0000 7f6120524640 1 -- start start 2024-09-16T05:37:52.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.442+0000 7f6120524640 10 monclient: init 2024-09-16T05:37:52.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient(hunting): picked mon.noname-b con 0x7f61180a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient(hunting): picked mon.noname-a con 0x7f61180a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient(hunting): picked mon.noname-c con 0x7f61180a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61180a53b0 con 0x7f61180a07a0 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61180a0290 con 0x7f61180a4600 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61180a0dc0 con 0x7f61180a3b10 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient(hunting): authenticate will time out at 1129.224417s 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611eac1640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f61180a4600 legacy=0x7f611813f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611e2c0640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f61180a3b10 0x7f61180a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38752/0 (socket says 172.21.15.63:38752) 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611e2c0640 1 -- 172.21.15.63:0/2536930114 learned_addr learned my addr 172.21.15.63:0/2536930114 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:52.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611dabf640 1 --1- 172.21.15.63:0/2536930114 >> v1:172.21.15.6:6789/0 conn(0x7f61180a07a0 0x7f611813be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33630/0 (socket says 172.21.15.63:33630) 2024-09-16T05:37:52.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1261448903 0 0) 0x7f61180a53b0 con 0x7f61180a07a0 2024-09-16T05:37:52.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient(hunting): _negotiate 2024-09-16T05:37:52.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:52.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:52.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient(hunting): my global_id is 5364 2024-09-16T05:37:52.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6104003530 con 0x7f61180a07a0 2024-09-16T05:37:52.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824006667 0 0) 0x7f61180a0dc0 con 0x7f61180a3b10 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient(hunting): _negotiate 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient(hunting): my global_id is 5351 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61180a53b0 con 0x7f61180a3b10 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1223185830 0 0) 0x7f6104003530 con 0x7f61180a07a0 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 >> v1:172.21.15.6:6790/0 conn(0x7f61180a3b10 legacy=0x7f61180a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 >> v1:172.21.15.63:6789/0 conn(0x7f61180a4600 legacy=0x7f611813f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: _finish_hunting 0 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 monclient: found mon.noname-a 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:52.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61180a10a0 con 0x7f61180a07a0 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: _finish_auth 0 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: _check_auth_tickets 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6108001620 con 0x7f61180a07a0 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 5 monclient: authenticate success, global_id 5364 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:52.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: dump: 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61080018d0 con 0x7f61180a07a0 2024-09-16T05:37:52.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 4 set_mon_vals no callback set 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 1 -- 172.21.15.63:0/2536930114 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6108001ca0 con 0x7f61180a07a0 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f611d2be640 10 monclient: dump: 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:52.460 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient: shutdown 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- 172.21.15.63:0/2536930114 >> v1:172.21.15.6:6789/0 conn(0x7f61180a07a0 legacy=0x7f611813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- 172.21.15.63:0/2536930114 shutdown_connections 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- 172.21.15.63:0/2536930114 >> 172.21.15.63:0/2536930114 conn(0x7f6118075470 msgr2=0x7f6118075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- 172.21.15.63:0/2536930114 shutdown_connections 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- 172.21.15.63:0/2536930114 wait complete. 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient: build_initial_monmap 2024-09-16T05:37:52.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient: monmap: 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 Processor -- start 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 1 -- start start 2024-09-16T05:37:52.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.446+0000 7f6120524640 10 monclient: init 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient(hunting): picked mon.noname-a con 0x7f61180a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient(hunting): picked mon.noname-b con 0x7f61180a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient(hunting): picked mon.noname-c con 0x7f6118136790 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6118123810 con 0x7f61180a3b10 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61180a88d0 con 0x7f61180a4600 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6118121520 con 0x7f6118136790 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f611dabf640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f61180a4600 legacy=0x7f61181d05c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient(hunting): authenticate will time out at 1129.228308s 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f611e2c0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61180a3b10 0x7f6118136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33644/0 (socket says 172.21.15.63:33644) 2024-09-16T05:37:52.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f611e2c0640 1 -- 172.21.15.63:0/1377869421 learned_addr learned my addr 172.21.15.63:0/1377869421 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479620026 0 0) 0x7f6118123810 con 0x7f61180a3b10 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): my global_id is 5367 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60fc003530 con 0x7f61180a3b10 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2809111205 0 0) 0x7f6118121520 con 0x7f6118136790 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): my global_id is 5354 2024-09-16T05:37:52.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6118123810 con 0x7f6118136790 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 567497663 0 0) 0x7f60fc003530 con 0x7f61180a3b10 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6118121520 con 0x7f61180a3b10 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f61140018c0 con 0x7f61180a3b10 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): dump: 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:52.465 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2063263432 0 0) 0x7f6118123810 con 0x7f6118136790 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60fc003530 con 0x7f6118136790 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f61100043d0 con 0x7f6118136790 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient(hunting): dump: 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:52.466 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1169263382 0 0) 0x7f6118121520 con 0x7f61180a3b10 2024-09-16T05:37:52.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 >> v1:172.21.15.6:6790/0 conn(0x7f6118136790 legacy=0x7f61181d3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 >> v1:172.21.15.63:6789/0 conn(0x7f61180a4600 legacy=0x7f61181d05c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 monclient: found mon.a 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6118123530 con 0x7f61180a3b10 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient: _finish_auth 0 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 5 monclient: authenticate success, global_id 5367 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient: _renew_subs 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:52.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6118121e00 con 0x7f61180a3b10 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient: _renew_subs 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f61181cb690 con 0x7f61180a3b10 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61140047f0 con 0x7f61180a3b10 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6114004b40 con 0x7f61180a3b10 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 10 monclient: dump: 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.469 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f611cabd640 4 set_mon_vals no callback set 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6100005090 con 0x7f61180a3b10 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f611400f8e0 con 0x7f61180a3b10 2024-09-16T05:37:52.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(103..103 src has 1..103) ==== 5192+0+0 (unknown 353123105 0 0) 0x7f61140448a0 con 0x7f61180a3b10 2024-09-16T05:37:52.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.450+0000 7f611dabf640 10 monclient: get_auth_request con 0x7f60fc036a00 auth_method 0 2024-09-16T05:37:52.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.466+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6114019120 con 0x7f61180a3b10 2024-09-16T05:37:52.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.466+0000 7f610effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:52.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.466+0000 7f610effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:52.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.630+0000 7f6120524640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T05:37:52.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.630+0000 7f6120524640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T05:37:52.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.630+0000 7f6120524640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:52.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:52.630+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) -- 0x7f6100005380 con 0x7f61180a3b10 2024-09-16T05:37:53.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.290+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map v104) ==== 193+0+0 (unknown 2076039358 0 0) 0x7f611401d9e0 con 0x7f61180a3b10 2024-09-16T05:37:53.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.290+0000 7f610effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T05:37:53.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.290+0000 7f610effd640 10 monclient: _finish_command 2 = system:0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map 2024-09-16T05:37:53.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.394+0000 7f6120524640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T05:37:53.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.394+0000 7f6120524640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T05:37:53.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.394+0000 7f6120524640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:53.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.394+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) -- 0x7f6100005be0 con 0x7f61180a3b10 2024-09-16T05:37:53.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.394+0000 7f610effd640 1 -- 172.21.15.63:0/1377869421 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change v104) ==== 191+0+0 (unknown 4038475469 0 0) 0x7f6114013150 con 0x7f61180a3b10 2024-09-16T05:37:53.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.394+0000 7f610effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T05:37:53.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.394+0000 7f610effd640 10 monclient: _finish_command 3 = system:0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-16T05:37:53.400 INFO:tasks.workunit.client.0.smithi063.stderr:set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-16T05:37:53.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.398+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 >> v1:172.21.15.63:6812/3643207211 conn(0x7f60fc036a00 legacy=0x7f60fc038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.398+0000 7f6120524640 10 monclient: shutdown 2024-09-16T05:37:53.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.398+0000 7f6120524640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:53.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.398+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 >> v1:172.21.15.6:6789/0 conn(0x7f61180a3b10 legacy=0x7f6118136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.398+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 shutdown_connections 2024-09-16T05:37:53.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.398+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 >> 172.21.15.63:0/1377869421 conn(0x7f6118075470 msgr2=0x7f61180a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:53.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.398+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 shutdown_connections 2024-09-16T05:37:53.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.398+0000 7f6120524640 1 -- 172.21.15.63:0/1377869421 wait complete. 2024-09-16T05:37:53.418 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:53.418 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep osd.11 2024-09-16T05:37:53.418 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 123 2024-09-16T05:37:53.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.526+0000 7f5089e30640 10 monclient: get_monmap_and_config 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.526+0000 7f5089e30640 10 monclient: build_initial_monmap 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.526+0000 7f5089e30640 10 monclient: monmap: 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:53.531218+0000 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:53.531218+0000 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:53.532 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:53.533 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.526+0000 7f5089e30640 1 Processor -- start 2024-09-16T05:37:53.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.526+0000 7f5089e30640 1 -- start start 2024-09-16T05:37:53.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.526+0000 7f5089e30640 10 monclient: init 2024-09-16T05:37:53.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:53.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:53.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient(hunting): picked mon.noname-a con 0x7f508409e790 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:53.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient(hunting): picked mon.noname-c con 0x7f508409fd70 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:53.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient(hunting): picked mon.noname-b con 0x7f50841394f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:53.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508409c500 con 0x7f508409e790 2024-09-16T05:37:53.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5082ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f508409fd70 legacy=0x7f5084137dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:53.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50840a1260 con 0x7f508409fd70 2024-09-16T05:37:53.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508409c9c0 con 0x7f50841394f0 2024-09-16T05:37:53.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:53.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient(hunting): authenticate will time out at 1130.308393s 2024-09-16T05:37:53.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50837fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f508409e790 0x7f508409eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33658/0 (socket says 172.21.15.63:33658) 2024-09-16T05:37:53.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50837fe640 1 -- 172.21.15.63:0/18749185 learned_addr learned my addr 172.21.15.63:0/18749185 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:53.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082938931 0 0) 0x7f508409c500 con 0x7f508409e790 2024-09-16T05:37:53.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:53.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:53.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:53.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient(hunting): my global_id is 5370 2024-09-16T05:37:53.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f506c003530 con 0x7f508409e790 2024-09-16T05:37:53.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2094719433 0 0) 0x7f508409c9c0 con 0x7f50841394f0 2024-09-16T05:37:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient(hunting): my global_id is 5357 2024-09-16T05:37:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f508409c500 con 0x7f50841394f0 2024-09-16T05:37:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1896803863 0 0) 0x7f506c003530 con 0x7f508409e790 2024-09-16T05:37:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 >> v1:172.21.15.6:6790/0 conn(0x7f50841394f0 legacy=0x7f508413b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 >> v1:172.21.15.63:6789/0 conn(0x7f508409fd70 legacy=0x7f5084137dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 monclient: found mon.noname-a 2024-09-16T05:37:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50840a0180 con 0x7f508409e790 2024-09-16T05:37:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 5 monclient: authenticate success, global_id 5370 2024-09-16T05:37:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f507c0019a0 con 0x7f508409e790 2024-09-16T05:37:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: dump: 2024-09-16T05:37:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.552 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.552 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.552 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.552 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.552 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.552 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.552 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.552 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.553 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f507c001c00 con 0x7f508409e790 2024-09-16T05:37:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 4 set_mon_vals no callback set 2024-09-16T05:37:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 1 -- 172.21.15.63:0/18749185 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f507c001f50 con 0x7f508409e790 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f50827fc640 10 monclient: dump: 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.554 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient: shutdown 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- 172.21.15.63:0/18749185 >> v1:172.21.15.6:6789/0 conn(0x7f508409e790 legacy=0x7f508409eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- 172.21.15.63:0/18749185 shutdown_connections 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- 172.21.15.63:0/18749185 >> 172.21.15.63:0/18749185 conn(0x7f50840a1b60 msgr2=0x7f50840a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:53.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- 172.21.15.63:0/18749185 shutdown_connections 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- 172.21.15.63:0/18749185 wait complete. 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient: build_initial_monmap 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient: monmap: 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.556 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 Processor -- start 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 1 -- start start 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.530+0000 7f5089e30640 10 monclient: init 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient(hunting): picked mon.noname-a con 0x7f508409e790 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient(hunting): picked mon.noname-c con 0x7f508409fd70 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient(hunting): picked mon.noname-b con 0x7f50841394f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5084121bc0 con 0x7f508409e790 2024-09-16T05:37:53.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508409c0f0 con 0x7f50841394f0 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5084075b90 con 0x7f508409fd70 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5083fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f50841394f0 legacy=0x7f50841d4020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient(hunting): authenticate will time out at 1130.312173s 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f50837fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f508409e790 0x7f50841cb910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33674/0 (socket says 172.21.15.63:33674) 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f50837fe640 1 -- 172.21.15.63:0/991297016 learned_addr learned my addr 172.21.15.63:0/991297016 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f50837fe640 1 -- 172.21.15.63:0/991297016 >> v1:172.21.15.6:6789/0 conn(0x7f508409e790 legacy=0x7f50841cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f50837fe640 1 -- 172.21.15.63:0/991297016 >> v1:172.21.15.6:6789/0 conn(0x7f508409e790 legacy=0x7f50841cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f50837fe640 1 --1- 172.21.15.63:0/991297016 >> v1:172.21.15.6:6789/0 conn(0x7f508409e790 0x7f50841cb910 :-1 s=OPENED pgs=500 cs=1 l=1).handle_message read tag failed 2024-09-16T05:37:53.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f50837fe640 1 --1- 172.21.15.63:0/991297016 >> v1:172.21.15.6:6789/0 conn(0x7f508409e790 0x7f50841cb910 :-1 s=OPENED pgs=500 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 773606867 0 0) 0x7f5084075b90 con 0x7f508409fd70 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): my global_id is 5360 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5070003530 con 0x7f508409fd70 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3907963101 0 0) 0x7f5070003530 con 0x7f508409fd70 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5084075b90 con 0x7f508409fd70 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f50780040d0 con 0x7f508409fd70 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:53.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): dump: 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.560 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 723277971 0 0) 0x7f5084075b90 con 0x7f508409fd70 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 >> v1:172.21.15.63:6789/0 conn(0x7f50841394f0 legacy=0x7f50841d4020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 >> v1:172.21.15.6:6789/0 conn(0x7f508409e790 legacy=0x7f50841cb910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 monclient: found mon.c 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50840a30a0 con 0x7f508409fd70 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient: _finish_auth 0 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:53.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:53.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 5 monclient: authenticate success, global_id 5360 2024-09-16T05:37:53.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient: _renew_subs 2024-09-16T05:37:53.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5084121600 con 0x7f508409fd70 2024-09-16T05:37:53.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient: _renew_subs 2024-09-16T05:37:53.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f508411ec40 con 0x7f508409fd70 2024-09-16T05:37:53.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5078004380 con 0x7f508409fd70 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5078004750 con 0x7f508409fd70 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5081ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 10 monclient: dump: 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.563 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5064005090 con 0x7f508409fd70 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f50780048f0 con 0x7f508409fd70 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(104..104 src has 1..104) ==== 5214+0+0 (unknown 1643844226 0 0) 0x7f5078044720 con 0x7f508409fd70 2024-09-16T05:37:53.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.534+0000 7f50837fe640 10 monclient: get_auth_request con 0x7f5070036930 auth_method 0 2024-09-16T05:37:53.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.546+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5078015340 con 0x7f508409fd70 2024-09-16T05:37:53.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.546+0000 7f506bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:53.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.546+0000 7f506bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:53.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.706+0000 7f5089e30640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:53.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.706+0000 7f5089e30640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:53.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.706+0000 7f5089e30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.706+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f5064005b90 con 0x7f508409fd70 2024-09-16T05:37:53.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.706+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v104) ==== 56+0+1173 (unknown 4054326052 0 4180959835) 0x7f5078014c40 con 0x7f508409fd70 2024-09-16T05:37:53.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.706+0000 7f506bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:53.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.706+0000 7f506bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:53.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.802+0000 7f5089e30640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:53.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.802+0000 7f5089e30640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:53.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.802+0000 7f5089e30640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.802+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f50640008e0 con 0x7f508409fd70 2024-09-16T05:37:53.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.802+0000 7f506bfff640 1 -- 172.21.15.63:0/991297016 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v104) ==== 56+0+1173 (unknown 4054326052 0 4180959835) 0x7f5078018f90 con 0x7f508409fd70 2024-09-16T05:37:53.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.802+0000 7f506bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:53.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.802+0000 7f506bfff640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:53.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.806+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5070036930 legacy=0x7f5070038df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.806+0000 7f5089e30640 10 monclient: shutdown 2024-09-16T05:37:53.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.806+0000 7f5089e30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:53.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.806+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 >> v1:172.21.15.6:6790/0 conn(0x7f508409fd70 legacy=0x7f50841d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.806+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 shutdown_connections 2024-09-16T05:37:53.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.806+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 >> 172.21.15.63:0/991297016 conn(0x7f50840a1b60 msgr2=0x7f5084071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:53.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.806+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 shutdown_connections 2024-09-16T05:37:53.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.806+0000 7f5089e30640 1 -- 172.21.15.63:0/991297016 wait complete. 2024-09-16T05:37:53.823 INFO:tasks.workunit.client.0.smithi063.stdout: 11 123.00000 osd.11 down 1.00000 1.00000 2024-09-16T05:37:53.823 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush reweight osd.11 113 2024-09-16T05:37:53.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient: get_monmap_and_config 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient: build_initial_monmap 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient: monmap: 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:53.942052+0000 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:53.942052+0000 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:53.943 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 1 Processor -- start 2024-09-16T05:37:53.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 1 -- start start 2024-09-16T05:37:53.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient: init 2024-09-16T05:37:53.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:53.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient(hunting): picked mon.noname-a con 0x7feedc0a1520 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient(hunting): picked mon.noname-b con 0x7feedc09b540 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient(hunting): picked mon.noname-c con 0x7feedc0a0240 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc09f890 con 0x7feedc0a1520 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc072e80 con 0x7feedc0a0240 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc09ce30 con 0x7feedc09b540 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee3db9640 10 monclient(hunting): authenticate will time out at 1130.719555s 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee2356640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7feedc0a0240 legacy=0x7feedc13be90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee1b55640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feedc0a1520 0x7feedc09ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33686/0 (socket says 172.21.15.63:33686) 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee1b55640 1 -- 172.21.15.63:0/1396418557 learned_addr learned my addr 172.21.15.63:0/1396418557 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.938+0000 7feee1354640 1 --1- 172.21.15.63:0/1396418557 >> v1:172.21.15.6:6790/0 conn(0x7feedc09b540 0x7feedc09fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:38798/0 (socket says 172.21.15.63:38798) 2024-09-16T05:37:53.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719388752 0 0) 0x7feedc09f890 con 0x7feedc0a1520 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient(hunting): _negotiate 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient(hunting): my global_id is 5376 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feec8003530 con 0x7feedc0a1520 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224176354 0 0) 0x7feedc09ce30 con 0x7feedc09b540 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient(hunting): _negotiate 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient(hunting): my global_id is 5363 2024-09-16T05:37:53.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feedc09f890 con 0x7feedc09b540 2024-09-16T05:37:53.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4125383261 0 0) 0x7feec8003530 con 0x7feedc0a1520 2024-09-16T05:37:53.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 >> v1:172.21.15.6:6790/0 conn(0x7feedc09b540 legacy=0x7feedc09fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 >> v1:172.21.15.63:6789/0 conn(0x7feedc0a0240 legacy=0x7feedc13be90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:53.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: _finish_hunting 0 2024-09-16T05:37:53.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 monclient: found mon.noname-a 2024-09-16T05:37:53.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:53.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc09d110 con 0x7feedc0a1520 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: _finish_auth 0 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: _check_auth_tickets 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feed8001620 con 0x7feedc0a1520 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: dump: 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.953 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 5 monclient: authenticate success, global_id 5376 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feed8001880 con 0x7feedc0a1520 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 1 -- 172.21.15.63:0/1396418557 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feed8001bd0 con 0x7feedc0a1520 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:53.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee0b53640 10 monclient: dump: 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 4 set_mon_vals no callback set 2024-09-16T05:37:53.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient: shutdown 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- 172.21.15.63:0/1396418557 >> v1:172.21.15.6:6789/0 conn(0x7feedc0a1520 legacy=0x7feedc09ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- 172.21.15.63:0/1396418557 shutdown_connections 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- 172.21.15.63:0/1396418557 >> 172.21.15.63:0/1396418557 conn(0x7feedc0a5290 msgr2=0x7feedc0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- 172.21.15.63:0/1396418557 shutdown_connections 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- 172.21.15.63:0/1396418557 wait complete. 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient: build_initial_monmap 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient: monmap: 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:53.956 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 Processor -- start 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- start start 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient: init 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient(hunting): picked mon.noname-a con 0x7feedc09b540 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient(hunting): picked mon.noname-b con 0x7feedc0a0240 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:53.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient(hunting): picked mon.noname-c con 0x7feedc0a1520 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc123640 con 0x7feedc09b540 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc123060 con 0x7feedc0a0240 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc0a4030 con 0x7feedc0a1520 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee1354640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7feedc0a0240 legacy=0x7feedc1d01c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee3db9640 10 monclient(hunting): authenticate will time out at 1130.723653s 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee1b55640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feedc09b540 0x7feedc12fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33698/0 (socket says 172.21.15.63:33698) 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.942+0000 7feee1b55640 1 -- 172.21.15.63:0/364852605 learned_addr learned my addr 172.21.15.63:0/364852605 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202357563 0 0) 0x7feedc123640 con 0x7feedc09b540 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:53.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): my global_id is 5379 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feec0003560 con 0x7feedc09b540 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3601829597 0 0) 0x7feedc0a4030 con 0x7feedc0a1520 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): my global_id is 5366 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feedc123640 con 0x7feedc0a1520 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1796590232 0 0) 0x7feedc123640 con 0x7feedc0a1520 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feedc0a4030 con 0x7feedc0a1520 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feed4004340 con 0x7feedc0a1520 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:53.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): dump: 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1743669507 0 0) 0x7feec0003560 con 0x7feedc09b540 2024-09-16T05:37:53.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feedc123640 con 0x7feedc09b540 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feed80015a0 con 0x7feedc09b540 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient(hunting): dump: 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.961 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4140073046 0 0) 0x7feedc0a4030 con 0x7feedc0a1520 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 >> v1:172.21.15.63:6789/0 conn(0x7feedc0a0240 legacy=0x7feedc1d01c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 >> v1:172.21.15.6:6789/0 conn(0x7feedc09b540 legacy=0x7feedc12fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 monclient: found mon.c 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc1240a0 con 0x7feedc0a1520 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:53.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 5 monclient: authenticate success, global_id 5366 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 10 monclient: _renew_subs 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feedc124380 con 0x7feedc0a1520 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 10 monclient: _renew_subs 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feedc1caea0 con 0x7feedc0a1520 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feed40044e0 con 0x7feedc0a1520 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feed40048b0 con 0x7feedc0a1520 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:53.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 10 monclient: dump: 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:53.964 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed3fff640 4 set_mon_vals no callback set 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec4005090 con 0x7feedc0a1520 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7feed4004ad0 con 0x7feedc0a1520 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(104..104 src has 1..104) ==== 5214+0+0 (unknown 1643844226 0 0) 0x7feed40448e0 con 0x7feedc0a1520 2024-09-16T05:37:53.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.946+0000 7feee1b55640 10 monclient: get_auth_request con 0x7feec0036920 auth_method 0 2024-09-16T05:37:53.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.958+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7feed40154a0 con 0x7feedc0a1520 2024-09-16T05:37:53.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.958+0000 7feed27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:53.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:53.958+0000 7feed27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:54.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.126+0000 7feee3db9640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T05:37:54.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.126+0000 7feee3db9640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T05:37:54.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.126+0000 7feee3db9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:54.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.126+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) -- 0x7feec4005b90 con 0x7feedc0a1520 2024-09-16T05:37:54.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.314+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v105) ==== 156+0+0 (unknown 1438710079 0 0) 0x7feed4019150 con 0x7feedc0a1520 2024-09-16T05:37:54.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.314+0000 7feed27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T05:37:54.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.314+0000 7feed27fc640 10 monclient: _finish_command 2 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T05:37:54.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.410+0000 7feee3db9640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T05:37:54.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.410+0000 7feee3db9640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T05:37:54.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.410+0000 7feee3db9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:54.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:54.410+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) -- 0x7feec40010a0 con 0x7feedc0a1520 2024-09-16T05:37:55.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.318+0000 7feed27fc640 1 -- 172.21.15.63:0/364852605 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v106) ==== 156+0+0 (unknown 360924597 0 0) 0x7feed401da10 con 0x7feedc0a1520 2024-09-16T05:37:55.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.318+0000 7feed27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T05:37:55.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.318+0000 7feed27fc640 10 monclient: _finish_command 3 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T05:37:55.324 INFO:tasks.workunit.client.0.smithi063.stderr:reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T05:37:55.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.322+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 >> v1:172.21.15.63:6812/3643207211 conn(0x7feec0036920 legacy=0x7feec0038de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:55.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.322+0000 7feee3db9640 10 monclient: shutdown 2024-09-16T05:37:55.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.322+0000 7feee3db9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:55.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.322+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 >> v1:172.21.15.6:6790/0 conn(0x7feedc0a1520 legacy=0x7feedc1d38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:55.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.322+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 shutdown_connections 2024-09-16T05:37:55.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.322+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 >> 172.21.15.63:0/364852605 conn(0x7feedc0a5290 msgr2=0x7feedc0a4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:55.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.322+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 shutdown_connections 2024-09-16T05:37:55.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.322+0000 7feee3db9640 1 -- 172.21.15.63:0/364852605 wait complete. 2024-09-16T05:37:55.348 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush reweight osd.11 123456 2024-09-16T05:37:55.348 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:37:55.348 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush reweight osd.11 123456 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.466+0000 7f0273c76640 10 monclient: get_monmap_and_config 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.466+0000 7f0273c76640 10 monclient: build_initial_monmap 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.466+0000 7f0273c76640 10 monclient: monmap: 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:55.472344+0000 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:55.472344+0000 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.473 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:55.474 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:55.474 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:55.474 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 Processor -- start 2024-09-16T05:37:55.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- start start 2024-09-16T05:37:55.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient: init 2024-09-16T05:37:55.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:55.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:55.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient(hunting): picked mon.noname-a con 0x7f026c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:55.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient(hunting): picked mon.noname-c con 0x7f026c0a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:55.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient(hunting): picked mon.noname-b con 0x7f026c09b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026c074200 con 0x7f026c09ac90 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026c09f900 con 0x7f026c0a0280 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026c09e870 con 0x7f026c09b780 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient(hunting): authenticate will time out at 1132.249296s 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0271211640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f026c0a0280 legacy=0x7f026c1398b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0271a12640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f026c09ac90 0x7f026c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40966/0 (socket says 172.21.15.63:40966) 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0272213640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f026c09b780 0x7f026c13cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35544/0 (socket says 172.21.15.63:35544) 2024-09-16T05:37:55.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0271a12640 1 -- 172.21.15.63:0/1076845664 learned_addr learned my addr 172.21.15.63:0/1076845664 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3214371156 0 0) 0x7f026c074200 con 0x7f026c09ac90 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient(hunting): _negotiate 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient(hunting): my global_id is 5382 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0258003530 con 0x7f026c09ac90 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700502489 0 0) 0x7f026c09e870 con 0x7f026c09b780 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient(hunting): _negotiate 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient(hunting): my global_id is 5369 2024-09-16T05:37:55.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f026c074200 con 0x7f026c09b780 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3325502080 0 0) 0x7f0258003530 con 0x7f026c09ac90 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 >> v1:172.21.15.6:6790/0 conn(0x7f026c09b780 legacy=0x7f026c13cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 >> v1:172.21.15.63:6789/0 conn(0x7f026c0a0280 legacy=0x7f026c1398b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: _finish_hunting 0 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 monclient: found mon.noname-a 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c141730 con 0x7f026c09ac90 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: _finish_auth 0 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: _check_auth_tickets 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:55.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f02680019a0 con 0x7f026c09ac90 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: dump: 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.481 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 5 monclient: authenticate success, global_id 5382 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0268001c00 con 0x7f026c09ac90 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 1 -- 172.21.15.63:0/1076845664 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0268001f90 con 0x7f026c09ac90 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 4 set_mon_vals no callback set 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0270a10640 10 monclient: dump: 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.482 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 10 monclient: shutdown 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- 172.21.15.63:0/1076845664 >> v1:172.21.15.6:6789/0 conn(0x7f026c09ac90 legacy=0x7f026c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- 172.21.15.63:0/1076845664 shutdown_connections 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- 172.21.15.63:0/1076845664 >> 172.21.15.63:0/1076845664 conn(0x7f026c0a5290 msgr2=0x7f026c0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:55.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- 172.21.15.63:0/1076845664 shutdown_connections 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.470+0000 7f0273c76640 1 -- 172.21.15.63:0/1076845664 wait complete. 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: build_initial_monmap 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: monmap: 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.484 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:55.485 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:55.485 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:55.485 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 1 Processor -- start 2024-09-16T05:37:55.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 1 -- start start 2024-09-16T05:37:55.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: init 2024-09-16T05:37:55.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:55.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:37:55.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient(hunting): picked mon.noname-b con 0x7f026c09ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:55.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient(hunting): picked mon.noname-c con 0x7f026c09b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient(hunting): picked mon.noname-a con 0x7f026c0a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026c11db40 con 0x7f026c0a0280 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026c06dfd0 con 0x7f026c09ac90 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026c06da80 con 0x7f026c09b780 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0271a12640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f026c09ac90 legacy=0x7f026c1cb8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient(hunting): authenticate will time out at 1132.253078s 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0271211640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f026c09b780 0x7f026c1d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35552/0 (socket says 172.21.15.63:35552) 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0271211640 1 -- 172.21.15.63:0/2936496123 learned_addr learned my addr 172.21.15.63:0/2936496123 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1015763041 0 0) 0x7f026c06da80 con 0x7f026c09b780 2024-09-16T05:37:55.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): my global_id is 5372 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0250003530 con 0x7f026c09b780 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1071412554 0 0) 0x7f026c11db40 con 0x7f026c0a0280 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): my global_id is 5385 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f026c06da80 con 0x7f026c0a0280 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2078499002 0 0) 0x7f0250003530 con 0x7f026c09b780 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f026c11db40 con 0x7f026c09b780 2024-09-16T05:37:55.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f025c004150 con 0x7f026c09b780 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): dump: 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.492 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3176965983 0 0) 0x7f026c06da80 con 0x7f026c0a0280 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0250003530 con 0x7f026c0a0280 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0264001950 con 0x7f026c0a0280 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient(hunting): dump: 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.493 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1372334105 0 0) 0x7f026c11db40 con 0x7f026c09b780 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 >> v1:172.21.15.63:6789/0 conn(0x7f026c09ac90 legacy=0x7f026c1cb8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 >> v1:172.21.15.6:6789/0 conn(0x7f026c0a0280 legacy=0x7f026c1d3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 monclient: found mon.c 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c11d580 con 0x7f026c09b780 2024-09-16T05:37:55.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 5 monclient: authenticate success, global_id 5372 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: _renew_subs 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f026c11d860 con 0x7f026c09b780 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: _renew_subs 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f026c140760 con 0x7f026c09b780 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f025c004400 con 0x7f026c09b780 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f025c0047d0 con 0x7f026c09b780 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:55.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f02627fc640 10 monclient: dump: 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0263fff640 4 set_mon_vals no callback set 2024-09-16T05:37:55.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:55.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:55.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.474+0000 7f0273c76640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0254005090 con 0x7f026c09b780 2024-09-16T05:37:55.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.478+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f025c0049f0 con 0x7f026c09b780 2024-09-16T05:37:55.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.478+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(106..106 src has 1..106) ==== 5243+0+0 (unknown 814335168 0 0) 0x7f025c0446f0 con 0x7f026c09b780 2024-09-16T05:37:55.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.478+0000 7f0271a12640 10 monclient: get_auth_request con 0x7f02500369d0 auth_method 0 2024-09-16T05:37:55.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.490+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f025c0153d0 con 0x7f026c09b780 2024-09-16T05:37:55.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.490+0000 7f02627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:55.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.490+0000 7f02627fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:55.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.650+0000 7f0273c76640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T05:37:55.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.650+0000 7f0273c76640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T05:37:55.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.650+0000 7f0273c76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.650+0000 7f0273c76640 1 -- 172.21.15.63:0/2936496123 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0} v 0) -- 0x7f0254005b90 con 0x7f026c09b780 2024-09-16T05:37:55.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.654+0000 7f02627fc640 1 -- 172.21.15.63:0/2936496123 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}]=-75 (75) Value too large for defined data type v106) ==== 146+0+0 (unknown 3513823355 0 0) 0x7f025c019080 con 0x7f026c09b780 2024-09-16T05:37:55.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.654+0000 7f02627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T05:37:55.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.654+0000 7f02627fc640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-16T05:37:55.659 INFO:tasks.workunit.client.0.smithi063.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-16T05:37:55.664 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:37:55.664 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:37:55.664 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep osd.11 2024-09-16T05:37:55.665 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 113 2024-09-16T05:37:55.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.774+0000 7eff80478640 10 monclient: get_monmap_and_config 2024-09-16T05:37:55.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.774+0000 7eff80478640 10 monclient: build_initial_monmap 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.774+0000 7eff80478640 10 monclient: monmap: 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:55.779107+0000 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:55.779107+0000 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:55.780 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.774+0000 7eff80478640 1 Processor -- start 2024-09-16T05:37:55.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.774+0000 7eff80478640 1 -- start start 2024-09-16T05:37:55.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.774+0000 7eff80478640 10 monclient: init 2024-09-16T05:37:55.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.774+0000 7eff80478640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.774+0000 7eff80478640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): picked mon.noname-a con 0x7eff7809ae20 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): picked mon.noname-c con 0x7eff7809b910 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): picked mon.noname-b con 0x7eff78139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff780a0ff0 con 0x7eff7809ae20 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff7809da90 con 0x7eff7809b910 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff780a1430 con 0x7eff78139fc0 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): authenticate will time out at 1132.556102s 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7da13640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7eff7809b910 legacy=0x7eff781388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7e214640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7eff7809ae20 0x7eff7809b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40984/0 (socket says 172.21.15.63:40984) 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7ea15640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7eff78139fc0 0x7eff7813c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35562/0 (socket says 172.21.15.63:35562) 2024-09-16T05:37:55.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7e214640 1 -- 172.21.15.63:0/1006152387 learned_addr learned my addr 172.21.15.63:0/1006152387 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3935370053 0 0) 0x7eff780a1430 con 0x7eff78139fc0 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient(hunting): _negotiate 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient(hunting): my global_id is 5375 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff64005740 con 0x7eff78139fc0 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 186513684 0 0) 0x7eff780a0ff0 con 0x7eff7809ae20 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient(hunting): _negotiate 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:55.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient(hunting): my global_id is 5388 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff780a1430 con 0x7eff7809ae20 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607888154 0 0) 0x7eff64005740 con 0x7eff78139fc0 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 >> v1:172.21.15.63:6789/0 conn(0x7eff7809b910 legacy=0x7eff781388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 >> v1:172.21.15.6:6789/0 conn(0x7eff7809ae20 legacy=0x7eff7809b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: _finish_hunting 0 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 monclient: found mon.noname-b 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7809d1f0 con 0x7eff78139fc0 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: _finish_auth 0 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: _check_auth_tickets 2024-09-16T05:37:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff700018d0 con 0x7eff78139fc0 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: dump: 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.793 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 5 monclient: authenticate success, global_id 5375 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff70001b80 con 0x7eff78139fc0 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 1 -- 172.21.15.63:0/1006152387 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff70001f50 con 0x7eff78139fc0 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7d212640 10 monclient: dump: 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.794 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 4 set_mon_vals no callback set 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient: shutdown 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- 172.21.15.63:0/1006152387 >> v1:172.21.15.6:6790/0 conn(0x7eff78139fc0 legacy=0x7eff7813c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- 172.21.15.63:0/1006152387 shutdown_connections 2024-09-16T05:37:55.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- 172.21.15.63:0/1006152387 >> 172.21.15.63:0/1006152387 conn(0x7eff780a1b60 msgr2=0x7eff780a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- 172.21.15.63:0/1006152387 shutdown_connections 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- 172.21.15.63:0/1006152387 wait complete. 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient: build_initial_monmap 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient: monmap: 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:55.796 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 Processor -- start 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- start start 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient: init 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): picked mon.noname-c con 0x7eff7809ae20 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): picked mon.noname-b con 0x7eff7809b910 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): picked mon.noname-a con 0x7eff78139fc0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff781bf580 con 0x7eff78139fc0 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff78122e00 con 0x7eff7809b910 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff7da13640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7eff7809b910 legacy=0x7eff781d03b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff78122b20 con 0x7eff7809ae20 2024-09-16T05:37:55.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.778+0000 7eff80478640 10 monclient(hunting): authenticate will time out at 1132.559767s 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff7e214640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7eff7809ae20 0x7eff781346f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35574/0 (socket says 172.21.15.63:35574) 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff7ea15640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7eff78139fc0 0x7eff781d3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40996/0 (socket says 172.21.15.63:40996) 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff7e214640 1 -- 172.21.15.63:0/3675795534 learned_addr learned my addr 172.21.15.63:0/3675795534 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 547591278 0 0) 0x7eff78122b20 con 0x7eff7809ae20 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): my global_id is 5378 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff5c003530 con 0x7eff7809ae20 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3292040475 0 0) 0x7eff781bf580 con 0x7eff78139fc0 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:55.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): my global_id is 5391 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff78122b20 con 0x7eff78139fc0 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3596585238 0 0) 0x7eff5c003530 con 0x7eff7809ae20 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff781bf580 con 0x7eff7809ae20 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff740018c0 con 0x7eff7809ae20 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): dump: 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.799 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2682954358 0 0) 0x7eff78122b20 con 0x7eff78139fc0 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff5c003530 con 0x7eff78139fc0 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff70001ed0 con 0x7eff78139fc0 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:55.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient(hunting): dump: 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1177344669 0 0) 0x7eff781bf580 con 0x7eff7809ae20 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 >> v1:172.21.15.63:6789/0 conn(0x7eff7809b910 legacy=0x7eff781d03b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:55.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 >> v1:172.21.15.6:6789/0 conn(0x7eff78139fc0 legacy=0x7eff781d3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 monclient: found mon.c 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff781216d0 con 0x7eff7809ae20 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient: _finish_auth 0 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 5 monclient: authenticate success, global_id 5378 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 10 monclient: _renew_subs 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff78134e00 con 0x7eff7809ae20 2024-09-16T05:37:55.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 10 monclient: _renew_subs 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7eff78139580 con 0x7eff7809ae20 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff740047f0 con 0x7eff7809ae20 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7eff74004b40 con 0x7eff7809ae20 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6ffff640 4 set_mon_vals no callback set 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 10 monclient: dump: 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:55.803 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff60005090 con 0x7eff7809ae20 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7eff7400f8e0 con 0x7eff7809ae20 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(106..106 src has 1..106) ==== 5243+0+0 (unknown 814335168 0 0) 0x7eff740447c0 con 0x7eff7809ae20 2024-09-16T05:37:55.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.782+0000 7eff7da13640 10 monclient: get_auth_request con 0x7eff5c036a80 auth_method 0 2024-09-16T05:37:55.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.794+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7eff740153c0 con 0x7eff7809ae20 2024-09-16T05:37:55.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.794+0000 7eff6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:55.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.794+0000 7eff6effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:55.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.958+0000 7eff80478640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:55.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.958+0000 7eff80478640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:55.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.958+0000 7eff80478640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:55.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.958+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7eff60005b90 con 0x7eff7809ae20 2024-09-16T05:37:55.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.962+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v106) ==== 56+0+1173 (unknown 3369195045 0 3201743914) 0x7eff74014cc0 con 0x7eff7809ae20 2024-09-16T05:37:55.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.962+0000 7eff6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:37:55.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:55.962+0000 7eff6effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.054+0000 7eff80478640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:37:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.054+0000 7eff80478640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.054+0000 7eff80478640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.054+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7eff600008e0 con 0x7eff7809ae20 2024-09-16T05:37:56.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff6effd640 1 -- 172.21.15.63:0/3675795534 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v107) ==== 56+0+1173 (unknown 1453001693 0 3201743914) 0x7eff74019010 con 0x7eff7809ae20 2024-09-16T05:37:56.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff6effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:37:56.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff6effd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:37:56.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 >> v1:172.21.15.63:6812/3643207211 conn(0x7eff5c036a80 legacy=0x7eff5c038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:56.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff80478640 10 monclient: shutdown 2024-09-16T05:37:56.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff80478640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:56.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 >> v1:172.21.15.6:6790/0 conn(0x7eff7809ae20 legacy=0x7eff781346f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:56.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 shutdown_connections 2024-09-16T05:37:56.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 >> 172.21.15.63:0/3675795534 conn(0x7eff780a1b60 msgr2=0x7eff780a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:56.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 shutdown_connections 2024-09-16T05:37:56.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.058+0000 7eff80478640 1 -- 172.21.15.63:0/3675795534 wait complete. 2024-09-16T05:37:56.077 INFO:tasks.workunit.client.0.smithi063.stdout: 11 113.00000 osd.11 down 1.00000 1.00000 2024-09-16T05:37:56.077 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm osd.11 2024-09-16T05:37:56.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.190+0000 7f18b460a640 10 monclient: get_monmap_and_config 2024-09-16T05:37:56.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.190+0000 7f18b460a640 10 monclient: build_initial_monmap 2024-09-16T05:37:56.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.190+0000 7f18b460a640 10 monclient: monmap: 2024-09-16T05:37:56.196 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:56.196 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:56.196 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:56.195746+0000 2024-09-16T05:37:56.196 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:56.195746+0000 2024-09-16T05:37:56.197 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:56.197 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:56.197 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:56.197 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:56.197 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:56.197 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:56.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 Processor -- start 2024-09-16T05:37:56.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- start start 2024-09-16T05:37:56.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient: init 2024-09-16T05:37:56.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:56.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient(hunting): picked mon.noname-b con 0x7f18ac0a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient(hunting): picked mon.noname-a con 0x7f18ac0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient(hunting): picked mon.noname-c con 0x7f18ac0a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18ac0a4fd0 con 0x7f18ac0a08a0 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18ac0a5b40 con 0x7f18ac0a4790 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18ac0a1a70 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient(hunting): authenticate will time out at 1132.972819s 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b2ba7640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f18ac0a4790 legacy=0x7f18ac13f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b23a6640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f18ac0a3ca0 0x7f18ac0a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35588/0 (socket says 172.21.15.63:35588) 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b1ba5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f18ac0a08a0 0x7f18ac13beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41006/0 (socket says 172.21.15.63:41006) 2024-09-16T05:37:56.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b23a6640 1 -- 172.21.15.63:0/668279067 learned_addr learned my addr 172.21.15.63:0/668279067 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217329409 0 0) 0x7f18ac0a1a70 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient(hunting): _negotiate 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient(hunting): my global_id is 5381 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1898003530 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473047741 0 0) 0x7f18ac0a4fd0 con 0x7f18ac0a08a0 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient(hunting): _negotiate 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient(hunting): my global_id is 5394 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18ac0a1a70 con 0x7f18ac0a08a0 2024-09-16T05:37:56.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3204823728 0 0) 0x7f1898003530 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 >> v1:172.21.15.63:6789/0 conn(0x7f18ac0a4790 legacy=0x7f18ac13f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 >> v1:172.21.15.6:6789/0 conn(0x7f18ac0a08a0 legacy=0x7f18ac13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: _finish_hunting 0 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 monclient: found mon.noname-b 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ac0a53e0 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: _finish_auth 0 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: _check_auth_tickets 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18a8001620 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 5 monclient: authenticate success, global_id 5381 2024-09-16T05:37:56.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: dump: 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:56.204 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18a8001850 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 1 -- 172.21.15.63:0/668279067 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18a8001c60 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 4 set_mon_vals no callback set 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b13a4640 10 monclient: dump: 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:56.205 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient: shutdown 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- 172.21.15.63:0/668279067 >> v1:172.21.15.6:6790/0 conn(0x7f18ac0a3ca0 legacy=0x7f18ac0a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- 172.21.15.63:0/668279067 shutdown_connections 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- 172.21.15.63:0/668279067 >> 172.21.15.63:0/668279067 conn(0x7f18ac0a67c0 msgr2=0x7f18ac075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:56.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- 172.21.15.63:0/668279067 shutdown_connections 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 1 -- 172.21.15.63:0/668279067 wait complete. 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient: build_initial_monmap 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.194+0000 7f18b460a640 10 monclient: monmap: 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:56.207 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:56.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 1 Processor -- start 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 1 -- start start 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: init 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient(hunting): picked mon.noname-a con 0x7f18ac0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient(hunting): picked mon.noname-b con 0x7f18ac0a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient(hunting): picked mon.noname-c con 0x7f18ac0a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18ac127450 con 0x7f18ac0a08a0 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18ac0a8500 con 0x7f18ac0a3ca0 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18ac127730 con 0x7f18ac0a4790 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:56.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b1ba5640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f18ac0a3ca0 legacy=0x7f18ac1d01a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient(hunting): authenticate will time out at 1132.976615s 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b23a6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f18ac0a08a0 0x7f18ac136d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41008/0 (socket says 172.21.15.63:41008) 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b23a6640 1 -- 172.21.15.63:0/436152353 learned_addr learned my addr 172.21.15.63:0/436152353 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4061282933 0 0) 0x7f18ac127730 con 0x7f18ac0a4790 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): my global_id is 5384 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1890003530 con 0x7f18ac0a4790 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2511014449 0 0) 0x7f18ac127450 con 0x7f18ac0a08a0 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:56.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): my global_id is 5397 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18ac127730 con 0x7f18ac0a08a0 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3925265711 0 0) 0x7f1890003530 con 0x7f18ac0a4790 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18ac127450 con 0x7f18ac0a4790 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18a4004340 con 0x7f18ac0a4790 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3959827178 0 0) 0x7f18ac127730 con 0x7f18ac0a08a0 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1890003530 con 0x7f18ac0a08a0 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18a8001be0 con 0x7f18ac0a08a0 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient(hunting): dump: 2024-09-16T05:37:56.215 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1876712106 0 0) 0x7f18ac127450 con 0x7f18ac0a4790 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 >> v1:172.21.15.63:6789/0 conn(0x7f18ac0a3ca0 legacy=0x7f18ac1d01a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:56.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 >> v1:172.21.15.6:6789/0 conn(0x7f18ac0a08a0 legacy=0x7f18ac136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 monclient: found mon.c 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ac0a7fb0 con 0x7f18ac0a4790 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 5 monclient: authenticate success, global_id 5384 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: _renew_subs 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18ac0a0500 con 0x7f18ac0a4790 2024-09-16T05:37:56.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: _renew_subs 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f18ac137450 con 0x7f18ac0a4790 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18a40045a0 con 0x7f18ac0a4790 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18a40048f0 con 0x7f18ac0a4790 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b0ba3640 4 set_mon_vals no callback set 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 10 monclient: dump: 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:56.218 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1894005090 con 0x7f18ac0a4790 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f18a4004b10 con 0x7f18ac0a4790 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.198+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(107..107 src has 1..107) ==== 5243+0+0 (unknown 3120333743 0 0) 0x7f18a4044960 con 0x7f18ac0a4790 2024-09-16T05:37:56.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.202+0000 7f18b23a6640 10 monclient: get_auth_request con 0x7f18900369d0 auth_method 0 2024-09-16T05:37:56.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.214+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f18a4015560 con 0x7f18ac0a4790 2024-09-16T05:37:56.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.214+0000 7f18a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:56.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.214+0000 7f18a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:56.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.382+0000 7f18b460a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T05:37:56.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.382+0000 7f18b460a640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T05:37:56.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.382+0000 7f18b460a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:56.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:56.382+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) -- 0x7f1894005380 con 0x7f18ac0a4790 2024-09-16T05:37:57.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.330+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 removed item id 11 name 'osd.11' from crush map v108) ==== 125+0+0 (unknown 3433084678 0 0) 0x7f18a4014e60 con 0x7f18ac0a4790 2024-09-16T05:37:57.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.330+0000 7f18a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T05:37:57.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.330+0000 7f18a2ffd640 10 monclient: _finish_command 2 = system:0 removed item id 11 name 'osd.11' from crush map 2024-09-16T05:37:57.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.438+0000 7f18b460a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T05:37:57.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.438+0000 7f18b460a640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T05:37:57.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.438+0000 7f18b460a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:57.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.438+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) -- 0x7f18940010a0 con 0x7f18ac0a4790 2024-09-16T05:37:57.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.438+0000 7f18a2ffd640 1 -- 172.21.15.63:0/436152353 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 device 'osd.11' does not appear in the crush map v108) ==== 126+0+0 (unknown 4129756445 0 0) 0x7f18a40191b0 con 0x7f18ac0a4790 2024-09-16T05:37:57.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.438+0000 7f18a2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T05:37:57.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.438+0000 7f18a2ffd640 10 monclient: _finish_command 3 = system:0 device 'osd.11' does not appear in the crush map 2024-09-16T05:37:57.444 INFO:tasks.workunit.client.0.smithi063.stderr:device 'osd.11' does not appear in the crush map 2024-09-16T05:37:57.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.442+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 >> v1:172.21.15.63:6812/3643207211 conn(0x7f18900369d0 legacy=0x7f1890038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:57.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.442+0000 7f18b460a640 10 monclient: shutdown 2024-09-16T05:37:57.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.442+0000 7f18b460a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:57.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.442+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 >> v1:172.21.15.6:6790/0 conn(0x7f18ac0a4790 legacy=0x7f18ac1d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:57.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.442+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 shutdown_connections 2024-09-16T05:37:57.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.442+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 >> 172.21.15.63:0/436152353 conn(0x7f18ac0a67c0 msgr2=0x7f18ac0a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:57.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.442+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 shutdown_connections 2024-09-16T05:37:57.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.442+0000 7f18b460a640 1 -- 172.21.15.63:0/436152353 wait complete. 2024-09-16T05:37:57.460 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd rm osd.11 2024-09-16T05:37:57.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient: get_monmap_and_config 2024-09-16T05:37:57.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient: build_initial_monmap 2024-09-16T05:37:57.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient: monmap: 2024-09-16T05:37:57.578 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:57.578 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:57.578 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:57.577849+0000 2024-09-16T05:37:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:57.577849+0000 2024-09-16T05:37:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:57.579 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:57.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 1 Processor -- start 2024-09-16T05:37:57.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 1 -- start start 2024-09-16T05:37:57.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient: init 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient(hunting): picked mon.noname-a con 0x7f40880a1030 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient(hunting): picked mon.noname-b con 0x7f408809deb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient(hunting): picked mon.noname-c con 0x7f40880a1b20 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408809d1f0 con 0x7f40880a1030 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40880a2c10 con 0x7f40880a1b20 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408809f960 con 0x7f408809deb0 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f408dca0640 10 monclient(hunting): authenticate will time out at 1134.355601s 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.574+0000 7f4087fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f40880a1b20 legacy=0x7f408813d300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:57.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f4086ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f408809deb0 0x7f4088139bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35610/0 (socket says 172.21.15.63:35610) 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40877fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f40880a1030 0x7f40880a1410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41022/0 (socket says 172.21.15.63:41022) 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f4086ffd640 1 -- 172.21.15.63:0/1252597988 learned_addr learned my addr 172.21.15.63:0/1252597988 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3173241913 0 0) 0x7f408809f960 con 0x7f408809deb0 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient(hunting): my global_id is 5387 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4070003530 con 0x7f408809deb0 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316336761 0 0) 0x7f408809d1f0 con 0x7f40880a1030 2024-09-16T05:37:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient(hunting): _negotiate 2024-09-16T05:37:57.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:57.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:57.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient(hunting): my global_id is 5400 2024-09-16T05:37:57.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f408809f960 con 0x7f40880a1030 2024-09-16T05:37:57.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2472249306 0 0) 0x7f4070003530 con 0x7f408809deb0 2024-09-16T05:37:57.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 >> v1:172.21.15.63:6789/0 conn(0x7f40880a1b20 legacy=0x7f408813d300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:57.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 >> v1:172.21.15.6:6789/0 conn(0x7f40880a1030 legacy=0x7f40880a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:57.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: _finish_hunting 0 2024-09-16T05:37:57.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 monclient: found mon.noname-b 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408809d4d0 con 0x7f408809deb0 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: _finish_auth 0 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: _check_auth_tickets 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f407c001620 con 0x7f408809deb0 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 5 monclient: authenticate success, global_id 5387 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: dump: 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:57.590 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f407c0018c0 con 0x7f408809deb0 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 4 set_mon_vals no callback set 2024-09-16T05:37:57.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 1 -- 172.21.15.63:0/1252597988 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f407c001cd0 con 0x7f408809deb0 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40867fc640 10 monclient: dump: 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:57.592 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient: shutdown 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- 172.21.15.63:0/1252597988 >> v1:172.21.15.6:6790/0 conn(0x7f408809deb0 legacy=0x7f4088139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- 172.21.15.63:0/1252597988 shutdown_connections 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- 172.21.15.63:0/1252597988 >> 172.21.15.63:0/1252597988 conn(0x7f40880a5c20 msgr2=0x7f40880a2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- 172.21.15.63:0/1252597988 shutdown_connections 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- 172.21.15.63:0/1252597988 wait complete. 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient: build_initial_monmap 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient: monmap: 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:57.593 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:57.594 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:57.594 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:57.594 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:57.594 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:57.594 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:57.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 Processor -- start 2024-09-16T05:37:57.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- start start 2024-09-16T05:37:57.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient: init 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient(hunting): picked mon.noname-c con 0x7f408809deb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient(hunting): picked mon.noname-b con 0x7f40880a1030 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient(hunting): picked mon.noname-a con 0x7f40880a1b20 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408812f280 con 0x7f40880a1b20 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408812f560 con 0x7f40880a1030 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4088072de0 con 0x7f408809deb0 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f4086ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f40880a1030 legacy=0x7f4088134d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f408dca0640 10 monclient(hunting): authenticate will time out at 1134.359692s 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.578+0000 7f40877fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f408809deb0 0x7f40881cb290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35620/0 (socket says 172.21.15.63:35620) 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f40877fe640 1 -- 172.21.15.63:0/3752807438 learned_addr learned my addr 172.21.15.63:0/3752807438 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:57.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124494844 0 0) 0x7f4088072de0 con 0x7f408809deb0 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): my global_id is 5390 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4074003530 con 0x7f408809deb0 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786777960 0 0) 0x7f408812f280 con 0x7f40880a1b20 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): my global_id is 5403 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4088072de0 con 0x7f40880a1b20 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2777755446 0 0) 0x7f4074003530 con 0x7f408809deb0 2024-09-16T05:37:57.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f408812f280 con 0x7f408809deb0 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4078001380 con 0x7f408809deb0 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): dump: 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:57.597 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1475537508 0 0) 0x7f4088072de0 con 0x7f40880a1b20 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4074003530 con 0x7f40880a1b20 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4080004340 con 0x7f40880a1b20 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient(hunting): dump: 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3786644370 0 0) 0x7f408812f280 con 0x7f408809deb0 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 >> v1:172.21.15.63:6789/0 conn(0x7f40880a1030 legacy=0x7f4088134d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 >> v1:172.21.15.6:6789/0 conn(0x7f40880a1b20 legacy=0x7f40881d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 monclient: found mon.c 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:57.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408809c810 con 0x7f408809deb0 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient: _finish_auth 0 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 5 monclient: authenticate success, global_id 5390 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 10 monclient: _renew_subs 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f408812d4e0 con 0x7f408809deb0 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 10 monclient: _renew_subs 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f40881354a0 con 0x7f408809deb0 2024-09-16T05:37:57.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4078001630 con 0x7f408809deb0 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f4085ffb640 4 set_mon_vals no callback set 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4078004c40 con 0x7f408809deb0 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 10 monclient: dump: 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:57.601 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4068005090 con 0x7f408809deb0 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f407800f9e0 con 0x7f408809deb0 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(108..108 src has 1..108) ==== 5192+0+0 (unknown 4043511758 0 0) 0x7f4078044850 con 0x7f408809deb0 2024-09-16T05:37:57.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.582+0000 7f4086ffd640 10 monclient: get_auth_request con 0x7f4074036a20 auth_method 0 2024-09-16T05:37:57.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.594+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4078015480 con 0x7f408809deb0 2024-09-16T05:37:57.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.594+0000 7f406ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:57.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.594+0000 7f406ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:57.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.770+0000 7f408dca0640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T05:37:57.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.770+0000 7f408dca0640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T05:37:57.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.770+0000 7f408dca0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:57.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:57.770+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) -- 0x7f4068005380 con 0x7f408809deb0 2024-09-16T05:37:57.774 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:37:57.769+0000 7f3419a35640 -1 mon.a@0(leader).osd e108 definitely_dead 0 2024-09-16T05:37:58.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.334+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 removed osd.11 v109) ==== 87+0+0 (unknown 418860530 0 0) 0x7f4078014d80 con 0x7f408809deb0 2024-09-16T05:37:58.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.334+0000 7f406ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T05:37:58.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.334+0000 7f406ffff640 10 monclient: _finish_command 2 = system:0 removed osd.11 2024-09-16T05:37:58.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.446+0000 7f408dca0640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T05:37:58.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.446+0000 7f408dca0640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T05:37:58.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.446+0000 7f408dca0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:37:58.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.446+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) -- 0x7f4068005ac0 con 0x7f408809deb0 2024-09-16T05:37:58.452 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:37:58.449+0000 7f3419a35640 -1 mon.a@0(leader).osd e109 definitely_dead 0 2024-09-16T05:37:58.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f406ffff640 1 -- 172.21.15.63:0/3752807438 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 osd.11 does not exist. v109) ==== 96+0+0 (unknown 3013441707 0 0) 0x7f40780190d0 con 0x7f408809deb0 2024-09-16T05:37:58.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f406ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T05:37:58.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f406ffff640 10 monclient: _finish_command 3 = system:0 osd.11 does not exist. 2024-09-16T05:37:58.453 INFO:tasks.workunit.client.0.smithi063.stderr:osd.11 does not exist. 2024-09-16T05:37:58.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4074036a20 legacy=0x7f4074038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:58.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f408dca0640 10 monclient: shutdown 2024-09-16T05:37:58.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f408dca0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:58.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 >> v1:172.21.15.6:6790/0 conn(0x7f408809deb0 legacy=0x7f40881cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:58.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 shutdown_connections 2024-09-16T05:37:58.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 >> 172.21.15.63:0/3752807438 conn(0x7f40880a5c20 msgr2=0x7f40880a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:58.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 shutdown_connections 2024-09-16T05:37:58.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.450+0000 7f408dca0640 1 -- 172.21.15.63:0/3752807438 wait complete. 2024-09-16T05:37:58.472 INFO:tasks.workunit.client.0.smithi063.stderr:++ ceph osd create 2024-09-16T05:37:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.586+0000 7f25cccd7640 10 monclient: get_monmap_and_config 2024-09-16T05:37:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.586+0000 7f25cccd7640 10 monclient: build_initial_monmap 2024-09-16T05:37:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.586+0000 7f25cccd7640 10 monclient: monmap: 2024-09-16T05:37:58.591 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:37:58.590976+0000 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:37:58.590976+0000 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:37:58.592 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:58.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.586+0000 7f25cccd7640 1 Processor -- start 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.586+0000 7f25cccd7640 1 -- start start 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.586+0000 7f25cccd7640 10 monclient: init 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient(hunting): picked mon.noname-c con 0x7f25c809acb0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient(hunting): picked mon.noname-a con 0x7f25c80a13f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient(hunting): picked mon.noname-b con 0x7f25c809b7a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c80a0fd0 con 0x7f25c80a13f0 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c809f110 con 0x7f25c809acb0 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c809ec90 con 0x7f25c809b7a0 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient(hunting): authenticate will time out at 1135.368248s 2024-09-16T05:37:58.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f25c809acb0 legacy=0x7f25c809b090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c5d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f25c80a13f0 0x7f25c8139ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41040/0 (socket says 172.21.15.63:41040) 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c5d9b640 1 -- 172.21.15.63:0/3881618978 learned_addr learned my addr 172.21.15.63:0/3881618978 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 73750949 0 0) 0x7f25c80a0fd0 con 0x7f25c80a13f0 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient(hunting): my global_id is 5406 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25b4003530 con 0x7f25c80a13f0 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2442243273 0 0) 0x7f25c809ec90 con 0x7f25c809b7a0 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient(hunting): _negotiate 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:58.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient(hunting): my global_id is 5393 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c80a0fd0 con 0x7f25c809b7a0 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2922601010 0 0) 0x7f25b4003530 con 0x7f25c80a13f0 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 >> v1:172.21.15.6:6790/0 conn(0x7f25c809b7a0 legacy=0x7f25c813d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 >> v1:172.21.15.63:6789/0 conn(0x7f25c809acb0 legacy=0x7f25c809b090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: _finish_hunting 0 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 monclient: found mon.noname-a 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c80a0730 con 0x7f25c80a13f0 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: _finish_auth 0 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: _check_auth_tickets 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f25b80019a0 con 0x7f25c80a13f0 2024-09-16T05:37:58.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: dump: 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 5 monclient: authenticate success, global_id 5406 2024-09-16T05:37:58.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25b8001c50 con 0x7f25c80a13f0 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 1 -- 172.21.15.63:0/3881618978 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f25b8002020 con 0x7f25c80a13f0 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25c559a640 10 monclient: dump: 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:58.599 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:58.600 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:58.600 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:58.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient: get_monmap_and_config success 2024-09-16T05:37:58.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 4 set_mon_vals no callback set 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient: shutdown 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- 172.21.15.63:0/3881618978 >> v1:172.21.15.6:6789/0 conn(0x7f25c80a13f0 legacy=0x7f25c8139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- 172.21.15.63:0/3881618978 shutdown_connections 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- 172.21.15.63:0/3881618978 >> 172.21.15.63:0/3881618978 conn(0x7f25c80a1b60 msgr2=0x7f25c80a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- 172.21.15.63:0/3881618978 shutdown_connections 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- 172.21.15.63:0/3881618978 wait complete. 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient: build_initial_monmap 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient: monmap: 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:37:58.605 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 Processor -- start 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 1 -- start start 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.590+0000 7f25cccd7640 10 monclient: init 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient: _reopen_session rank -1 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:37:58.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient(hunting): picked mon.noname-c con 0x7f25c809acb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient(hunting): picked mon.noname-b con 0x7f25c809b7a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient(hunting): picked mon.noname-a con 0x7f25c80a13f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c8126630 con 0x7f25c80a13f0 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c809d250 con 0x7f25c809b7a0 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c8125180 con 0x7f25c809acb0 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient(hunting): _renew_subs 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient(hunting): authenticate will time out at 1135.372612s 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25c5d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f25c809b7a0 legacy=0x7f25c809cb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25c6d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f25c80a13f0 0x7f25c81d3910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41054/0 (socket says 172.21.15.63:41054) 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25c659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f25c809acb0 0x7f25c809c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35638/0 (socket says 172.21.15.63:35638) 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25c6d9d640 1 -- 172.21.15.63:0/3050200825 learned_addr learned my addr 172.21.15.63:0/3050200825 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633528316 0 0) 0x7f25c8126630 con 0x7f25c80a13f0 2024-09-16T05:37:58.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): my global_id is 5409 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25a8003530 con 0x7f25c80a13f0 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1534201636 0 0) 0x7f25c8125180 con 0x7f25c809acb0 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): _negotiate 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): my global_id is 5396 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c8126630 con 0x7f25c809acb0 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3140390582 0 0) 0x7f25a8003530 con 0x7f25c80a13f0 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25c8125180 con 0x7f25c80a13f0 2024-09-16T05:37:58.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f25c00018e0 con 0x7f25c80a13f0 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): dump: 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:58.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 503152065 0 0) 0x7f25c8126630 con 0x7f25c809acb0 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25a8003530 con 0x7f25c809acb0 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f25b00040d0 con 0x7f25c809acb0 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient(hunting): dump: 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:58.610 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 692922447 0 0) 0x7f25c8125180 con 0x7f25c80a13f0 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 >> v1:172.21.15.6:6790/0 conn(0x7f25c809acb0 legacy=0x7f25c809c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 >> v1:172.21.15.63:6789/0 conn(0x7f25c809b7a0 legacy=0x7f25c809cb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient: _finish_hunting 0 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 monclient: found mon.a 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c806dd60 con 0x7f25c80a13f0 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient: _finish_auth 0 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient: _check_auth_tickets 2024-09-16T05:37:58.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 5 monclient: authenticate success, global_id 5409 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient: _renew_subs 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25c809c0f0 con 0x7f25c80a13f0 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient: _renew_subs 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f25c8124e20 con 0x7f25c80a13f0 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25c0003620 con 0x7f25c80a13f0 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f25c0003970 con 0x7f25c80a13f0 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25c4d99640 4 set_mon_vals no callback set 2024-09-16T05:37:58.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 10 monclient: dump: 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:58.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:58.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25ac005090 con 0x7f25c80a13f0 2024-09-16T05:37:58.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f25c0003b10 con 0x7f25c80a13f0 2024-09-16T05:37:58.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(109..109 src has 1..109) ==== 5240+0+0 (unknown 1104133391 0 0) 0x7f25c0044780 con 0x7f25c80a13f0 2024-09-16T05:37:58.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.594+0000 7f25c659c640 10 monclient: get_auth_request con 0x7f25a8036a70 auth_method 0 2024-09-16T05:37:58.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.610+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f25c0015320 con 0x7f25c80a13f0 2024-09-16T05:37:58.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.610+0000 7f25beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:37:58.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.610+0000 7f25beffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:37:58.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.782+0000 7f25cccd7640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:37:58.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.782+0000 7f25cccd7640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T05:37:58.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.782+0000 7f25cccd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:58.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:58.782+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f25ac005380 con 0x7f25c80a13f0 2024-09-16T05:37:59.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:59.338+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v110) ==== 58+0+3 (unknown 252105186 0 212448812) 0x7f25c0018fd0 con 0x7f25c80a13f0 2024-09-16T05:37:59.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:59.338+0000 7f25beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T05:37:59.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:59.338+0000 7f25beffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:37:59.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:59.450+0000 7f25cccd7640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:37:59.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:59.450+0000 7f25cccd7640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T05:37:59.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:59.450+0000 7f25cccd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:37:59.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:37:59.450+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f25ac005ac0 con 0x7f25c80a13f0 2024-09-16T05:38:00.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25beffd640 1 -- 172.21.15.63:0/3050200825 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v111) ==== 58+0+3 (unknown 3128102678 0 944643765) 0x7f25c001d890 con 0x7f25c80a13f0 2024-09-16T05:38:00.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25beffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T05:38:00.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25beffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:00.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 >> v1:172.21.15.63:6812/3643207211 conn(0x7f25a8036a70 legacy=0x7f25a8038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:00.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25cccd7640 10 monclient: shutdown 2024-09-16T05:38:00.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25cccd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:00.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 >> v1:172.21.15.6:6789/0 conn(0x7f25c80a13f0 legacy=0x7f25c81d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:00.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 shutdown_connections 2024-09-16T05:38:00.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 >> 172.21.15.63:0/3050200825 conn(0x7f25c80a1b60 msgr2=0x7f25c80a3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:00.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 shutdown_connections 2024-09-16T05:38:00.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.350+0000 7f25cccd7640 1 -- 172.21.15.63:0/3050200825 wait complete. 2024-09-16T05:38:00.376 INFO:tasks.workunit.client.0.smithi063.stderr:+ o4=12 2024-09-16T05:38:00.377 INFO:tasks.workunit.client.0.smithi063.stderr:++ ceph osd create 2024-09-16T05:38:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.490+0000 7f72cf429640 10 monclient: get_monmap_and_config 2024-09-16T05:38:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.490+0000 7f72cf429640 10 monclient: build_initial_monmap 2024-09-16T05:38:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.490+0000 7f72cf429640 10 monclient: monmap: 2024-09-16T05:38:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:00.495869+0000 2024-09-16T05:38:00.497 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:00.495869+0000 2024-09-16T05:38:00.497 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:00.497 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:00.497 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:00.497 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:00.497 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:00.497 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:00.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 Processor -- start 2024-09-16T05:38:00.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- start start 2024-09-16T05:38:00.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient: init 2024-09-16T05:38:00.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:00.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:00.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient(hunting): picked mon.noname-b con 0x7f72c809ee70 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient(hunting): picked mon.noname-c con 0x7f72c80a45c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient(hunting): picked mon.noname-a con 0x7f72c809f960 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c80a4ee0 con 0x7f72c809f960 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c81413d0 con 0x7f72c80a45c0 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c81425b0 con 0x7f72c809ee70 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient(hunting): authenticate will time out at 1137.273009s 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cc9c4640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f72c80a45c0 legacy=0x7f72c809f250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cd1c5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f72c809ee70 0x7f72c80a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35650/0 (socket says 172.21.15.63:35650) 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cd1c5640 1 -- 172.21.15.63:0/849453350 learned_addr learned my addr 172.21.15.63:0/849453350 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:00.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847100771 0 0) 0x7f72c81425b0 con 0x7f72c809ee70 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient(hunting): my global_id is 5399 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b4003530 con 0x7f72c809ee70 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22560829 0 0) 0x7f72c80a4ee0 con 0x7f72c809f960 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient(hunting): my global_id is 5412 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72c81425b0 con 0x7f72c809f960 2024-09-16T05:38:00.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3207578303 0 0) 0x7f72b4003530 con 0x7f72c809ee70 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 >> v1:172.21.15.63:6789/0 conn(0x7f72c80a45c0 legacy=0x7f72c809f250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 >> v1:172.21.15.6:6789/0 conn(0x7f72c809f960 legacy=0x7f72c813eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 monclient: found mon.noname-b 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c8143790 con 0x7f72c809ee70 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: _finish_auth 0 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72c4001620 con 0x7f72c809ee70 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:00.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: dump: 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 5 monclient: authenticate success, global_id 5399 2024-09-16T05:38:00.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72c40018d0 con 0x7f72c809ee70 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 1 -- 172.21.15.63:0/849453350 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72c4001ca0 con 0x7f72c809ee70 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72bffff640 10 monclient: dump: 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:00.505 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 4 set_mon_vals no callback set 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient: shutdown 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- 172.21.15.63:0/849453350 >> v1:172.21.15.6:6790/0 conn(0x7f72c809ee70 legacy=0x7f72c80a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- 172.21.15.63:0/849453350 shutdown_connections 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- 172.21.15.63:0/849453350 >> 172.21.15.63:0/849453350 conn(0x7f72c80a5d10 msgr2=0x7f72c80a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- 172.21.15.63:0/849453350 shutdown_connections 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 1 -- 172.21.15.63:0/849453350 wait complete. 2024-09-16T05:38:00.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient: build_initial_monmap 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.494+0000 7f72cf429640 10 monclient: monmap: 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:00.507 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:00.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 1 Processor -- start 2024-09-16T05:38:00.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 1 -- start start 2024-09-16T05:38:00.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: init 2024-09-16T05:38:00.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:00.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:00.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient(hunting): picked mon.noname-a con 0x7f72c809ee70 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:00.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient(hunting): picked mon.noname-b con 0x7f72c809f960 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient(hunting): picked mon.noname-c con 0x7f72c80a45c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c81389b0 con 0x7f72c809ee70 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c8138c90 con 0x7f72c809f960 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c81420f0 con 0x7f72c80a45c0 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient(hunting): authenticate will time out at 1137.276758s 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cc9c4640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f72c809f960 legacy=0x7f72c8070600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cd9c6640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f72c80a45c0 0x7f72c81d36f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35658/0 (socket says 172.21.15.63:35658) 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cd9c6640 1 -- 172.21.15.63:0/1466233972 learned_addr learned my addr 172.21.15.63:0/1466233972 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2312139454 0 0) 0x7f72c81420f0 con 0x7f72c80a45c0 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:00.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): my global_id is 5402 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72ac003560 con 0x7f72c80a45c0 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 396617936 0 0) 0x7f72c81389b0 con 0x7f72c809ee70 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): my global_id is 5415 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72c81420f0 con 0x7f72c809ee70 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2647783752 0 0) 0x7f72ac003560 con 0x7f72c80a45c0 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72c81389b0 con 0x7f72c80a45c0 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72c0001c10 con 0x7f72c80a45c0 2024-09-16T05:38:00.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): dump: 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:00.514 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 880274144 0 0) 0x7f72c81420f0 con 0x7f72c809ee70 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72ac003560 con 0x7f72c809ee70 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72c4001c20 con 0x7f72c809ee70 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient(hunting): dump: 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:00.515 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1266962425 0 0) 0x7f72c81389b0 con 0x7f72c80a45c0 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 >> v1:172.21.15.63:6789/0 conn(0x7f72c809f960 legacy=0x7f72c8070600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 >> v1:172.21.15.6:6789/0 conn(0x7f72c809ee70 legacy=0x7f72c806fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 monclient: found mon.c 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c80a0500 con 0x7f72c80a45c0 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:00.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 5 monclient: authenticate success, global_id 5402 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: _renew_subs 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72c8072aa0 con 0x7f72c80a45c0 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: _renew_subs 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f72c8071d50 con 0x7f72c80a45c0 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72c00039a0 con 0x7f72c80a45c0 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f72c0003cf0 con 0x7f72c80a45c0 2024-09-16T05:38:00.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bf7fe640 4 set_mon_vals no callback set 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 10 monclient: dump: 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:00.518 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a8005090 con 0x7f72c80a45c0 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f72c0003f10 con 0x7f72c80a45c0 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.498+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(111..111 src has 1..111) ==== 5359+0+0 (unknown 4102425176 0 0) 0x7f72c0044c30 con 0x7f72c80a45c0 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.502+0000 7f72cd1c5640 10 monclient: get_auth_request con 0x7f72ac036a30 auth_method 0 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.514+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f72c0015820 con 0x7f72c80a45c0 2024-09-16T05:38:00.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.514+0000 7f72bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:00.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.514+0000 7f72bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:00.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.686+0000 7f72cf429640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:38:00.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.686+0000 7f72cf429640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T05:38:00.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.686+0000 7f72cf429640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:00.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:00.686+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f72a8005380 con 0x7f72c80a45c0 2024-09-16T05:38:01.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:01.370+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v112) ==== 58+0+3 (unknown 1912018311 0 736921282) 0x7f72c0015120 con 0x7f72c80a45c0 2024-09-16T05:38:01.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:01.370+0000 7f72bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T05:38:01.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:01.370+0000 7f72bdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:01.478+0000 7f72cf429640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T05:38:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:01.478+0000 7f72cf429640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T05:38:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:01.478+0000 7f72cf429640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:01.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:01.478+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f72a8000bd0 con 0x7f72c80a45c0 2024-09-16T05:38:02.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72bdffb640 1 -- 172.21.15.63:0/1466233972 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v113) ==== 58+0+3 (unknown 3296906099 0 1367428999) 0x7f72c0013480 con 0x7f72c80a45c0 2024-09-16T05:38:02.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72bdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T05:38:02.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72bdffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 >> v1:172.21.15.63:6812/3643207211 conn(0x7f72ac036a30 legacy=0x7f72ac038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72cf429640 10 monclient: shutdown 2024-09-16T05:38:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72cf429640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 >> v1:172.21.15.6:6790/0 conn(0x7f72c80a45c0 legacy=0x7f72c81d36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 shutdown_connections 2024-09-16T05:38:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 >> 172.21.15.63:0/1466233972 conn(0x7f72c80a5d10 msgr2=0x7f72c80a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 shutdown_connections 2024-09-16T05:38:02.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.374+0000 7f72cf429640 1 -- 172.21.15.63:0/1466233972 wait complete. 2024-09-16T05:38:02.397 INFO:tasks.workunit.client.0.smithi063.stderr:+ o5=14 2024-09-16T05:38:02.397 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add 12 123 root=default host=foobaz 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.522+0000 7ff498d55640 10 monclient: get_monmap_and_config 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.522+0000 7ff498d55640 10 monclient: build_initial_monmap 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.522+0000 7ff498d55640 10 monclient: monmap: 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:02.526547+0000 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:02.526547+0000 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:02.527 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:02.528 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:02.528 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:02.528 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:02.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.522+0000 7ff498d55640 1 Processor -- start 2024-09-16T05:38:02.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.522+0000 7ff498d55640 1 -- start start 2024-09-16T05:38:02.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.522+0000 7ff498d55640 10 monclient: init 2024-09-16T05:38:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient(hunting): picked mon.noname-a con 0x7ff4940a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient(hunting): picked mon.noname-c con 0x7ff4940a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient(hunting): picked mon.noname-b con 0x7ff4940a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4940a53b0 con 0x7ff4940a3b10 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4940a0290 con 0x7ff4940a07a0 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4940a0dc0 con 0x7ff4940a4600 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient(hunting): authenticate will time out at 1139.304354s 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff491d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff4940a07a0 legacy=0x7ff49413be20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff492d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff4940a4600 0x7ff49413f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35670/0 (socket says 172.21.15.63:35670) 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff492d9d640 1 -- 172.21.15.63:0/835654703 learned_addr learned my addr 172.21.15.63:0/835654703 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49259c640 1 --1- 172.21.15.63:0/835654703 >> v1:172.21.15.6:6789/0 conn(0x7ff4940a3b10 0x7ff4940a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41080/0 (socket says 172.21.15.63:41080) 2024-09-16T05:38:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511972228 0 0) 0x7ff4940a0dc0 con 0x7ff4940a4600 2024-09-16T05:38:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient(hunting): my global_id is 5405 2024-09-16T05:38:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff47c003530 con 0x7ff4940a4600 2024-09-16T05:38:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1027775380 0 0) 0x7ff4940a53b0 con 0x7ff4940a3b10 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient(hunting): my global_id is 5418 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4940a0dc0 con 0x7ff4940a3b10 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 507957337 0 0) 0x7ff47c003530 con 0x7ff4940a4600 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 >> v1:172.21.15.63:6789/0 conn(0x7ff4940a07a0 legacy=0x7ff49413be20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 >> v1:172.21.15.6:6789/0 conn(0x7ff4940a3b10 legacy=0x7ff4940a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 monclient: found mon.noname-b 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4940a10a0 con 0x7ff4940a4600 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: _finish_auth 0 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 5 monclient: authenticate success, global_id 5405 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff48c0019a0 con 0x7ff4940a4600 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: dump: 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff48c001c00 con 0x7ff4940a4600 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 1 -- 172.21.15.63:0/835654703 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff48c001f90 con 0x7ff4940a4600 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:02.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff49159a640 10 monclient: dump: 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 4 set_mon_vals no callback set 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient: shutdown 2024-09-16T05:38:02.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- 172.21.15.63:0/835654703 >> v1:172.21.15.6:6790/0 conn(0x7ff4940a4600 legacy=0x7ff49413f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- 172.21.15.63:0/835654703 shutdown_connections 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- 172.21.15.63:0/835654703 >> 172.21.15.63:0/835654703 conn(0x7ff494075470 msgr2=0x7ff494075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- 172.21.15.63:0/835654703 shutdown_connections 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- 172.21.15.63:0/835654703 wait complete. 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient: build_initial_monmap 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient: monmap: 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:02.538 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 Processor -- start 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 1 -- start start 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.526+0000 7ff498d55640 10 monclient: init 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:02.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient(hunting): picked mon.noname-a con 0x7ff4940a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient(hunting): picked mon.noname-c con 0x7ff4940a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient(hunting): picked mon.noname-b con 0x7ff4940a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4940a0500 con 0x7ff4940a07a0 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff494125ad0 con 0x7ff4940a4600 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff494127920 con 0x7ff4940a3b10 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient(hunting): authenticate will time out at 1139.308290s 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff492d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff4940a4600 legacy=0x7ff4941d3ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:02.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff491d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff4940a3b10 0x7ff4941d0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:35684/0 (socket says 172.21.15.63:35684) 2024-09-16T05:38:02.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff49259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff4940a07a0 0x7ff494136f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41096/0 (socket says 172.21.15.63:41096) 2024-09-16T05:38:02.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff491d9b640 1 -- 172.21.15.63:0/3372575552 learned_addr learned my addr 172.21.15.63:0/3372575552 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:02.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2105675639 0 0) 0x7ff494127920 con 0x7ff4940a3b10 2024-09-16T05:38:02.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:02.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:02.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:02.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): my global_id is 5408 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff474003530 con 0x7ff4940a3b10 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220383803 0 0) 0x7ff4840011e0 con 0x7ff4940a07a0 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): my global_id is 5421 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff494127920 con 0x7ff4940a07a0 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1696942598 0 0) 0x7ff474003530 con 0x7ff4940a3b10 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4840011e0 con 0x7ff4940a3b10 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1460890625 0 0) 0x7ff494127920 con 0x7ff4940a07a0 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff474003530 con 0x7ff4940a07a0 2024-09-16T05:38:02.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff488004150 con 0x7ff4940a3b10 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:02.546 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff4840014b0 con 0x7ff4940a07a0 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:02.547 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4158093216 0 0) 0x7ff4840011e0 con 0x7ff4940a3b10 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 >> v1:172.21.15.63:6789/0 conn(0x7ff4940a4600 legacy=0x7ff4941d3ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 >> v1:172.21.15.6:6789/0 conn(0x7ff4940a07a0 legacy=0x7ff494136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 monclient: found mon.c 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4940a88a0 con 0x7ff4940a3b10 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:02.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 5 monclient: authenticate success, global_id 5408 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient: _renew_subs 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4940a8390 con 0x7ff4940a3b10 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient: _renew_subs 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff494127640 con 0x7ff4940a3b10 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4880047d0 con 0x7ff4940a3b10 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff488004b20 con 0x7ff4940a3b10 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:02.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 10 monclient: dump: 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff490d99640 4 set_mon_vals no callback set 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:02.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:02.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:02.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff478005090 con 0x7ff4940a3b10 2024-09-16T05:38:02.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff48800f8c0 con 0x7ff4940a3b10 2024-09-16T05:38:02.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(113..113 src has 1..113) ==== 5597+0+0 (unknown 477505959 0 0) 0x7ff488044910 con 0x7ff4940a3b10 2024-09-16T05:38:02.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.530+0000 7ff49259c640 10 monclient: get_auth_request con 0x7ff474036a50 auth_method 0 2024-09-16T05:38:02.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.546+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff488015410 con 0x7ff4940a3b10 2024-09-16T05:38:02.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.546+0000 7ff482ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:02.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.546+0000 7ff482ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:02.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.714+0000 7ff498d55640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:02.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.714+0000 7ff498d55640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:02.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.714+0000 7ff498d55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:02.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:02.714+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) -- 0x7ff478005380 con 0x7ff4940a3b10 2024-09-16T05:38:03.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.382+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map v114) ==== 220+0+0 (unknown 2006609178 0 0) 0x7ff488014d10 con 0x7ff4940a3b10 2024-09-16T05:38:03.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.382+0000 7ff482ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:03.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.382+0000 7ff482ffd640 10 monclient: _finish_command 2 = system:0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-16T05:38:03.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.486+0000 7ff498d55640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:03.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.486+0000 7ff498d55640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:03.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.486+0000 7ff498d55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:03.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.486+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) -- 0x7ff478005c50 con 0x7ff4940a3b10 2024-09-16T05:38:03.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff482ffd640 1 -- 172.21.15.63:0/3372575552 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change v114) ==== 218+0+0 (unknown 4101822027 0 0) 0x7ff488019060 con 0x7ff4940a3b10 2024-09-16T05:38:03.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff482ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:03.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff482ffd640 10 monclient: _finish_command 3 = system:0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T05:38:03.493 INFO:tasks.workunit.client.0.smithi063.stderr:set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T05:38:03.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff474036a50 legacy=0x7ff474038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:03.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff498d55640 10 monclient: shutdown 2024-09-16T05:38:03.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff498d55640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:03.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 >> v1:172.21.15.6:6790/0 conn(0x7ff4940a3b10 legacy=0x7ff4941d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:03.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 shutdown_connections 2024-09-16T05:38:03.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 >> 172.21.15.63:0/3372575552 conn(0x7ff494075470 msgr2=0x7ff49406ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:03.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 shutdown_connections 2024-09-16T05:38:03.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.490+0000 7ff498d55640 1 -- 172.21.15.63:0/3372575552 wait complete. 2024-09-16T05:38:03.510 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add 14 123 root=default host=foobaz 2024-09-16T05:38:03.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.630+0000 7fe34fbde640 10 monclient: get_monmap_and_config 2024-09-16T05:38:03.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.630+0000 7fe34fbde640 10 monclient: build_initial_monmap 2024-09-16T05:38:03.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.630+0000 7fe34fbde640 10 monclient: monmap: 2024-09-16T05:38:03.635 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:03.635 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:03.635 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:03.634688+0000 2024-09-16T05:38:03.636 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:03.634688+0000 2024-09-16T05:38:03.636 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:03.636 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:03.636 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:03.636 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:03.636 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:03.636 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.630+0000 7fe34fbde640 1 Processor -- start 2024-09-16T05:38:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.630+0000 7fe34fbde640 1 -- start start 2024-09-16T05:38:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.630+0000 7fe34fbde640 10 monclient: init 2024-09-16T05:38:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient(hunting): picked mon.noname-b con 0x7fe3480a22a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient(hunting): picked mon.noname-a con 0x7fe3480731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient(hunting): picked mon.noname-c con 0x7fe3481395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe34809c580 con 0x7fe3480731d0 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3480a1300 con 0x7fe3481395a0 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe34809d0f0 con 0x7fe3480a22a0 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient(hunting): authenticate will time out at 1140.412235s 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34e17b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe3481395a0 legacy=0x7fe34813b990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34d179640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe3480731d0 0x7fe34809fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41106/0 (socket says 172.21.15.63:41106) 2024-09-16T05:38:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34d179640 1 -- 172.21.15.63:0/1355414945 learned_addr learned my addr 172.21.15.63:0/1355414945 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:03.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724779869 0 0) 0x7fe34809c580 con 0x7fe3480731d0 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient(hunting): _negotiate 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient(hunting): my global_id is 5424 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe334003530 con 0x7fe3480731d0 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062343223 0 0) 0x7fe34809d0f0 con 0x7fe3480a22a0 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient(hunting): _negotiate 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient(hunting): my global_id is 5411 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe34809c580 con 0x7fe3480a22a0 2024-09-16T05:38:03.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3998900985 0 0) 0x7fe334003530 con 0x7fe3480731d0 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 >> v1:172.21.15.6:6790/0 conn(0x7fe3480a22a0 legacy=0x7fe34809f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 >> v1:172.21.15.63:6789/0 conn(0x7fe3481395a0 legacy=0x7fe34813b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: _finish_hunting 0 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 monclient: found mon.noname-a 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34809d3d0 con 0x7fe3480731d0 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: _finish_auth 0 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: _check_auth_tickets 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe338001620 con 0x7fe3480731d0 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:03.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: dump: 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 5 monclient: authenticate success, global_id 5424 2024-09-16T05:38:03.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe338001880 con 0x7fe3480731d0 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 1 -- 172.21.15.63:0/1355414945 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe338001c90 con 0x7fe3480731d0 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34c978640 10 monclient: dump: 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:03.644 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 4 set_mon_vals no callback set 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient: shutdown 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 1 -- 172.21.15.63:0/1355414945 >> v1:172.21.15.6:6789/0 conn(0x7fe3480731d0 legacy=0x7fe34809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 1 -- 172.21.15.63:0/1355414945 shutdown_connections 2024-09-16T05:38:03.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 1 -- 172.21.15.63:0/1355414945 >> 172.21.15.63:0/1355414945 conn(0x7fe3480a1e90 msgr2=0x7fe348075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 1 -- 172.21.15.63:0/1355414945 shutdown_connections 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 1 -- 172.21.15.63:0/1355414945 wait complete. 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient: build_initial_monmap 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.634+0000 7fe34fbde640 10 monclient: monmap: 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:03.646 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:03.647 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:03.647 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:03.647 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:03.647 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:03.647 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:03.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 1 Processor -- start 2024-09-16T05:38:03.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 1 -- start start 2024-09-16T05:38:03.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: init 2024-09-16T05:38:03.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:03.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:03.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient(hunting): picked mon.noname-b con 0x7fe3480731d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:03.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient(hunting): picked mon.noname-a con 0x7fe3480a22a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient(hunting): picked mon.noname-c con 0x7fe3481395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34d97a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe3480731d0 legacy=0x7fe348134a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe34811f130 con 0x7fe3480a22a0 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe34811f410 con 0x7fe3480731d0 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3480a4370 con 0x7fe3481395a0 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient(hunting): authenticate will time out at 1140.416960s 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34d179640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe3480a22a0 0x7fe3481cbdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:41114/0 (socket says 172.21.15.63:41114) 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34d179640 1 -- 172.21.15.63:0/422013992 learned_addr learned my addr 172.21.15.63:0/422013992 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:03.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1535992369 0 0) 0x7fe34811f130 con 0x7fe3480a22a0 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): my global_id is 5427 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe32c003530 con 0x7fe3480a22a0 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025872715 0 0) 0x7fe3480a4370 con 0x7fe3481395a0 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): my global_id is 5414 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe34811f130 con 0x7fe3481395a0 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1811686721 0 0) 0x7fe34811f130 con 0x7fe3481395a0 2024-09-16T05:38:03.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3480a4370 con 0x7fe3481395a0 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe340004340 con 0x7fe3481395a0 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:03.654 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 420184706 0 0) 0x7fe32c003530 con 0x7fe3480a22a0 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe34811f130 con 0x7fe3480a22a0 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe338001bd0 con 0x7fe3480a22a0 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:03.655 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1370483603 0 0) 0x7fe3480a4370 con 0x7fe3481395a0 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 >> v1:172.21.15.63:6789/0 conn(0x7fe3480731d0 legacy=0x7fe348134a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 >> v1:172.21.15.6:6789/0 conn(0x7fe3480a22a0 legacy=0x7fe3481cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 monclient: found mon.c 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:03.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3480a3e20 con 0x7fe3481395a0 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 5 monclient: authenticate success, global_id 5414 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: _renew_subs 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe34811dea0 con 0x7fe3481395a0 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: _renew_subs 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe34811da00 con 0x7fe3481395a0 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe3400045f0 con 0x7fe3481395a0 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:03.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe3400049c0 con 0x7fe3481395a0 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33ffff640 4 set_mon_vals no callback set 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe33e7fc640 10 monclient: dump: 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:03.658 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:03.659 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:03.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:03.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:03.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:03.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.638+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe330005090 con 0x7fe3481395a0 2024-09-16T05:38:03.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.642+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe340004be0 con 0x7fe3481395a0 2024-09-16T05:38:03.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.642+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(114..114 src has 1..114) ==== 5781+0+0 (unknown 1620876870 0 0) 0x7fe340044c40 con 0x7fe3481395a0 2024-09-16T05:38:03.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.642+0000 7fe34d97a640 10 monclient: get_auth_request con 0x7fe32c038be0 auth_method 0 2024-09-16T05:38:03.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.654+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe340015620 con 0x7fe3481395a0 2024-09-16T05:38:03.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.654+0000 7fe33e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:03.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.654+0000 7fe33e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:03.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.818+0000 7fe34fbde640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:03.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.818+0000 7fe34fbde640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:03.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.818+0000 7fe34fbde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:03.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:03.818+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) -- 0x7fe330005380 con 0x7fe3481395a0 2024-09-16T05:38:04.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.402+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map v115) ==== 220+0+0 (unknown 1838144071 0 0) 0x7fe340014f20 con 0x7fe3481395a0 2024-09-16T05:38:04.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.402+0000 7fe33e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:04.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.402+0000 7fe33e7fc640 10 monclient: _finish_command 2 = system:0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-16T05:38:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.506+0000 7fe34fbde640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.506+0000 7fe34fbde640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.506+0000 7fe34fbde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.506+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) -- 0x7fe330005c50 con 0x7fe3481395a0 2024-09-16T05:38:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe33e7fc640 1 -- 172.21.15.63:0/422013992 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change v115) ==== 218+0+0 (unknown 1139850629 0 0) 0x7fe340019270 con 0x7fe3481395a0 2024-09-16T05:38:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe33e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T05:38:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe33e7fc640 10 monclient: _finish_command 3 = system:0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T05:38:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T05:38:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe32c038be0 legacy=0x7fe32c03b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe34fbde640 10 monclient: shutdown 2024-09-16T05:38:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe34fbde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 >> v1:172.21.15.6:6790/0 conn(0x7fe3481395a0 legacy=0x7fe3481cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 shutdown_connections 2024-09-16T05:38:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 >> 172.21.15.63:0/422013992 conn(0x7fe3480a1e90 msgr2=0x7fe3481377b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 shutdown_connections 2024-09-16T05:38:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.510+0000 7fe34fbde640 1 -- 172.21.15.63:0/422013992 wait complete. 2024-09-16T05:38:04.531 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:38:04.531 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep osd.12 2024-09-16T05:38:04.532 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 123 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient: get_monmap_and_config 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient: build_initial_monmap 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient: monmap: 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:04.650596+0000 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:04.650596+0000 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:04.651 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:04.652 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:04.652 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:04.652 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:04.652 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 1 Processor -- start 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 1 -- start start 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient: init 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient(hunting): picked mon.noname-c con 0x7f4acc09acb0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient(hunting): picked mon.noname-b con 0x7f4acc0a13f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient(hunting): picked mon.noname-a con 0x7f4acc09b7a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc0a0fd0 con 0x7f4acc09b7a0 2024-09-16T05:38:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc09f110 con 0x7f4acc09acb0 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc09ec90 con 0x7f4acc0a13f0 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad2bdc640 10 monclient(hunting): authenticate will time out at 1141.427556s 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4ad0978640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4acc09acb0 legacy=0x7f4acc09b090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4acbfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4acc0a13f0 0x7f4acc139ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47834/0 (socket says 172.21.15.63:47834) 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.646+0000 7f4acbfff640 1 -- 172.21.15.63:0/104442225 learned_addr learned my addr 172.21.15.63:0/104442225 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614973095 0 0) 0x7f4acc09ec90 con 0x7f4acc0a13f0 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient(hunting): my global_id is 5417 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ab4003530 con 0x7f4acc0a13f0 2024-09-16T05:38:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907996477 0 0) 0x7f4acc0a0fd0 con 0x7f4acc09b7a0 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient(hunting): my global_id is 5430 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4acc09ec90 con 0x7f4acc09b7a0 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3699507126 0 0) 0x7f4acc09ec90 con 0x7f4acc09b7a0 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 >> v1:172.21.15.6:6790/0 conn(0x7f4acc0a13f0 legacy=0x7f4acc139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 >> v1:172.21.15.63:6789/0 conn(0x7f4acc09acb0 legacy=0x7f4acc09b090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc0a0730 con 0x7f4acc09b7a0 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4ac4001980 con 0x7f4acc09b7a0 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: dump: 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 5 monclient: authenticate success, global_id 5430 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ac4001c70 con 0x7f4acc09b7a0 2024-09-16T05:38:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 1 -- 172.21.15.63:0/104442225 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4ac4002040 con 0x7f4acc09b7a0 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4acb7fe640 10 monclient: dump: 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 4 set_mon_vals no callback set 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient: shutdown 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/104442225 >> v1:172.21.15.6:6789/0 conn(0x7f4acc09b7a0 legacy=0x7f4acc13d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/104442225 shutdown_connections 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/104442225 >> 172.21.15.63:0/104442225 conn(0x7f4acc0a1b60 msgr2=0x7f4acc0a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/104442225 shutdown_connections 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/104442225 wait complete. 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient: build_initial_monmap 2024-09-16T05:38:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient: monmap: 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:04.662 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 Processor -- start 2024-09-16T05:38:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- start start 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient: init 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient(hunting): picked mon.noname-c con 0x7f4acc09acb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient(hunting): picked mon.noname-a con 0x7f4acc09b7a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient(hunting): picked mon.noname-b con 0x7f4acc0a13f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc1c6330 con 0x7f4acc09b7a0 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc09c0f0 con 0x7f4acc0a13f0 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc125660 con 0x7f4acc09acb0 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad2bdc640 10 monclient(hunting): authenticate will time out at 1141.431604s 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad1179640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4acc0a13f0 legacy=0x7f4acc1d1880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad0978640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4acc09acb0 0x7f4acc134c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47840/0 (socket says 172.21.15.63:47840) 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.650+0000 7f4ad0978640 1 -- 172.21.15.63:0/2730403599 learned_addr learned my addr 172.21.15.63:0/2730403599 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859751755 0 0) 0x7f4acc125660 con 0x7f4acc09acb0 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): my global_id is 5420 2024-09-16T05:38:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ab8003560 con 0x7f4acc09acb0 2024-09-16T05:38:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2438624788 0 0) 0x7f4ab4003530 con 0x7f4acc09b7a0 2024-09-16T05:38:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): my global_id is 5433 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4acc125660 con 0x7f4acc09b7a0 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1034958147 0 0) 0x7f4acc125660 con 0x7f4acc09b7a0 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ab4003530 con 0x7f4acc09b7a0 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4ac00018e0 con 0x7f4acc09b7a0 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): dump: 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 905523802 0 0) 0x7f4ab8003560 con 0x7f4acc09acb0 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4acc125660 con 0x7f4acc09acb0 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4abc004160 con 0x7f4acc09acb0 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:04.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient(hunting): dump: 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3455430290 0 0) 0x7f4ab4003530 con 0x7f4acc09b7a0 2024-09-16T05:38:04.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 >> v1:172.21.15.6:6790/0 conn(0x7f4acc09acb0 legacy=0x7f4acc134c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 >> v1:172.21.15.63:6789/0 conn(0x7f4acc0a13f0 legacy=0x7f4acc1d1880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient: _finish_hunting 0 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 monclient: found mon.a 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc0719f0 con 0x7f4acc09b7a0 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient: _finish_auth 0 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 5 monclient: authenticate success, global_id 5433 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 10 monclient: _renew_subs 2024-09-16T05:38:04.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4acc123370 con 0x7f4acc09b7a0 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 10 monclient: _renew_subs 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4acc123ff0 con 0x7f4acc09b7a0 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ac00047f0 con 0x7f4acc09b7a0 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4ac0004b40 con 0x7f4acc09b7a0 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 10 monclient: dump: 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:04.674 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4acaffd640 4 set_mon_vals no callback set 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aac005090 con 0x7f4acc09b7a0 2024-09-16T05:38:04.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4ac0004d60 con 0x7f4acc09b7a0 2024-09-16T05:38:04.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(115..115 src has 1..115) ==== 5803+0+0 (unknown 2054104351 0 0) 0x7f4ac0044a00 con 0x7f4acc09b7a0 2024-09-16T05:38:04.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.654+0000 7f4ad0978640 10 monclient: get_auth_request con 0x7f4ab8036aa0 auth_method 0 2024-09-16T05:38:04.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.670+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4acc1c6330 con 0x7f4acc09b7a0 2024-09-16T05:38:04.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.670+0000 7f4ac97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:04.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.670+0000 7f4ac97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.834+0000 7f4ad2bdc640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:38:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.834+0000 7f4ad2bdc640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:38:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.834+0000 7f4ad2bdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:04.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.834+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f4aac005b90 con 0x7f4acc09b7a0 2024-09-16T05:38:04.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.838+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v115) ==== 56+0+1449 (unknown 2103320864 0 1010395886) 0x7f4ac00153d0 con 0x7f4acc09b7a0 2024-09-16T05:38:04.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.838+0000 7f4ac97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:38:04.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.838+0000 7f4ac97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:04.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:38:04.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:38:04.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:04.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f4aac005d60 con 0x7f4acc09b7a0 2024-09-16T05:38:04.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ac97fa640 1 -- 172.21.15.63:0/2730403599 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v115) ==== 56+0+1449 (unknown 2103320864 0 1010395886) 0x7f4ac0014cd0 con 0x7f4acc09b7a0 2024-09-16T05:38:04.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ac97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:38:04.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ac97fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:04.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4ab8036aa0 legacy=0x7f4ab8038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:04.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 10 monclient: shutdown 2024-09-16T05:38:04.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:04.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 >> v1:172.21.15.6:6789/0 conn(0x7f4acc09b7a0 legacy=0x7f4acc1ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:04.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 shutdown_connections 2024-09-16T05:38:04.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 >> 172.21.15.63:0/2730403599 conn(0x7f4acc0a1b60 msgr2=0x7f4acc0a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:04.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 shutdown_connections 2024-09-16T05:38:04.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:04.938+0000 7f4ad2bdc640 1 -- 172.21.15.63:0/2730403599 wait complete. 2024-09-16T05:38:04.959 INFO:tasks.workunit.client.0.smithi063.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2024-09-16T05:38:04.960 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:38:04.961 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep osd.14 2024-09-16T05:38:04.961 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 123 2024-09-16T05:38:05.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.074+0000 7f68239e2640 10 monclient: get_monmap_and_config 2024-09-16T05:38:05.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.074+0000 7f68239e2640 10 monclient: build_initial_monmap 2024-09-16T05:38:05.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.074+0000 7f68239e2640 10 monclient: monmap: 2024-09-16T05:38:05.079 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:05.079 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.079 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:05.078800+0000 2024-09-16T05:38:05.079 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:05.078800+0000 2024-09-16T05:38:05.079 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:05.080 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.080 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:05.080 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:05.080 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:05.080 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.074+0000 7f68239e2640 1 Processor -- start 2024-09-16T05:38:05.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.074+0000 7f68239e2640 1 -- start start 2024-09-16T05:38:05.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.074+0000 7f68239e2640 10 monclient: init 2024-09-16T05:38:05.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:05.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient(hunting): picked mon.noname-b con 0x7f681c09acb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient(hunting): picked mon.noname-a con 0x7f681c0a13f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient(hunting): picked mon.noname-c con 0x7f681c09b7a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681c0a0fd0 con 0x7f681c0a13f0 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681c09f110 con 0x7f681c09b7a0 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681c09ec90 con 0x7f681c09acb0 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient(hunting): authenticate will time out at 1141.856347s 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6821f7f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f681c09b7a0 legacy=0x7f681c13d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f682177e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f681c09acb0 0x7f681c09b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47850/0 (socket says 172.21.15.63:47850) 2024-09-16T05:38:05.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f682177e640 1 -- 172.21.15.63:0/1370701473 learned_addr learned my addr 172.21.15.63:0/1370701473 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:05.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015380496 0 0) 0x7f681c0a0fd0 con 0x7f681c0a13f0 2024-09-16T05:38:05.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:05.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:05.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:05.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient(hunting): my global_id is 5436 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6808003530 con 0x7f681c0a13f0 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112036625 0 0) 0x7f681c09ec90 con 0x7f681c09acb0 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient(hunting): my global_id is 5423 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f681c0a0fd0 con 0x7f681c09acb0 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1143587493 0 0) 0x7f6808003530 con 0x7f681c0a13f0 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 >> v1:172.21.15.6:6790/0 conn(0x7f681c09acb0 legacy=0x7f681c09b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 >> v1:172.21.15.63:6789/0 conn(0x7f681c09b7a0 legacy=0x7f681c13d1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 monclient: found mon.noname-a 2024-09-16T05:38:05.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c0a0730 con 0x7f681c0a13f0 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: _finish_auth 0 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f680c001550 con 0x7f681c0a13f0 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: dump: 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.089 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 5 monclient: authenticate success, global_id 5436 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f680c0017b0 con 0x7f681c0a13f0 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 1 -- 172.21.15.63:0/1370701473 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f680c001b00 con 0x7f681c0a13f0 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f6813fff640 10 monclient: dump: 2024-09-16T05:38:05.090 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 4 set_mon_vals no callback set 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient: shutdown 2024-09-16T05:38:05.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- 172.21.15.63:0/1370701473 >> v1:172.21.15.6:6789/0 conn(0x7f681c0a13f0 legacy=0x7f681c139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- 172.21.15.63:0/1370701473 shutdown_connections 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- 172.21.15.63:0/1370701473 >> 172.21.15.63:0/1370701473 conn(0x7f681c0a1b60 msgr2=0x7f681c0a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- 172.21.15.63:0/1370701473 shutdown_connections 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- 172.21.15.63:0/1370701473 wait complete. 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient: build_initial_monmap 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient: monmap: 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.092 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 Processor -- start 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 1 -- start start 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.078+0000 7f68239e2640 10 monclient: init 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient(hunting): picked mon.noname-b con 0x7f681c09acb0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient(hunting): picked mon.noname-c con 0x7f681c09b7a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient(hunting): picked mon.noname-a con 0x7f681c0a13f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681c121450 con 0x7f681c0a13f0 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f682177e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f681c09acb0 legacy=0x7f681c133cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:05.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681c06de40 con 0x7f681c09acb0 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681c11f160 con 0x7f681c09b7a0 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient(hunting): authenticate will time out at 1141.860258s 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6820f7d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f681c09b7a0 0x7f681c1ce180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47856/0 (socket says 172.21.15.63:47856) 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6820f7d640 1 -- 172.21.15.63:0/3496449115 learned_addr learned my addr 172.21.15.63:0/3496449115 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1415205619 0 0) 0x7f681c11f160 con 0x7f681c09b7a0 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:05.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): my global_id is 5426 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6800003530 con 0x7f681c09b7a0 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278143477 0 0) 0x7f681c121450 con 0x7f681c0a13f0 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): my global_id is 5439 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f681c11f160 con 0x7f681c0a13f0 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1822076572 0 0) 0x7f6800003530 con 0x7f681c09b7a0 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f681c121450 con 0x7f681c09b7a0 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f680c001bc0 con 0x7f681c09b7a0 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:05.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4288182358 0 0) 0x7f681c11f160 con 0x7f681c0a13f0 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6800003530 con 0x7f681c0a13f0 2024-09-16T05:38:05.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6814004340 con 0x7f681c0a13f0 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.097 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2913963613 0 0) 0x7f681c121450 con 0x7f681c09b7a0 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 >> v1:172.21.15.63:6789/0 conn(0x7f681c09acb0 legacy=0x7f681c133cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 >> v1:172.21.15.6:6789/0 conn(0x7f681c0a13f0 legacy=0x7f681c1d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 monclient: found mon.c 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c121170 con 0x7f681c09b7a0 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 5 monclient: authenticate success, global_id 5426 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient: _renew_subs 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f681c11fa40 con 0x7f681c09b7a0 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient: _renew_subs 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f681c1345a0 con 0x7f681c09b7a0 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f680c001790 con 0x7f681c09b7a0 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f680c003b60 con 0x7f681c09b7a0 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68137fe640 4 set_mon_vals no callback set 2024-09-16T05:38:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 10 monclient: dump: 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:05.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67fc005090 con 0x7f681c09b7a0 2024-09-16T05:38:05.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f680c003d80 con 0x7f681c09b7a0 2024-09-16T05:38:05.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(115..115 src has 1..115) ==== 5803+0+0 (unknown 2054104351 0 0) 0x7f680c044a20 con 0x7f681c09b7a0 2024-09-16T05:38:05.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.082+0000 7f682177e640 10 monclient: get_auth_request con 0x7f68000369b0 auth_method 0 2024-09-16T05:38:05.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.094+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f680c019040 con 0x7f681c09b7a0 2024-09-16T05:38:05.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.094+0000 7f6811ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:05.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.094+0000 7f6811ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.254+0000 7f68239e2640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:38:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.254+0000 7f68239e2640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:38:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.254+0000 7f68239e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.254+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f67fc005b90 con 0x7f681c09b7a0 2024-09-16T05:38:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.254+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v115) ==== 56+0+1449 (unknown 2103320864 0 1010395886) 0x7f680c01d900 con 0x7f681c09b7a0 2024-09-16T05:38:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.254+0000 7f6811ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:38:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.254+0000 7f6811ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:05.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.350+0000 7f68239e2640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:38:05.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.350+0000 7f68239e2640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:38:05.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.350+0000 7f68239e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.350+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f67fc0008e0 con 0x7f681c09b7a0 2024-09-16T05:38:05.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.350+0000 7f6811ffb640 1 -- 172.21.15.63:0/3496449115 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v115) ==== 56+0+1449 (unknown 2103320864 0 1010395886) 0x7f680c040260 con 0x7f681c09b7a0 2024-09-16T05:38:05.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.350+0000 7f6811ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:38:05.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.350+0000 7f6811ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:05.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.354+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 >> v1:172.21.15.63:6812/3643207211 conn(0x7f68000369b0 legacy=0x7f6800038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:05.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.354+0000 7f68239e2640 10 monclient: shutdown 2024-09-16T05:38:05.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.354+0000 7f68239e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:05.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.354+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 >> v1:172.21.15.6:6790/0 conn(0x7f681c09b7a0 legacy=0x7f681c1ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:05.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.354+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 shutdown_connections 2024-09-16T05:38:05.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.354+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 >> 172.21.15.63:0/3496449115 conn(0x7f681c0a1b60 msgr2=0x7f681c1349c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:05.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.354+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 shutdown_connections 2024-09-16T05:38:05.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.354+0000 7f68239e2640 1 -- 172.21.15.63:0/3496449115 wait complete. 2024-09-16T05:38:05.370 INFO:tasks.workunit.client.0.smithi063.stdout: 14 123.00000 osd.14 down 1.00000 1.00000 2024-09-16T05:38:05.371 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush reweight-subtree foobaz 155 2024-09-16T05:38:05.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.486+0000 7ff578f08640 10 monclient: get_monmap_and_config 2024-09-16T05:38:05.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.486+0000 7ff578f08640 10 monclient: build_initial_monmap 2024-09-16T05:38:05.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.486+0000 7ff578f08640 10 monclient: monmap: 2024-09-16T05:38:05.492 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:05.492 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.493 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:05.491688+0000 2024-09-16T05:38:05.493 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:05.491688+0000 2024-09-16T05:38:05.493 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:05.493 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.493 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:05.493 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:05.493 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:05.493 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.486+0000 7ff578f08640 1 Processor -- start 2024-09-16T05:38:05.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.486+0000 7ff578f08640 1 -- start start 2024-09-16T05:38:05.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.486+0000 7ff578f08640 10 monclient: init 2024-09-16T05:38:05.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:05.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:05.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient(hunting): picked mon.noname-b con 0x7ff57409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:05.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient(hunting): picked mon.noname-c con 0x7ff5740a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:05.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient(hunting): picked mon.noname-a con 0x7ff57409b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:05.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff574074200 con 0x7ff57409b780 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57409f900 con 0x7ff5740a0280 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57409e870 con 0x7ff57409ac90 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient(hunting): authenticate will time out at 1142.268635s 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff571d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff5740a0280 legacy=0x7ff5741398b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff572d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff57409b780 0x7ff57413cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56136/0 (socket says 172.21.15.63:56136) 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff57409ac90 0x7ff57409b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47866/0 (socket says 172.21.15.63:47866) 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff572d9d640 1 -- 172.21.15.63:0/2209114643 learned_addr learned my addr 172.21.15.63:0/2209114643 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1526780927 0 0) 0x7ff57409e870 con 0x7ff57409ac90 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:05.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient(hunting): my global_id is 5429 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff55c003530 con 0x7ff57409ac90 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3149303500 0 0) 0x7ff574074200 con 0x7ff57409b780 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient(hunting): my global_id is 5442 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff57409e870 con 0x7ff57409b780 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1676704133 0 0) 0x7ff55c003530 con 0x7ff57409ac90 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 >> v1:172.21.15.63:6789/0 conn(0x7ff5740a0280 legacy=0x7ff5741398b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 >> v1:172.21.15.6:6789/0 conn(0x7ff57409b780 legacy=0x7ff57413cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:05.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 monclient: found mon.noname-b 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff574141730 con 0x7ff57409ac90 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: _finish_auth 0 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff564001620 con 0x7ff57409ac90 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: dump: 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.507 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 5 monclient: authenticate success, global_id 5429 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff564001880 con 0x7ff57409ac90 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:05.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 1 -- 172.21.15.63:0/2209114643 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff564001bd0 con 0x7ff57409ac90 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff57159a640 10 monclient: dump: 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.509 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 4 set_mon_vals no callback set 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient: shutdown 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 -- 172.21.15.63:0/2209114643 >> v1:172.21.15.6:6790/0 conn(0x7ff57409ac90 legacy=0x7ff57409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 -- 172.21.15.63:0/2209114643 shutdown_connections 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 -- 172.21.15.63:0/2209114643 >> 172.21.15.63:0/2209114643 conn(0x7ff5740a5290 msgr2=0x7ff5740a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 -- 172.21.15.63:0/2209114643 shutdown_connections 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 -- 172.21.15.63:0/2209114643 wait complete. 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient: build_initial_monmap 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 10 monclient: monmap: 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:05.510 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.490+0000 7ff578f08640 1 Processor -- start 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 1 -- start start 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: init 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:05.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient(hunting): picked mon.noname-c con 0x7ff57409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient(hunting): picked mon.noname-a con 0x7ff57409b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient(hunting): picked mon.noname-b con 0x7ff5740a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57409d250 con 0x7ff57409b780 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57409e0d0 con 0x7ff5740a0280 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff574140c70 con 0x7ff57409ac90 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff572d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff5740a0280 legacy=0x7ff5741d36b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient(hunting): authenticate will time out at 1142.272584s 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff571d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff57409b780 0x7ff57409cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56140/0 (socket says 172.21.15.63:56140) 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff571d9b640 1 -- 172.21.15.63:0/2744746663 learned_addr learned my addr 172.21.15.63:0/2744746663 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:05.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 935415490 0 0) 0x7ff574140c70 con 0x7ff57409ac90 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): my global_id is 5432 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff554003530 con 0x7ff57409ac90 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805327160 0 0) 0x7ff57409d250 con 0x7ff57409b780 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): my global_id is 5445 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff574140c70 con 0x7ff57409b780 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3415524132 0 0) 0x7ff554003530 con 0x7ff57409ac90 2024-09-16T05:38:05.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff57409d250 con 0x7ff57409ac90 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff5640015e0 con 0x7ff57409ac90 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.514 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2569186687 0 0) 0x7ff574140c70 con 0x7ff57409b780 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff554003530 con 0x7ff57409b780 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff568004150 con 0x7ff57409b780 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.515 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2464517792 0 0) 0x7ff57409d250 con 0x7ff57409ac90 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 >> v1:172.21.15.63:6789/0 conn(0x7ff5740a0280 legacy=0x7ff5741d36b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 >> v1:172.21.15.6:6789/0 conn(0x7ff57409b780 legacy=0x7ff57409cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 monclient: found mon.c 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:05.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57413f820 con 0x7ff57409ac90 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 5 monclient: authenticate success, global_id 5432 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: _renew_subs 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff57413f4e0 con 0x7ff57409ac90 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: _renew_subs 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5741d4dd0 con 0x7ff57409ac90 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff564001840 con 0x7ff57409ac90 2024-09-16T05:38:05.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff564004c40 con 0x7ff57409ac90 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff570d99640 4 set_mon_vals no callback set 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 10 monclient: dump: 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:05.518 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff558005090 con 0x7ff57409ac90 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.494+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff564004e60 con 0x7ff57409ac90 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.498+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(115..115 src has 1..115) ==== 5803+0+0 (unknown 2054104351 0 0) 0x7ff564044a60 con 0x7ff57409ac90 2024-09-16T05:38:05.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.498+0000 7ff571d9b640 10 monclient: get_auth_request con 0x7ff554036a20 auth_method 0 2024-09-16T05:38:05.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.510+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff564015430 con 0x7ff57409ac90 2024-09-16T05:38:05.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.510+0000 7ff562ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:05.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.510+0000 7ff562ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:05.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.682+0000 7ff578f08640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T05:38:05.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.682+0000 7ff578f08640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T05:38:05.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.682+0000 7ff578f08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:05.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:05.682+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) -- 0x7ff558005380 con 0x7ff57409ac90 2024-09-16T05:38:06.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:06.418+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v116) ==== 168+0+0 (unknown 2088554451 0 0) 0x7ff564019080 con 0x7ff57409ac90 2024-09-16T05:38:06.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:06.418+0000 7ff562ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T05:38:06.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:06.418+0000 7ff562ffd640 10 monclient: _finish_command 2 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T05:38:06.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:06.522+0000 7ff578f08640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T05:38:06.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:06.522+0000 7ff578f08640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T05:38:06.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:06.522+0000 7ff578f08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:06.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:06.522+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) -- 0x7ff558005ac0 con 0x7ff57409ac90 2024-09-16T05:38:07.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.422+0000 7ff562ffd640 1 -- 172.21.15.63:0/2744746663 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v117) ==== 168+0+0 (unknown 2794465517 0 0) 0x7ff56401d940 con 0x7ff57409ac90 2024-09-16T05:38:07.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.422+0000 7ff562ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T05:38:07.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.422+0000 7ff562ffd640 10 monclient: _finish_command 3 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T05:38:07.426 INFO:tasks.workunit.client.0.smithi063.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T05:38:07.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.426+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff554036a20 legacy=0x7ff554038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:07.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.426+0000 7ff578f08640 10 monclient: shutdown 2024-09-16T05:38:07.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.426+0000 7ff578f08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:07.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.426+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 >> v1:172.21.15.6:6790/0 conn(0x7ff57409ac90 legacy=0x7ff57409c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:07.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.426+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 shutdown_connections 2024-09-16T05:38:07.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.426+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 >> 172.21.15.63:0/2744746663 conn(0x7ff5740a5290 msgr2=0x7ff574137180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:07.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.426+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 shutdown_connections 2024-09-16T05:38:07.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.426+0000 7ff578f08640 1 -- 172.21.15.63:0/2744746663 wait complete. 2024-09-16T05:38:07.452 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2024-09-16T05:38:07.452 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:38:07.452 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2024-09-16T05:38:07.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient: get_monmap_and_config 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient: build_initial_monmap 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient: monmap: 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:07.577369+0000 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:07.577369+0000 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:07.578 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 1 Processor -- start 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 1 -- start start 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient: init 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient(hunting): picked mon.noname-c con 0x7fabc809b540 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient(hunting): picked mon.noname-b con 0x7fabc80a1520 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient(hunting): picked mon.noname-a con 0x7fabc8138d70 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:07.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc809f890 con 0x7fabc8138d70 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc8072e80 con 0x7fabc809b540 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc809ce30 con 0x7fabc80a1520 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 10 monclient(hunting): authenticate will time out at 1144.354260s 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fabc809b540 legacy=0x7fabc809fb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc77fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fabc80a1520 0x7fabc80a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47902/0 (socket says 172.21.15.63:47902) 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc77fe640 1 -- 172.21.15.63:0/3544531252 learned_addr learned my addr 172.21.15.63:0/3544531252 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2838563264 0 0) 0x7fabc809ce30 con 0x7fabc80a1520 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient(hunting): my global_id is 5435 2024-09-16T05:38:07.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabb0003450 con 0x7fabc80a1520 2024-09-16T05:38:07.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220816981 0 0) 0x7fabc809f890 con 0x7fabc8138d70 2024-09-16T05:38:07.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:07.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:07.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:07.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient(hunting): my global_id is 5448 2024-09-16T05:38:07.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabc809ce30 con 0x7fabc8138d70 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3068784145 0 0) 0x7fabb0003450 con 0x7fabc80a1520 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 >> v1:172.21.15.63:6789/0 conn(0x7fabc809b540 legacy=0x7fabc809fb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 >> v1:172.21.15.6:6789/0 conn(0x7fabc8138d70 legacy=0x7fabc813b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 monclient: found mon.noname-b 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc809d110 con 0x7fabc80a1520 2024-09-16T05:38:07.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fabbc001620 con 0x7fabc80a1520 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabc6ffd640 10 monclient: dump: 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 5 monclient: authenticate success, global_id 5435 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.574+0000 7fabce355640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fabbc001880 con 0x7fabc80a1520 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc6ffd640 1 -- 172.21.15.63:0/3544531252 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fabbc001bd0 con 0x7fabc80a1520 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc6ffd640 10 monclient: dump: 2024-09-16T05:38:07.589 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 4 set_mon_vals no callback set 2024-09-16T05:38:07.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient: shutdown 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- 172.21.15.63:0/3544531252 >> v1:172.21.15.6:6790/0 conn(0x7fabc80a1520 legacy=0x7fabc80a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- 172.21.15.63:0/3544531252 shutdown_connections 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- 172.21.15.63:0/3544531252 >> 172.21.15.63:0/3544531252 conn(0x7fabc80a5290 msgr2=0x7fabc80a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- 172.21.15.63:0/3544531252 shutdown_connections 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- 172.21.15.63:0/3544531252 wait complete. 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient: build_initial_monmap 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient: monmap: 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:07.591 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 Processor -- start 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- start start 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient: init 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient(hunting): picked mon.noname-c con 0x7fabc809b540 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient(hunting): picked mon.noname-a con 0x7fabc80a1520 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient(hunting): picked mon.noname-b con 0x7fabc8138d70 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:07.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc8121a80 con 0x7fabc80a1520 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc8121d00 con 0x7fabc8138d70 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc81231b0 con 0x7fabc809b540 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabce355640 10 monclient(hunting): authenticate will time out at 1144.358412s 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabcc8f2640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fabc8138d70 legacy=0x7fabc81cfae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fabc809b540 0x7fabc8137750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47908/0 (socket says 172.21.15.63:47908) 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc7fff640 1 -- 172.21.15.63:0/925583815 learned_addr learned my addr 172.21.15.63:0/925583815 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2777651336 0 0) 0x7fabc81231b0 con 0x7fabc809b540 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): my global_id is 5438 2024-09-16T05:38:07.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabb4003560 con 0x7fabc809b540 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2726445544 0 0) 0x7fabc8121a80 con 0x7fabc80a1520 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): my global_id is 5451 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabc81231b0 con 0x7fabc80a1520 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 151735902 0 0) 0x7fabb4003560 con 0x7fabc809b540 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabc8121a80 con 0x7fabc809b540 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fabb8004090 con 0x7fabc809b540 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:07.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1025206492 0 0) 0x7fabc81231b0 con 0x7fabc80a1520 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabb4003560 con 0x7fabc80a1520 2024-09-16T05:38:07.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.578+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fabbc0015e0 con 0x7fabc80a1520 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.596 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2757082435 0 0) 0x7fabc8121a80 con 0x7fabc809b540 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 >> v1:172.21.15.63:6789/0 conn(0x7fabc8138d70 legacy=0x7fabc81cfae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 >> v1:172.21.15.6:6789/0 conn(0x7fabc80a1520 legacy=0x7fabc81cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 monclient: found mon.c 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc8072b70 con 0x7fabc809b540 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:07.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 5 monclient: authenticate success, global_id 5438 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 10 monclient: _renew_subs 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fabc8122bf0 con 0x7fabc809b540 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 10 monclient: _renew_subs 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fabc81205f0 con 0x7fabc809b540 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fabb8004340 con 0x7fabc809b540 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc67fc640 4 set_mon_vals no callback set 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fabb8004710 con 0x7fabc809b540 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:07.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 10 monclient: dump: 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabce355640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba8005090 con 0x7fabc809b540 2024-09-16T05:38:07.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fabb8004930 con 0x7fabc809b540 2024-09-16T05:38:07.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(117..117 src has 1..117) ==== 5803+0+0 (unknown 70293949 0 0) 0x7fabb80449d0 con 0x7fabc809b540 2024-09-16T05:38:07.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.582+0000 7fabc77fe640 10 monclient: get_auth_request con 0x7fabb4036a50 auth_method 0 2024-09-16T05:38:07.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.594+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fabb80153a0 con 0x7fabc809b540 2024-09-16T05:38:07.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.594+0000 7fabc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:07.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.594+0000 7fabc4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:07.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.766+0000 7fabce355640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T05:38:07.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.766+0000 7fabce355640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T05:38:07.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.766+0000 7fabce355640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.766+0000 7fabce355640 1 -- 172.21.15.63:0/925583815 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0} v 0) -- 0x7faba8005380 con 0x7fabc809b540 2024-09-16T05:38:07.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.770+0000 7fabc4ff9640 1 -- 172.21.15.63:0/925583815 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}]=-75 (75) Value too large for defined data type v117) ==== 154+0+0 (unknown 2572779666 0 0) 0x7fabb8014ca0 con 0x7fabc809b540 2024-09-16T05:38:07.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.770+0000 7fabc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T05:38:07.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.770+0000 7fabc4ff9640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-16T05:38:07.773 INFO:tasks.workunit.client.0.smithi063.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-16T05:38:07.777 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:38:07.777 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:38:07.778 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep osd.12 2024-09-16T05:38:07.778 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 155 2024-09-16T05:38:07.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.890+0000 7f8cc0b40640 10 monclient: get_monmap_and_config 2024-09-16T05:38:07.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.890+0000 7f8cc0b40640 10 monclient: build_initial_monmap 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.890+0000 7f8cc0b40640 10 monclient: monmap: 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:07.896161+0000 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:07.896161+0000 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:07.897 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 Processor -- start 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 -- start start 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient: init 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient(hunting): picked mon.noname-c con 0x7f8cbc136da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient(hunting): picked mon.noname-a con 0x7f8cbc0a4e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient(hunting): picked mon.noname-b con 0x7f8cbc09fe80 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc0a0ea0 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc0a43d0 con 0x7f8cbc136da0 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc0a4850 con 0x7f8cbc09fe80 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient(hunting): authenticate will time out at 1144.673602s 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cba59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8cbc136da0 legacy=0x7f8cbc137180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cbad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8cbc09fe80 0x7f8cbc13f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47912/0 (socket says 172.21.15.63:47912) 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cbad9d640 1 -- 172.21.15.63:0/2484028745 learned_addr learned my addr 172.21.15.63:0/2484028745 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2469593436 0 0) 0x7f8cbc0a4850 con 0x7f8cbc09fe80 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient(hunting): my global_id is 5441 2024-09-16T05:38:07.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ca8003530 con 0x7f8cbc09fe80 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189002662 0 0) 0x7f8cbc0a0ea0 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient(hunting): my global_id is 5454 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cbc0a4850 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1293188842 0 0) 0x7f8ca8003530 con 0x7f8cbc09fe80 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 >> v1:172.21.15.63:6789/0 conn(0x7f8cbc136da0 legacy=0x7f8cbc137180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 >> v1:172.21.15.6:6789/0 conn(0x7f8cbc0a4e40 legacy=0x7f8cbc13bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 monclient: found mon.noname-b 2024-09-16T05:38:07.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc137890 con 0x7f8cbc09fe80 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: _finish_auth 0 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8cb40019a0 con 0x7f8cbc09fe80 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: dump: 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.903 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 5 monclient: authenticate success, global_id 5441 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8cb4001cc0 con 0x7f8cbc09fe80 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 1 -- 172.21.15.63:0/2484028745 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8cb4002090 con 0x7f8cbc09fe80 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:07.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 4 set_mon_vals no callback set 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cb9d9b640 10 monclient: dump: 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.905 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 10 monclient: shutdown 2024-09-16T05:38:07.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 -- 172.21.15.63:0/2484028745 >> v1:172.21.15.6:6790/0 conn(0x7f8cbc09fe80 legacy=0x7f8cbc13f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 -- 172.21.15.63:0/2484028745 shutdown_connections 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 -- 172.21.15.63:0/2484028745 >> 172.21.15.63:0/2484028745 conn(0x7f8cbc0a5e80 msgr2=0x7f8cbc0a6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.894+0000 7f8cc0b40640 1 -- 172.21.15.63:0/2484028745 shutdown_connections 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 1 -- 172.21.15.63:0/2484028745 wait complete. 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: build_initial_monmap 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: monmap: 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:07.911 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 1 Processor -- start 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 1 -- start start 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: init 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient(hunting): picked mon.noname-b con 0x7f8cbc09fe80 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient(hunting): picked mon.noname-a con 0x7f8cbc0a4e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient(hunting): picked mon.noname-c con 0x7f8cbc136da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:07.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc0a0570 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc0a7810 con 0x7f8cbc09fe80 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cba59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8cbc09fe80 legacy=0x7f8cbc135c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc0a72e0 con 0x7f8cbc136da0 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient(hunting): authenticate will time out at 1144.677559s 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8cbc0a4e40 0x7f8cbc1d0530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56176/0 (socket says 172.21.15.63:56176) 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb3fff640 1 -- 172.21.15.63:0/1398861866 learned_addr learned my addr 172.21.15.63:0/1398861866 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cbad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8cbc136da0 0x7f8cbc1d3c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47916/0 (socket says 172.21.15.63:47916) 2024-09-16T05:38:07.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84794509 0 0) 0x7f8cbc0a72e0 con 0x7f8cbc136da0 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): my global_id is 5444 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c9c003530 con 0x7f8cbc136da0 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821597801 0 0) 0x7f8cbc0a0570 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): my global_id is 5457 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cbc0a72e0 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 667261665 0 0) 0x7f8c9c003530 con 0x7f8cbc136da0 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8cbc0a0570 con 0x7f8cbc136da0 2024-09-16T05:38:07.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8cb40017c0 con 0x7f8cbc136da0 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.915 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2624717522 0 0) 0x7f8cbc0a72e0 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c9c003530 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8ca4001900 con 0x7f8cbc0a4e40 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.916 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1883451606 0 0) 0x7f8cbc0a0570 con 0x7f8cbc136da0 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 >> v1:172.21.15.63:6789/0 conn(0x7f8cbc09fe80 legacy=0x7f8cbc135c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 >> v1:172.21.15.6:6789/0 conn(0x7f8cbc0a4e40 legacy=0x7f8cbc1d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 monclient: found mon.c 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc123320 con 0x7f8cbc136da0 2024-09-16T05:38:07.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 5 monclient: authenticate success, global_id 5444 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: _renew_subs 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8cbc123600 con 0x7f8cbc136da0 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: _renew_subs 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8cbc136350 con 0x7f8cbc136da0 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8cb4001cc0 con 0x7f8cbc136da0 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8cb4004fb0 con 0x7f8cbc136da0 2024-09-16T05:38:07.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb959a640 4 set_mon_vals no callback set 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.898+0000 7f8cb2ffd640 10 monclient: dump: 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:07.919 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:07.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.902+0000 7f8cc0b40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:07.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.902+0000 7f8cc0b40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:07.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.902+0000 7f8cc0b40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:07.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.902+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca0005090 con 0x7f8cbc136da0 2024-09-16T05:38:07.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.902+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8cb400fd50 con 0x7f8cbc136da0 2024-09-16T05:38:07.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.902+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(117..117 src has 1..117) ==== 5803+0+0 (unknown 70293949 0 0) 0x7f8cb4044b80 con 0x7f8cbc136da0 2024-09-16T05:38:07.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.902+0000 7f8cba59c640 10 monclient: get_auth_request con 0x7f8c9c036a00 auth_method 0 2024-09-16T05:38:07.920 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.914+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8cb40191a0 con 0x7f8cbc136da0 2024-09-16T05:38:07.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.914+0000 7f8cb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:07.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:07.914+0000 7f8cb2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:08.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.082+0000 7f8cc0b40640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:38:08.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.082+0000 7f8cc0b40640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:38:08.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.082+0000 7f8cc0b40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.082+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f8ca0005b90 con 0x7f8cbc136da0 2024-09-16T05:38:08.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.086+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v117) ==== 56+0+1449 (unknown 919893539 0 1261197053) 0x7f8cb401da60 con 0x7f8cbc136da0 2024-09-16T05:38:08.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.086+0000 7f8cb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:38:08.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.086+0000 7f8cb2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.182+0000 7f8cc0b40640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:38:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.182+0000 7f8cc0b40640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:38:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.182+0000 7f8cc0b40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.182+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f8ca00008e0 con 0x7f8cbc136da0 2024-09-16T05:38:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cb2ffd640 1 -- 172.21.15.63:0/1398861866 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v117) ==== 56+0+1449 (unknown 919893539 0 1261197053) 0x7f8cb40405e0 con 0x7f8cbc136da0 2024-09-16T05:38:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cb2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:38:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cb2ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:08.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8c9c036a00 legacy=0x7f8c9c038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cc0b40640 10 monclient: shutdown 2024-09-16T05:38:08.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cc0b40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:08.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 >> v1:172.21.15.6:6790/0 conn(0x7f8cbc136da0 legacy=0x7f8cbc1d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 shutdown_connections 2024-09-16T05:38:08.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 >> 172.21.15.63:0/1398861866 conn(0x7f8cbc0a5e80 msgr2=0x7f8cbc1bca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:08.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 shutdown_connections 2024-09-16T05:38:08.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.186+0000 7f8cc0b40640 1 -- 172.21.15.63:0/1398861866 wait complete. 2024-09-16T05:38:08.206 INFO:tasks.workunit.client.0.smithi063.stdout: 12 155.00000 osd.12 down 1.00000 1.00000 2024-09-16T05:38:08.207 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd tree 2024-09-16T05:38:08.207 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep osd.14 2024-09-16T05:38:08.207 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 155 2024-09-16T05:38:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient: get_monmap_and_config 2024-09-16T05:38:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient: build_initial_monmap 2024-09-16T05:38:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient: monmap: 2024-09-16T05:38:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:08.326753+0000 2024-09-16T05:38:08.327 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:08.326753+0000 2024-09-16T05:38:08.328 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:08.328 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.328 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:08.328 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:08.328 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:08.328 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 1 Processor -- start 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 1 -- start start 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient: init 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient(hunting): picked mon.noname-b con 0x7f18d809acb0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient(hunting): picked mon.noname-a con 0x7f18d80a13f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient(hunting): picked mon.noname-c con 0x7f18d809b7a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d80a0fd0 con 0x7f18d80a13f0 2024-09-16T05:38:08.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d809f110 con 0x7f18d809b7a0 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d809ec90 con 0x7f18d809acb0 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18de486640 10 monclient(hunting): authenticate will time out at 1145.103869s 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.322+0000 7f18dca23640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f18d809b7a0 legacy=0x7f18d813d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f18d80a13f0 0x7f18d8139ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56184/0 (socket says 172.21.15.63:56184) 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f18d809acb0 0x7f18d809b090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47920/0 (socket says 172.21.15.63:47920) 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d77fe640 1 -- 172.21.15.63:0/2599106610 learned_addr learned my addr 172.21.15.63:0/2599106610 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225532437 0 0) 0x7f18d809ec90 con 0x7f18d809acb0 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:08.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient(hunting): my global_id is 5447 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18c0003530 con 0x7f18d809acb0 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3489136183 0 0) 0x7f18d80a0fd0 con 0x7f18d80a13f0 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient(hunting): my global_id is 5460 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18d809ec90 con 0x7f18d80a13f0 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 326770160 0 0) 0x7f18c0003530 con 0x7f18d809acb0 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 >> v1:172.21.15.63:6789/0 conn(0x7f18d809b7a0 legacy=0x7f18d813d1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 >> v1:172.21.15.6:6789/0 conn(0x7f18d80a13f0 legacy=0x7f18d8139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:08.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 monclient: found mon.noname-b 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d80a0730 con 0x7f18d809acb0 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18c8001620 con 0x7f18d809acb0 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 5 monclient: authenticate success, global_id 5447 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:08.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: dump: 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18c8001880 con 0x7f18d809acb0 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:08.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 1 -- 172.21.15.63:0/2599106610 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18c8001c10 con 0x7f18d809acb0 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 4 set_mon_vals no callback set 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d6ffd640 10 monclient: dump: 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.336 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient: shutdown 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- 172.21.15.63:0/2599106610 >> v1:172.21.15.6:6790/0 conn(0x7f18d809acb0 legacy=0x7f18d809b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- 172.21.15.63:0/2599106610 shutdown_connections 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- 172.21.15.63:0/2599106610 >> 172.21.15.63:0/2599106610 conn(0x7f18d80a1b60 msgr2=0x7f18d80a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- 172.21.15.63:0/2599106610 shutdown_connections 2024-09-16T05:38:08.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- 172.21.15.63:0/2599106610 wait complete. 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient: build_initial_monmap 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient: monmap: 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:08.341 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 Processor -- start 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- start start 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient: init 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient(hunting): picked mon.noname-a con 0x7f18d809acb0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient(hunting): picked mon.noname-b con 0x7f18d809b7a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient(hunting): picked mon.noname-c con 0x7f18d80a13f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d8121140 con 0x7f18d809acb0 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d8121420 con 0x7f18d809b7a0 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d811ff70 con 0x7f18d80a13f0 2024-09-16T05:38:08.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18de486640 10 monclient(hunting): authenticate will time out at 1145.107833s 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.326+0000 7f18d77fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f18d809b7a0 legacy=0x7f18d81d0140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f18d809acb0 0x7f18d8133d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56196/0 (socket says 172.21.15.63:56196) 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d7fff640 1 -- 172.21.15.63:0/432663423 learned_addr learned my addr 172.21.15.63:0/432663423 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2461239101 0 0) 0x7f18d811ff70 con 0x7f18d80a13f0 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): my global_id is 5450 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18c4003530 con 0x7f18d80a13f0 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041098306 0 0) 0x7f18d8121140 con 0x7f18d809acb0 2024-09-16T05:38:08.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): my global_id is 5463 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18d811ff70 con 0x7f18d809acb0 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 250358590 0 0) 0x7f18c4003530 con 0x7f18d80a13f0 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18d8121140 con 0x7f18d80a13f0 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18d0004340 con 0x7f18d80a13f0 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.344 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2585278501 0 0) 0x7f18d811ff70 con 0x7f18d809acb0 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18c4003530 con 0x7f18d809acb0 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18c8001520 con 0x7f18d809acb0 2024-09-16T05:38:08.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.346 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2364681382 0 0) 0x7f18d8121140 con 0x7f18d80a13f0 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 >> v1:172.21.15.63:6789/0 conn(0x7f18d809b7a0 legacy=0x7f18d81d0140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 >> v1:172.21.15.6:6789/0 conn(0x7f18d809acb0 legacy=0x7f18d8133d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 monclient: found mon.c 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d809c0f0 con 0x7f18d80a13f0 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 5 monclient: authenticate success, global_id 5450 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 10 monclient: _renew_subs 2024-09-16T05:38:08.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18d81346b0 con 0x7f18d80a13f0 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 10 monclient: _renew_subs 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f18d81cae20 con 0x7f18d80a13f0 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18d00044e0 con 0x7f18d80a13f0 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d5ffb640 4 set_mon_vals no callback set 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18d00048b0 con 0x7f18d80a13f0 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 10 monclient: dump: 2024-09-16T05:38:08.348 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:08.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b8005090 con 0x7f18d80a13f0 2024-09-16T05:38:08.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f18d0004ad0 con 0x7f18d80a13f0 2024-09-16T05:38:08.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(117..117 src has 1..117) ==== 5803+0+0 (unknown 70293949 0 0) 0x7f18d0044b40 con 0x7f18d80a13f0 2024-09-16T05:38:08.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.330+0000 7f18d7fff640 10 monclient: get_auth_request con 0x7f18c4036990 auth_method 0 2024-09-16T05:38:08.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.342+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f18d0015510 con 0x7f18d80a13f0 2024-09-16T05:38:08.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.342+0000 7f18d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:08.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.342+0000 7f18d4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:08.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.510+0000 7f18de486640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:38:08.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.510+0000 7f18de486640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T05:38:08.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.510+0000 7f18de486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.510+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f18b8005b90 con 0x7f18d80a13f0 2024-09-16T05:38:08.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.510+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v117) ==== 56+0+1449 (unknown 919893539 0 1261197053) 0x7f18d0014e10 con 0x7f18d80a13f0 2024-09-16T05:38:08.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.510+0000 7f18d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T05:38:08.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.510+0000 7f18d4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.606+0000 7f18de486640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T05:38:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.606+0000 7f18de486640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T05:38:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.606+0000 7f18de486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.606+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f18b80008e0 con 0x7f18d80a13f0 2024-09-16T05:38:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18d4ff9640 1 -- 172.21.15.63:0/432663423 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v117) ==== 56+0+1449 (unknown 919893539 0 1261197053) 0x7f18d0019160 con 0x7f18d80a13f0 2024-09-16T05:38:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18d4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T05:38:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18d4ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 >> v1:172.21.15.63:6812/3643207211 conn(0x7f18c4036990 legacy=0x7f18c4038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18de486640 10 monclient: shutdown 2024-09-16T05:38:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18de486640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 >> v1:172.21.15.6:6790/0 conn(0x7f18d80a13f0 legacy=0x7f18d81d3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 shutdown_connections 2024-09-16T05:38:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 >> 172.21.15.63:0/432663423 conn(0x7f18d80a1b60 msgr2=0x7f18d8134ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 shutdown_connections 2024-09-16T05:38:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.610+0000 7f18de486640 1 -- 172.21.15.63:0/432663423 wait complete. 2024-09-16T05:38:08.630 INFO:tasks.workunit.client.0.smithi063.stdout: 14 155.00000 osd.14 down 1.00000 1.00000 2024-09-16T05:38:08.631 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm osd.12 2024-09-16T05:38:08.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient: get_monmap_and_config 2024-09-16T05:38:08.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient: build_initial_monmap 2024-09-16T05:38:08.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient: monmap: 2024-09-16T05:38:08.754 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:08.754 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:08.753577+0000 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:08.753577+0000 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 1 Processor -- start 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 1 -- start start 2024-09-16T05:38:08.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient: init 2024-09-16T05:38:08.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:08.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:08.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient(hunting): picked mon.noname-b con 0x7fbd54136ac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:08.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient(hunting): picked mon.noname-a con 0x7fbd5409fd80 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:08.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient(hunting): picked mon.noname-c con 0x7fbd5413d0b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:08.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd54072cb0 con 0x7fbd5409fd80 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd54136670 con 0x7fbd5413d0b0 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd540a14f0 con 0x7fbd54136ac0 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 10 monclient(hunting): authenticate will time out at 1145.530480s 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5abed640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbd5413d0b0 legacy=0x7fbd5413f4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5a3ec640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbd54136ac0 0x7fbd54136ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47932/0 (socket says 172.21.15.63:47932) 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5a3ec640 1 -- 172.21.15.63:0/164949933 learned_addr learned my addr 172.21.15.63:0/164949933 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd59beb640 1 --1- 172.21.15.63:0/164949933 >> v1:172.21.15.6:6789/0 conn(0x7fbd5409fd80 0x7fbd5413b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56208/0 (socket says 172.21.15.63:56208) 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1584853541 0 0) 0x7fbd540a14f0 con 0x7fbd54136ac0 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient(hunting): _negotiate 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:08.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient(hunting): my global_id is 5453 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd40003530 con 0x7fbd54136ac0 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 619737610 0 0) 0x7fbd44001120 con 0x7fbd5409fd80 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient(hunting): _negotiate 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient(hunting): my global_id is 5466 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd540a14f0 con 0x7fbd5409fd80 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 265748488 0 0) 0x7fbd40003530 con 0x7fbd54136ac0 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 >> v1:172.21.15.63:6789/0 conn(0x7fbd5413d0b0 legacy=0x7fbd5413f4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 >> v1:172.21.15.6:6789/0 conn(0x7fbd5409fd80 legacy=0x7fbd5413b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient: _finish_hunting 0 2024-09-16T05:38:08.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 monclient: found mon.noname-b 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd540a0630 con 0x7fbd54136ac0 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient: _finish_auth 0 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient: _check_auth_tickets 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd50001620 con 0x7fbd54136ac0 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 5 monclient: authenticate success, global_id 5453 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd5c650640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:08.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.750+0000 7fbd593ea640 10 monclient: dump: 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd50001940 con 0x7fbd54136ac0 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd593ea640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:08.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd593ea640 1 -- 172.21.15.63:0/164949933 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd50001d50 con 0x7fbd54136ac0 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 4 set_mon_vals no callback set 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd593ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd593ea640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd593ea640 10 monclient: dump: 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.767 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient: shutdown 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- 172.21.15.63:0/164949933 >> v1:172.21.15.6:6790/0 conn(0x7fbd54136ac0 legacy=0x7fbd54136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- 172.21.15.63:0/164949933 shutdown_connections 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- 172.21.15.63:0/164949933 >> 172.21.15.63:0/164949933 conn(0x7fbd540a1c90 msgr2=0x7fbd540a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- 172.21.15.63:0/164949933 shutdown_connections 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- 172.21.15.63:0/164949933 wait complete. 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient: build_initial_monmap 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient: monmap: 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:08.768 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 Processor -- start 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- start start 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient: init 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:08.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient(hunting): picked mon.noname-a con 0x7fbd5409fd80 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient(hunting): picked mon.noname-b con 0x7fbd54136ac0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient(hunting): picked mon.noname-c con 0x7fbd5413d0b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd5413c780 con 0x7fbd5409fd80 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd59beb640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbd54136ac0 legacy=0x7fbd541d0680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd541206b0 con 0x7fbd54136ac0 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd54120c70 con 0x7fbd5413d0b0 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5c650640 10 monclient(hunting): authenticate will time out at 1145.534530s 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5a3ec640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbd5409fd80 0x7fbd541350a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56220/0 (socket says 172.21.15.63:56220) 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd5a3ec640 1 -- 172.21.15.63:0/1582442510 learned_addr learned my addr 172.21.15.63:0/1582442510 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 919071027 0 0) 0x7fbd54120c70 con 0x7fbd5413d0b0 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:08.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): my global_id is 5456 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd38003530 con 0x7fbd5413d0b0 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189384716 0 0) 0x7fbd540a0630 con 0x7fbd5409fd80 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): my global_id is 5469 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd54120c70 con 0x7fbd5409fd80 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1565673252 0 0) 0x7fbd38003530 con 0x7fbd5413d0b0 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd540a0630 con 0x7fbd5413d0b0 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd4c004340 con 0x7fbd5413d0b0 2024-09-16T05:38:08.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): dump: 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:08.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2896715201 0 0) 0x7fbd54120c70 con 0x7fbd5409fd80 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd38003530 con 0x7fbd5409fd80 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd500015e0 con 0x7fbd5409fd80 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.754+0000 7fbd4affd640 10 monclient(hunting): dump: 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.773 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4130917861 0 0) 0x7fbd540a0630 con 0x7fbd5413d0b0 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 >> v1:172.21.15.63:6789/0 conn(0x7fbd54136ac0 legacy=0x7fbd541d0680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 >> v1:172.21.15.6:6789/0 conn(0x7fbd5409fd80 legacy=0x7fbd541350a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 monclient: found mon.c 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:08.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd541203e0 con 0x7fbd5413d0b0 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 10 monclient: _finish_auth 0 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 5 monclient: authenticate success, global_id 5456 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 10 monclient: _renew_subs 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd5411f540 con 0x7fbd5413d0b0 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 10 monclient: _renew_subs 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd5411e1a0 con 0x7fbd5413d0b0 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd4c0044e0 con 0x7fbd5413d0b0 2024-09-16T05:38:08.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd4c0048b0 con 0x7fbd5413d0b0 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4bfff640 4 set_mon_vals no callback set 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 10 monclient: dump: 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:08.777 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd3c005090 con 0x7fbd5413d0b0 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbd4c004ad0 con 0x7fbd5413d0b0 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(117..117 src has 1..117) ==== 5803+0+0 (unknown 70293949 0 0) 0x7fbd4c044b30 con 0x7fbd5413d0b0 2024-09-16T05:38:08.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.758+0000 7fbd5a3ec640 10 monclient: get_auth_request con 0x7fbd38036990 auth_method 0 2024-09-16T05:38:08.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.770+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbd4c015500 con 0x7fbd5413d0b0 2024-09-16T05:38:08.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.770+0000 7fbd4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:08.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.770+0000 7fbd4affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:08.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.938+0000 7fbd5c650640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T05:38:08.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.938+0000 7fbd5c650640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T05:38:08.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.938+0000 7fbd5c650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:08.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:08.938+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) -- 0x7fbd3c005380 con 0x7fbd5413d0b0 2024-09-16T05:38:09.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.438+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 removed item id 12 name 'osd.12' from crush map v118) ==== 125+0+0 (unknown 608744303 0 0) 0x7fbd4c014e00 con 0x7fbd5413d0b0 2024-09-16T05:38:09.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.438+0000 7fbd4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T05:38:09.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.438+0000 7fbd4affd640 10 monclient: _finish_command 2 = system:0 removed item id 12 name 'osd.12' from crush map 2024-09-16T05:38:09.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.542+0000 7fbd5c650640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T05:38:09.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.542+0000 7fbd5c650640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T05:38:09.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.542+0000 7fbd5c650640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:09.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.542+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) -- 0x7fbd3c0010a0 con 0x7fbd5413d0b0 2024-09-16T05:38:09.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.546+0000 7fbd4affd640 1 -- 172.21.15.63:0/1582442510 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 device 'osd.12' does not appear in the crush map v118) ==== 126+0+0 (unknown 227904750 0 0) 0x7fbd4c019150 con 0x7fbd5413d0b0 2024-09-16T05:38:09.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.546+0000 7fbd4affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T05:38:09.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.546+0000 7fbd4affd640 10 monclient: _finish_command 3 = system:0 device 'osd.12' does not appear in the crush map 2024-09-16T05:38:09.550 INFO:tasks.workunit.client.0.smithi063.stderr:device 'osd.12' does not appear in the crush map 2024-09-16T05:38:09.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.546+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbd38036990 legacy=0x7fbd38038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:09.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.546+0000 7fbd5c650640 10 monclient: shutdown 2024-09-16T05:38:09.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.546+0000 7fbd5c650640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:09.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.546+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 >> v1:172.21.15.6:6790/0 conn(0x7fbd5413d0b0 legacy=0x7fbd541d3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:09.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.550+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 shutdown_connections 2024-09-16T05:38:09.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.550+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 >> 172.21.15.63:0/1582442510 conn(0x7fbd540a1c90 msgr2=0x7fbd5409cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:09.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.550+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 shutdown_connections 2024-09-16T05:38:09.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.550+0000 7fbd5c650640 1 -- 172.21.15.63:0/1582442510 wait complete. 2024-09-16T05:38:09.567 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm osd.14 2024-09-16T05:38:09.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.686+0000 7fe7dca02640 10 monclient: get_monmap_and_config 2024-09-16T05:38:09.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.686+0000 7fe7dca02640 10 monclient: build_initial_monmap 2024-09-16T05:38:09.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.686+0000 7fe7dca02640 10 monclient: monmap: 2024-09-16T05:38:09.691 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:09.691 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:09.692 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:09.691035+0000 2024-09-16T05:38:09.692 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:09.691035+0000 2024-09-16T05:38:09.692 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:09.692 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:09.692 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:09.692 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:09.692 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:09.692 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:09.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.686+0000 7fe7dca02640 1 Processor -- start 2024-09-16T05:38:09.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- start start 2024-09-16T05:38:09.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient: init 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d80a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d80a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d80a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d80a4fd0 con 0x7fe7d80a4790 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d80a5b40 con 0x7fe7d80a08a0 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d80a1a70 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7cdd9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe7d80a08a0 legacy=0x7fe7d813beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient(hunting): authenticate will time out at 1146.468847s 2024-09-16T05:38:09.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe7d80a3ca0 0x7fe7d80a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47936/0 (socket says 172.21.15.63:47936) 2024-09-16T05:38:09.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d6d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe7d80a4790 0x7fe7d813f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56224/0 (socket says 172.21.15.63:56224) 2024-09-16T05:38:09.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d659c640 1 -- 172.21.15.63:0/1141521194 learned_addr learned my addr 172.21.15.63:0/1141521194 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:09.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568890868 0 0) 0x7fe7d80a4fd0 con 0x7fe7d80a4790 2024-09-16T05:38:09.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:38:09.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:09.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:09.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient(hunting): my global_id is 5472 2024-09-16T05:38:09.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7c4003530 con 0x7fe7d80a4790 2024-09-16T05:38:09.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1199391823 0 0) 0x7fe7d80a1a70 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient(hunting): my global_id is 5459 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7d80a4fd0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 361263188 0 0) 0x7fe7c4003530 con 0x7fe7d80a4790 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 >> v1:172.21.15.6:6790/0 conn(0x7fe7d80a3ca0 legacy=0x7fe7d80a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 >> v1:172.21.15.63:6789/0 conn(0x7fe7d80a08a0 legacy=0x7fe7d813beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 monclient: found mon.noname-a 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:09.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d80a53e0 con 0x7fe7d80a4790 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: _finish_auth 0 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe7d00019a0 con 0x7fe7d80a4790 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: dump: 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:09.699 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 5 monclient: authenticate success, global_id 5472 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe7d0001c50 con 0x7fe7d80a4790 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 1 -- 172.21.15.63:0/1141521194 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe7d0002060 con 0x7fe7d80a4790 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7d5d9b640 10 monclient: dump: 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:09.700 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 4 set_mon_vals no callback set 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 10 monclient: shutdown 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:09.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- 172.21.15.63:0/1141521194 >> v1:172.21.15.6:6789/0 conn(0x7fe7d80a4790 legacy=0x7fe7d813f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- 172.21.15.63:0/1141521194 shutdown_connections 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- 172.21.15.63:0/1141521194 >> 172.21.15.63:0/1141521194 conn(0x7fe7d80a67c0 msgr2=0x7fe7d8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- 172.21.15.63:0/1141521194 shutdown_connections 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.690+0000 7fe7dca02640 1 -- 172.21.15.63:0/1141521194 wait complete. 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: build_initial_monmap 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: monmap: 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:09.702 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:09.703 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:09.703 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:09.703 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:09.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 1 Processor -- start 2024-09-16T05:38:09.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 1 -- start start 2024-09-16T05:38:09.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: init 2024-09-16T05:38:09.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:09.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:09.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d80a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:09.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d80a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:09.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d80a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:09.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d8125bb0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d81230c0 con 0x7fe7d80a4790 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d80a0500 con 0x7fe7d80a08a0 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient(hunting): authenticate will time out at 1146.473026s 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7d6d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe7d80a4790 legacy=0x7fe7d81d3900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7d659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe7d80a08a0 0x7fe7d81371e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47948/0 (socket says 172.21.15.63:47948) 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7d659c640 1 -- 172.21.15.63:0/2782262153 learned_addr learned my addr 172.21.15.63:0/2782262153 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2424810524 0 0) 0x7fe7d80a0500 con 0x7fe7d80a08a0 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): my global_id is 5462 2024-09-16T05:38:09.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7b8003530 con 0x7fe7d80a08a0 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 740503446 0 0) 0x7fe7d8125bb0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): my global_id is 5475 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7d80a0500 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3067540199 0 0) 0x7fe7d80a0500 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7d8125bb0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3068731945 0 0) 0x7fe7b8003530 con 0x7fe7d80a08a0 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7d80a0500 con 0x7fe7d80a08a0 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe7c0003f80 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:09.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe7c80018d0 con 0x7fe7d80a08a0 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:09.710 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 984535729 0 0) 0x7fe7d8125bb0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 >> v1:172.21.15.6:6790/0 conn(0x7fe7d80a08a0 legacy=0x7fe7d81371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 >> v1:172.21.15.63:6789/0 conn(0x7fe7d80a4790 legacy=0x7fe7d81d3900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 monclient: found mon.a 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d81238c0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 5 monclient: authenticate success, global_id 5475 2024-09-16T05:38:09.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: _renew_subs 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d81258d0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: _renew_subs 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe7d81378f0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe7c00044a0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe7c00047f0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7d559a640 4 set_mon_vals no callback set 2024-09-16T05:38:09.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7cf7fe640 10 monclient: dump: 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.694+0000 7fe7dca02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.698+0000 7fe7dca02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.698+0000 7fe7dca02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:09.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.698+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7bc005090 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.698+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe7c0004a10 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.698+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(118..118 src has 1..118) ==== 5781+0+0 (unknown 2732662353 0 0) 0x7fe7c00447f0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.698+0000 7fe7d659c640 10 monclient: get_auth_request con 0x7fe7b8036a10 auth_method 0 2024-09-16T05:38:09.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.710+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe7c00151d0 con 0x7fe7d80a3ca0 2024-09-16T05:38:09.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.710+0000 7fe7cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:09.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.710+0000 7fe7cf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:09.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.878+0000 7fe7dca02640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T05:38:09.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.878+0000 7fe7dca02640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T05:38:09.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.878+0000 7fe7dca02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:09.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:09.878+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) -- 0x7fe7bc005380 con 0x7fe7d80a3ca0 2024-09-16T05:38:10.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.438+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 removed item id 14 name 'osd.14' from crush map v119) ==== 125+0+0 (unknown 948963238 0 0) 0x7fe7c0014ad0 con 0x7fe7d80a3ca0 2024-09-16T05:38:10.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.438+0000 7fe7cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T05:38:10.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.438+0000 7fe7cf7fe640 10 monclient: _finish_command 2 = system:0 removed item id 14 name 'osd.14' from crush map 2024-09-16T05:38:10.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.542+0000 7fe7dca02640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T05:38:10.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.542+0000 7fe7dca02640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T05:38:10.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.542+0000 7fe7dca02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:10.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.542+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) -- 0x7fe7bc0010a0 con 0x7fe7d80a3ca0 2024-09-16T05:38:10.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7cf7fe640 1 -- 172.21.15.63:0/2782262153 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 device 'osd.14' does not appear in the crush map v119) ==== 126+0+0 (unknown 1996926225 0 0) 0x7fe7c0018e20 con 0x7fe7d80a3ca0 2024-09-16T05:38:10.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7cf7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T05:38:10.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7cf7fe640 10 monclient: _finish_command 3 = system:0 device 'osd.14' does not appear in the crush map 2024-09-16T05:38:10.549 INFO:tasks.workunit.client.0.smithi063.stderr:device 'osd.14' does not appear in the crush map 2024-09-16T05:38:10.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe7b8036a10 legacy=0x7fe7b8038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:10.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7dca02640 10 monclient: shutdown 2024-09-16T05:38:10.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7dca02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:10.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 >> v1:172.21.15.6:6789/0 conn(0x7fe7d80a3ca0 legacy=0x7fe7d81d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:10.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 shutdown_connections 2024-09-16T05:38:10.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 >> 172.21.15.63:0/2782262153 conn(0x7fe7d80a67c0 msgr2=0x7fe7d80a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:10.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 shutdown_connections 2024-09-16T05:38:10.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.546+0000 7fe7dca02640 1 -- 172.21.15.63:0/2782262153 wait complete. 2024-09-16T05:38:10.565 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd rm osd.12 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient: get_monmap_and_config 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient: build_initial_monmap 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient: monmap: 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:10.684707+0000 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:10.684707+0000 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:10.685 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:10.686 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:10.686 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 1 Processor -- start 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 1 -- start start 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient: init 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient(hunting): picked mon.noname-a con 0x7f20bc09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient(hunting): picked mon.noname-b con 0x7f20bc09b070 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient(hunting): picked mon.noname-c con 0x7f20bc13a9a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:10.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20bc0722a0 con 0x7f20bc09ac90 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20bc0a0510 con 0x7f20bc13a9a0 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20bc09e660 con 0x7f20bc09b070 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient(hunting): authenticate will time out at 1147.461823s 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c2618640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f20bc13a9a0 legacy=0x7f20bc13cd90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c1616640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f20bc09b070 0x7f20bc09b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47960/0 (socket says 172.21.15.63:47960) 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c1616640 1 -- 172.21.15.63:0/3078621750 learned_addr learned my addr 172.21.15.63:0/3078621750 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c1e17640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f20bc09ac90 0x7f20bc09f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56252/0 (socket says 172.21.15.63:56252) 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370533829 0 0) 0x7f20bc0722a0 con 0x7f20bc09ac90 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient(hunting): _negotiate 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:10.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient(hunting): my global_id is 5478 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20a8003530 con 0x7f20bc09ac90 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1509411599 0 0) 0x7f20bc09e660 con 0x7f20bc09b070 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient(hunting): _negotiate 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient(hunting): my global_id is 5465 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20bc0722a0 con 0x7f20bc09b070 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2828244593 0 0) 0x7f20a8003530 con 0x7f20bc09ac90 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 >> v1:172.21.15.6:6790/0 conn(0x7f20bc09b070 legacy=0x7f20bc09b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 >> v1:172.21.15.63:6789/0 conn(0x7f20bc13a9a0 legacy=0x7f20bc13cd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:10.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: _finish_hunting 0 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 monclient: found mon.noname-a 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc1414e0 con 0x7f20bc09ac90 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: _finish_auth 0 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: _check_auth_tickets 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20b8001620 con 0x7f20bc09ac90 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:10.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: dump: 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 5 monclient: authenticate success, global_id 5478 2024-09-16T05:38:10.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20b80018d0 con 0x7f20bc09ac90 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 1 -- 172.21.15.63:0/3078621750 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20b8001ca0 con 0x7f20bc09ac90 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c0e15640 10 monclient: dump: 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:10.694 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.682+0000 7f20c407b640 4 set_mon_vals no callback set 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient: shutdown 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- 172.21.15.63:0/3078621750 >> v1:172.21.15.6:6789/0 conn(0x7f20bc09ac90 legacy=0x7f20bc09f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- 172.21.15.63:0/3078621750 shutdown_connections 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- 172.21.15.63:0/3078621750 >> 172.21.15.63:0/3078621750 conn(0x7f20bc0a1bf0 msgr2=0x7f20bc0a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- 172.21.15.63:0/3078621750 shutdown_connections 2024-09-16T05:38:10.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- 172.21.15.63:0/3078621750 wait complete. 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient: build_initial_monmap 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient: monmap: 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:10.700 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 Processor -- start 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- start start 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient: init 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient(hunting): picked mon.noname-c con 0x7f20bc09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient(hunting): picked mon.noname-b con 0x7f20bc09b070 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient(hunting): picked mon.noname-a con 0x7f20bc13a9a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20bc137870 con 0x7f20bc13a9a0 2024-09-16T05:38:10.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20bc136790 con 0x7f20bc09b070 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20bc137af0 con 0x7f20bc09ac90 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c407b640 10 monclient(hunting): authenticate will time out at 1147.466097s 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c1616640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f20bc09b070 legacy=0x7f20bc136080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c1e17640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f20bc09ac90 0x7f20bc135970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47970/0 (socket says 172.21.15.63:47970) 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c1e17640 1 -- 172.21.15.63:0/2176965368 learned_addr learned my addr 172.21.15.63:0/2176965368 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20c2618640 1 --1- 172.21.15.63:0/2176965368 >> v1:172.21.15.6:6789/0 conn(0x7f20bc13a9a0 0x7f20bc1d1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56262/0 (socket says 172.21.15.63:56262) 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614032182 0 0) 0x7f20bc137af0 con 0x7f20bc09ac90 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): my global_id is 5468 2024-09-16T05:38:10.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20a0003530 con 0x7f20bc09ac90 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887762259 0 0) 0x7f20bc137870 con 0x7f20bc13a9a0 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): my global_id is 5481 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20bc137af0 con 0x7f20bc13a9a0 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1515282727 0 0) 0x7f20a0003530 con 0x7f20bc09ac90 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20bc137870 con 0x7f20bc09ac90 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 375757215 0 0) 0x7f20bc137af0 con 0x7f20bc13a9a0 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20a0003530 con 0x7f20bc13a9a0 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20b8001c20 con 0x7f20bc09ac90 2024-09-16T05:38:10.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): dump: 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:10.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:10.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20b4004340 con 0x7f20bc13a9a0 2024-09-16T05:38:10.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:10.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:10.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.686+0000 7f20b27fc640 10 monclient(hunting): dump: 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2000320369 0 0) 0x7f20a0003530 con 0x7f20bc13a9a0 2024-09-16T05:38:10.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 >> v1:172.21.15.6:6790/0 conn(0x7f20bc09ac90 legacy=0x7f20bc135970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 >> v1:172.21.15.63:6789/0 conn(0x7f20bc09b070 legacy=0x7f20bc136080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 monclient: found mon.a 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc138fa0 con 0x7f20bc13a9a0 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 5 monclient: authenticate success, global_id 5481 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 10 monclient: _renew_subs 2024-09-16T05:38:10.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20bc13e170 con 0x7f20bc13a9a0 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 10 monclient: _renew_subs 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f20bc138cc0 con 0x7f20bc13a9a0 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20b4004590 con 0x7f20bc13a9a0 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b3fff640 4 set_mon_vals no callback set 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20b4004960 con 0x7f20bc13a9a0 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 10 monclient: dump: 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:10.711 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a4005090 con 0x7f20bc13a9a0 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f20b4004b80 con 0x7f20bc13a9a0 2024-09-16T05:38:10.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(119..119 src has 1..119) ==== 5759+0+0 (unknown 860045538 0 0) 0x7f20b4044b20 con 0x7f20bc13a9a0 2024-09-16T05:38:10.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.690+0000 7f20c1e17640 10 monclient: get_auth_request con 0x7f20a00369a0 auth_method 0 2024-09-16T05:38:10.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.702+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f20b4015580 con 0x7f20bc13a9a0 2024-09-16T05:38:10.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.702+0000 7f20b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:10.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.702+0000 7f20b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:10.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.874+0000 7f20c407b640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T05:38:10.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.874+0000 7f20c407b640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T05:38:10.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.878+0000 7f20c407b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:10.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:10.878+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) -- 0x7f20a4005380 con 0x7f20bc13a9a0 2024-09-16T05:38:10.881 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:10.877+0000 7f3419a35640 -1 mon.a@0(leader).osd e119 definitely_dead 0 2024-09-16T05:38:11.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.450+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 removed osd.12 v120) ==== 87+0+0 (unknown 2809692732 0 0) 0x7f20b4014e80 con 0x7f20bc13a9a0 2024-09-16T05:38:11.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.450+0000 7f20b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T05:38:11.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.450+0000 7f20b27fc640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-16T05:38:11.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.562+0000 7f20c407b640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T05:38:11.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.562+0000 7f20c407b640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T05:38:11.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.562+0000 7f20c407b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:11.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.562+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) -- 0x7f20a4005ac0 con 0x7f20bc13a9a0 2024-09-16T05:38:11.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:11.565+0000 7f3419a35640 -1 mon.a@0(leader).osd e120 definitely_dead 0 2024-09-16T05:38:11.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20b27fc640 1 -- 172.21.15.63:0/2176965368 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 osd.12 does not exist. v120) ==== 96+0+0 (unknown 1584559775 0 0) 0x7f20b401da30 con 0x7f20bc13a9a0 2024-09-16T05:38:11.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20b27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T05:38:11.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20b27fc640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T05:38:11.569 INFO:tasks.workunit.client.0.smithi063.stderr:osd.12 does not exist. 2024-09-16T05:38:11.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 >> v1:172.21.15.63:6812/3643207211 conn(0x7f20a00369a0 legacy=0x7f20a0038e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:11.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20c407b640 10 monclient: shutdown 2024-09-16T05:38:11.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20c407b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:11.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 >> v1:172.21.15.6:6789/0 conn(0x7f20bc13a9a0 legacy=0x7f20bc1d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:11.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 shutdown_connections 2024-09-16T05:38:11.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 >> 172.21.15.63:0/2176965368 conn(0x7f20bc0a1bf0 msgr2=0x7f20bc075e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:11.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 shutdown_connections 2024-09-16T05:38:11.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.566+0000 7f20c407b640 1 -- 172.21.15.63:0/2176965368 wait complete. 2024-09-16T05:38:11.586 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd rm osd.14 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient: get_monmap_and_config 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient: build_initial_monmap 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient: monmap: 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:11.705047+0000 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:11.705047+0000 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:11.709 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 1 Processor -- start 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 1 -- start start 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient: init 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient(hunting): picked mon.noname-c con 0x7f627809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient(hunting): picked mon.noname-b con 0x7f627809b070 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient(hunting): picked mon.noname-a con 0x7f627813a9a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62780722a0 con 0x7f627813a9a0 2024-09-16T05:38:11.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62780a0510 con 0x7f627809ac90 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627809e660 con 0x7f627809b070 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient(hunting): authenticate will time out at 1148.481981s 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f627809ac90 legacy=0x7f627809f6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f6275d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f627809b070 0x7f627809b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47982/0 (socket says 172.21.15.63:47982) 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f6275d9b640 1 -- 172.21.15.63:0/3194595859 learned_addr learned my addr 172.21.15.63:0/3194595859 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2695532807 0 0) 0x7f627809e660 con 0x7f627809b070 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient(hunting): my global_id is 5471 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6260003530 con 0x7f627809b070 2024-09-16T05:38:11.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 698849103 0 0) 0x7f62780722a0 con 0x7f627813a9a0 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient(hunting): my global_id is 5484 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f627809e660 con 0x7f627813a9a0 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4016257682 0 0) 0x7f6260003530 con 0x7f627809b070 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 >> v1:172.21.15.63:6789/0 conn(0x7f627809ac90 legacy=0x7f627809f6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 >> v1:172.21.15.6:6789/0 conn(0x7f627813a9a0 legacy=0x7f627813cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 monclient: found mon.noname-b 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:11.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62781414e0 con 0x7f627809b070 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: _finish_auth 0 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f626c001620 con 0x7f627809b070 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 5 monclient: authenticate success, global_id 5471 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: dump: 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f626c0018d0 con 0x7f627809b070 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 1 -- 172.21.15.63:0/3194595859 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f626c001ce0 con 0x7f627809b070 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627cf89640 4 set_mon_vals no callback set 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:11.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.702+0000 7f627559a640 10 monclient: dump: 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: shutdown 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:11.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- 172.21.15.63:0/3194595859 >> v1:172.21.15.6:6790/0 conn(0x7f627809b070 legacy=0x7f627809b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- 172.21.15.63:0/3194595859 shutdown_connections 2024-09-16T05:38:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- 172.21.15.63:0/3194595859 >> 172.21.15.63:0/3194595859 conn(0x7f62780a1bf0 msgr2=0x7f62780a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- 172.21.15.63:0/3194595859 shutdown_connections 2024-09-16T05:38:11.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- 172.21.15.63:0/3194595859 wait complete. 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: build_initial_monmap 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: monmap: 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:11.719 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 Processor -- start 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- start start 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: init 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient(hunting): picked mon.noname-c con 0x7f627809ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient(hunting): picked mon.noname-a con 0x7f627809b070 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient(hunting): picked mon.noname-b con 0x7f627813a9a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:11.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627813d330 con 0x7f627809b070 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62780714c0 con 0x7f627813a9a0 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627809db70 con 0x7f627809ac90 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient(hunting): authenticate will time out at 1148.485600s 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6276d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f627813a9a0 legacy=0x7f62781cf420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f627809ac90 0x7f62780706a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47998/0 (socket says 172.21.15.63:47998) 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627659c640 1 -- 172.21.15.63:0/3793660256 learned_addr learned my addr 172.21.15.63:0/3793660256 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1723065661 0 0) 0x7f627809db70 con 0x7f627809ac90 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:11.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): my global_id is 5474 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6258003530 con 0x7f627809ac90 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370344132 0 0) 0x7f627813d330 con 0x7f627809b070 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): my global_id is 5487 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f627809db70 con 0x7f627809b070 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2336897643 0 0) 0x7f6258003530 con 0x7f627809ac90 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f627813d330 con 0x7f627809ac90 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62700044e0 con 0x7f627809ac90 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:11.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:11.723 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3121656340 0 0) 0x7f627809db70 con 0x7f627809b070 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6258003530 con 0x7f627809b070 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f626c001a80 con 0x7f627809b070 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:11.724 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1423688316 0 0) 0x7f627813d330 con 0x7f627809ac90 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 >> v1:172.21.15.63:6789/0 conn(0x7f627813a9a0 legacy=0x7f62781cf420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 >> v1:172.21.15.6:6789/0 conn(0x7f627809b070 legacy=0x7f6278070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 monclient: found mon.c 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6278140f70 con 0x7f627809ac90 2024-09-16T05:38:11.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 5 monclient: authenticate success, global_id 5474 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: _renew_subs 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6278072f10 con 0x7f627809ac90 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: _renew_subs 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f627813b920 con 0x7f627809ac90 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62700046f0 con 0x7f627809ac90 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6274d99640 4 set_mon_vals no callback set 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6270004ac0 con 0x7f627809ac90 2024-09-16T05:38:11.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.706+0000 7f6266ffd640 10 monclient: dump: 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:11.727 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.710+0000 7f627cf89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.710+0000 7f627cf89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.710+0000 7f627cf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.710+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f625c005090 con 0x7f627809ac90 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.710+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6270004ce0 con 0x7f627809ac90 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.710+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(120..120 src has 1..120) ==== 5759+0+0 (unknown 500677941 0 0) 0x7f6270044dc0 con 0x7f627809ac90 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.710+0000 7f6275d9b640 10 monclient: get_auth_request con 0x7f62580369e0 auth_method 0 2024-09-16T05:38:11.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.722+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6270015820 con 0x7f627809ac90 2024-09-16T05:38:11.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.722+0000 7f6266ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:11.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.722+0000 7f6266ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:11.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.894+0000 7f627cf89640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T05:38:11.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.894+0000 7f627cf89640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T05:38:11.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.894+0000 7f627cf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:11.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:11.894+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) -- 0x7f625c005380 con 0x7f627809ac90 2024-09-16T05:38:11.899 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:11.897+0000 7f3419a35640 -1 mon.a@0(leader).osd e120 definitely_dead 0 2024-09-16T05:38:12.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.470+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 removed osd.14 v121) ==== 87+0+0 (unknown 662752641 0 0) 0x7f6270015120 con 0x7f627809ac90 2024-09-16T05:38:12.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.470+0000 7f6266ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T05:38:12.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.470+0000 7f6266ffd640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-16T05:38:12.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.574+0000 7f627cf89640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T05:38:12.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.574+0000 7f627cf89640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T05:38:12.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.574+0000 7f627cf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:12.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.574+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) -- 0x7f625c005ac0 con 0x7f627809ac90 2024-09-16T05:38:12.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:12.577+0000 7f3419a35640 -1 mon.a@0(leader).osd e121 definitely_dead 0 2024-09-16T05:38:12.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.578+0000 7f6266ffd640 1 -- 172.21.15.63:0/3793660256 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 osd.14 does not exist. v121) ==== 96+0+0 (unknown 315867704 0 0) 0x7f627001dcd0 con 0x7f627809ac90 2024-09-16T05:38:12.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.578+0000 7f6266ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T05:38:12.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.578+0000 7f6266ffd640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-16T05:38:12.582 INFO:tasks.workunit.client.0.smithi063.stderr:osd.14 does not exist. 2024-09-16T05:38:12.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.578+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 >> v1:172.21.15.63:6812/3643207211 conn(0x7f62580369e0 legacy=0x7f6258038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:12.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.578+0000 7f627cf89640 10 monclient: shutdown 2024-09-16T05:38:12.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.578+0000 7f627cf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:12.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.578+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 >> v1:172.21.15.6:6790/0 conn(0x7f627809ac90 legacy=0x7f62780706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:12.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.582+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 shutdown_connections 2024-09-16T05:38:12.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.582+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 >> 172.21.15.63:0/3793660256 conn(0x7f62780a1bf0 msgr2=0x7f62780a4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:12.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.582+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 shutdown_connections 2024-09-16T05:38:12.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.582+0000 7f627cf89640 1 -- 172.21.15.63:0/3793660256 wait complete. 2024-09-16T05:38:12.599 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd set-require-min-compat-client luminous 2024-09-16T05:38:12.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.714+0000 7ff56bf2e640 10 monclient: get_monmap_and_config 2024-09-16T05:38:12.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.714+0000 7ff56bf2e640 10 monclient: build_initial_monmap 2024-09-16T05:38:12.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.714+0000 7ff56bf2e640 10 monclient: monmap: 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:12.720204+0000 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:12.720204+0000 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:12.721 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 Processor -- start 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- start start 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient: init 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff56409ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff56409b070 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff56413a9a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5640722a0 con 0x7ff56413a9a0 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5640a0510 con 0x7ff56409ac90 2024-09-16T05:38:12.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff56409e660 con 0x7ff56409b070 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient(hunting): authenticate will time out at 1149.497227s 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff569cca640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff56409ac90 legacy=0x7ff56409f6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56a4cb640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff56413a9a0 0x7ff56413cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56274/0 (socket says 172.21.15.63:56274) 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56a4cb640 1 -- 172.21.15.63:0/2902541745 learned_addr learned my addr 172.21.15.63:0/2902541745 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff5694c9640 1 --1- 172.21.15.63:0/2902541745 >> v1:172.21.15.6:6790/0 conn(0x7ff56409b070 0x7ff56409b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48010/0 (socket says 172.21.15.63:48010) 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1799441619 0 0) 0x7ff5640722a0 con 0x7ff56413a9a0 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient(hunting): _negotiate 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient(hunting): my global_id is 5490 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff550003530 con 0x7ff56413a9a0 2024-09-16T05:38:12.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264514773 0 0) 0x7ff56409e660 con 0x7ff56409b070 2024-09-16T05:38:12.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient(hunting): _negotiate 2024-09-16T05:38:12.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:12.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:12.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient(hunting): my global_id is 5477 2024-09-16T05:38:12.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5640722a0 con 0x7ff56409b070 2024-09-16T05:38:12.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1974197028 0 0) 0x7ff5640722a0 con 0x7ff56409b070 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 >> v1:172.21.15.63:6789/0 conn(0x7ff56409ac90 legacy=0x7ff56409f6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 >> v1:172.21.15.6:6789/0 conn(0x7ff56413a9a0 legacy=0x7ff56413cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: _finish_hunting 0 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 monclient: found mon.noname-b 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5641414e0 con 0x7ff56409b070 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: _finish_auth 0 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: _check_auth_tickets 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:12.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff554001580 con 0x7ff56409b070 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 5 monclient: authenticate success, global_id 5477 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: dump: 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:12.732 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5540017e0 con 0x7ff56409b070 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 1 -- 172.21.15.63:0/2902541745 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff554001bb0 con 0x7ff56409b070 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 4 set_mon_vals no callback set 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:12.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff568cc8640 10 monclient: dump: 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:12.734 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 10 monclient: shutdown 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- 172.21.15.63:0/2902541745 >> v1:172.21.15.6:6790/0 conn(0x7ff56409b070 legacy=0x7ff56409b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- 172.21.15.63:0/2902541745 shutdown_connections 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- 172.21.15.63:0/2902541745 >> 172.21.15.63:0/2902541745 conn(0x7ff5640a1bf0 msgr2=0x7ff5640a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- 172.21.15.63:0/2902541745 shutdown_connections 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.718+0000 7ff56bf2e640 1 -- 172.21.15.63:0/2902541745 wait complete. 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: build_initial_monmap 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: monmap: 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:12.735 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 1 Processor -- start 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 1 -- start start 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: init 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:12.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff56409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff56409b070 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff56413a9a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff56413ddb0 con 0x7ff56413a9a0 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff56411b800 con 0x7ff56409b070 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5641cc230 con 0x7ff56409ac90 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient(hunting): authenticate will time out at 1149.501123s 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff5694c9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff56409b070 legacy=0x7ff5641cbb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56a4cb640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff56413a9a0 0x7ff5641d3670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:56288/0 (socket says 172.21.15.63:56288) 2024-09-16T05:38:12.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff569cca640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff56409ac90 0x7ff5641cb410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48012/0 (socket says 172.21.15.63:48012) 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56a4cb640 1 -- 172.21.15.63:0/3944464612 learned_addr learned my addr 172.21.15.63:0/3944464612 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 410440576 0 0) 0x7ff5641cc230 con 0x7ff56409ac90 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): my global_id is 5480 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff548003530 con 0x7ff56409ac90 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3582433280 0 0) 0x7ff55c001650 con 0x7ff56413a9a0 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): my global_id is 5493 2024-09-16T05:38:12.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5641cc230 con 0x7ff56413a9a0 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3597035698 0 0) 0x7ff548003530 con 0x7ff56409ac90 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff55c001650 con 0x7ff56409ac90 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff5600040d0 con 0x7ff56409ac90 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:12.739 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3198763441 0 0) 0x7ff5641cc230 con 0x7ff56413a9a0 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff548003530 con 0x7ff56413a9a0 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff55c0018a0 con 0x7ff56413a9a0 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:12.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2689496231 0 0) 0x7ff55c001650 con 0x7ff56409ac90 2024-09-16T05:38:12.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 >> v1:172.21.15.63:6789/0 conn(0x7ff56409b070 legacy=0x7ff5641cbb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 >> v1:172.21.15.6:6789/0 conn(0x7ff56413a9a0 legacy=0x7ff5641d3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 monclient: found mon.c 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56413a2a0 con 0x7ff56409ac90 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 5 monclient: authenticate success, global_id 5480 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: _renew_subs 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:12.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff564072f10 con 0x7ff56409ac90 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: _renew_subs 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5640709e0 con 0x7ff56409ac90 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff560004330 con 0x7ff56409ac90 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff560004700 con 0x7ff56409ac90 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55bfff640 4 set_mon_vals no callback set 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff55a7fc640 10 monclient: dump: 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:12.743 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.722+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff54c005090 con 0x7ff56409ac90 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.726+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff5600048a0 con 0x7ff56409ac90 2024-09-16T05:38:12.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.726+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(121..121 src has 1..121) ==== 5759+0+0 (unknown 692468991 0 0) 0x7ff560044830 con 0x7ff56409ac90 2024-09-16T05:38:12.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.726+0000 7ff5694c9640 10 monclient: get_auth_request con 0x7ff548036aa0 auth_method 0 2024-09-16T05:38:12.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.738+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff560015290 con 0x7ff56409ac90 2024-09-16T05:38:12.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.738+0000 7ff55a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:12.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.738+0000 7ff55a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:12.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.910+0000 7ff56bf2e640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T05:38:12.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.910+0000 7ff56bf2e640 10 monclient: _send_command 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T05:38:12.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.910+0000 7ff56bf2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:12.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:12.910+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) -- 0x7ff54c005b90 con 0x7ff56409ac90 2024-09-16T05:38:13.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:13.482+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v122) ==== 145+0+0 (unknown 3106344471 0 0) 0x7ff560014b90 con 0x7ff56409ac90 2024-09-16T05:38:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:13.482+0000 7ff55a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T05:38:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:13.482+0000 7ff55a7fc640 10 monclient: _finish_command 2 = system:0 set require_min_compat_client to luminous 2024-09-16T05:38:13.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:13.590+0000 7ff56bf2e640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T05:38:13.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:13.590+0000 7ff56bf2e640 10 monclient: _send_command 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T05:38:13.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:13.590+0000 7ff56bf2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:13.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:13.590+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) -- 0x7ff54c0010a0 con 0x7ff56409ac90 2024-09-16T05:38:14.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.486+0000 7ff55a7fc640 1 -- 172.21.15.63:0/3944464612 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v123) ==== 145+0+0 (unknown 1929108452 0 0) 0x7ff56001d740 con 0x7ff56409ac90 2024-09-16T05:38:14.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.486+0000 7ff55a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T05:38:14.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.486+0000 7ff55a7fc640 10 monclient: _finish_command 3 = system:0 set require_min_compat_client to luminous 2024-09-16T05:38:14.493 INFO:tasks.workunit.client.0.smithi063.stderr:set require_min_compat_client to luminous 2024-09-16T05:38:14.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.490+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff548036aa0 legacy=0x7ff548038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:14.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.490+0000 7ff56bf2e640 10 monclient: shutdown 2024-09-16T05:38:14.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.490+0000 7ff56bf2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:14.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.490+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 >> v1:172.21.15.6:6790/0 conn(0x7ff56409ac90 legacy=0x7ff5641cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:14.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.490+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 shutdown_connections 2024-09-16T05:38:14.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.490+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 >> 172.21.15.63:0/3944464612 conn(0x7ff5640a1bf0 msgr2=0x7ff564140c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:14.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.490+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 shutdown_connections 2024-09-16T05:38:14.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.490+0000 7ff56bf2e640 1 -- 172.21.15.63:0/3944464612 wait complete. 2024-09-16T05:38:14.519 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set dump 2024-09-16T05:38:14.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient: get_monmap_and_config 2024-09-16T05:38:14.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient: build_initial_monmap 2024-09-16T05:38:14.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient: monmap: 2024-09-16T05:38:14.646 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:14.646 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:14.646 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:14.645924+0000 2024-09-16T05:38:14.647 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:14.645924+0000 2024-09-16T05:38:14.647 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:14.647 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:14.647 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:14.647 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:14.647 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:14.647 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 1 Processor -- start 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 1 -- start start 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient: init 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient(hunting): picked mon.noname-b con 0x7f4d7c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient(hunting): picked mon.noname-a con 0x7f4d7c06dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient(hunting): picked mon.noname-c con 0x7f4d7c0a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:14.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7c09f800 con 0x7f4d7c06dd30 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7c0a0ab0 con 0x7f4d7c0a0210 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7c0a0d90 con 0x7f4d7c09ac90 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d82061640 10 monclient(hunting): authenticate will time out at 1151.423350s 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d7bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4d7c0a0210 legacy=0x7f4d7c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d7affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4d7c06dd30 0x7f4d7c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48074/0 (socket says 172.21.15.63:48074) 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.642+0000 7f4d7affd640 1 -- 172.21.15.63:0/2209757971 learned_addr learned my addr 172.21.15.63:0/2209757971 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3694545027 0 0) 0x7f4d7c09f800 con 0x7f4d7c06dd30 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:14.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient(hunting): my global_id is 5496 2024-09-16T05:38:14.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d64003530 con 0x7f4d7c06dd30 2024-09-16T05:38:14.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 510273645 0 0) 0x7f4d7c0a0d90 con 0x7f4d7c09ac90 2024-09-16T05:38:14.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:14.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:14.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:14.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient(hunting): my global_id is 5483 2024-09-16T05:38:14.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d7c09f800 con 0x7f4d7c09ac90 2024-09-16T05:38:14.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3188787886 0 0) 0x7f4d64003530 con 0x7f4d7c06dd30 2024-09-16T05:38:14.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 >> v1:172.21.15.6:6790/0 conn(0x7f4d7c09ac90 legacy=0x7f4d7c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:14.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 >> v1:172.21.15.63:6789/0 conn(0x7f4d7c0a0210 legacy=0x7f4d7c13d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:14.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 monclient: found mon.noname-a 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c09e770 con 0x7f4d7c06dd30 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d70001620 con 0x7f4d7c06dd30 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: dump: 2024-09-16T05:38:14.658 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 5 monclient: authenticate success, global_id 5496 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:14.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d70001880 con 0x7f4d7c06dd30 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 1 -- 172.21.15.63:0/2209757971 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d70001c90 con 0x7f4d7c06dd30 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7a7fc640 10 monclient: dump: 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:14.660 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 4 set_mon_vals no callback set 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient: shutdown 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- 172.21.15.63:0/2209757971 >> v1:172.21.15.6:6789/0 conn(0x7f4d7c06dd30 legacy=0x7f4d7c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- 172.21.15.63:0/2209757971 shutdown_connections 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- 172.21.15.63:0/2209757971 >> 172.21.15.63:0/2209757971 conn(0x7f4d7c0a1c90 msgr2=0x7f4d7c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- 172.21.15.63:0/2209757971 shutdown_connections 2024-09-16T05:38:14.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- 172.21.15.63:0/2209757971 wait complete. 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient: build_initial_monmap 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient: monmap: 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:14.662 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 Processor -- start 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- start start 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient: init 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient(hunting): picked mon.noname-c con 0x7f4d7c06dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient(hunting): picked mon.noname-a con 0x7f4d7c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient(hunting): picked mon.noname-b con 0x7f4d7c0a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7c06e160 con 0x7f4d7c09ac90 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7c06e6b0 con 0x7f4d7c0a0210 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7c123310 con 0x7f4d7c06dd30 2024-09-16T05:38:14.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d82061640 10 monclient(hunting): authenticate will time out at 1151.427485s 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4d7c0a0210 legacy=0x7f4d7c1d1710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4d7c09ac90 0x7f4d7c1cdfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48076/0 (socket says 172.21.15.63:48076) 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7affd640 1 -- 172.21.15.63:0/3929507854 learned_addr learned my addr 172.21.15.63:0/3929507854 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.646+0000 7f4d7b7fe640 1 --1- 172.21.15.63:0/3929507854 >> v1:172.21.15.6:6790/0 conn(0x7f4d7c06dd30 0x7f4d7c134c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40762/0 (socket says 172.21.15.63:40762) 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1851306541 0 0) 0x7f4d7c06e160 con 0x7f4d7c09ac90 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): my global_id is 5499 2024-09-16T05:38:14.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d68003530 con 0x7f4d7c09ac90 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1937080992 0 0) 0x7f4d7c123310 con 0x7f4d7c06dd30 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): my global_id is 5486 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d7c06e160 con 0x7f4d7c06dd30 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 131750624 0 0) 0x7f4d68003530 con 0x7f4d7c09ac90 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d7c123310 con 0x7f4d7c09ac90 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d70001c50 con 0x7f4d7c09ac90 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:14.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): dump: 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 479896209 0 0) 0x7f4d7c06e160 con 0x7f4d7c06dd30 2024-09-16T05:38:14.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d68003530 con 0x7f4d7c06dd30 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d6c0019a0 con 0x7f4d7c06dd30 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient(hunting): dump: 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.667 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:14.668 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:14.668 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:14.668 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:14.668 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:14.668 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:14.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1638551460 0 0) 0x7f4d7c123310 con 0x7f4d7c09ac90 2024-09-16T05:38:14.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 >> v1:172.21.15.6:6790/0 conn(0x7f4d7c06dd30 legacy=0x7f4d7c134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:14.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 >> v1:172.21.15.63:6789/0 conn(0x7f4d7c0a0210 legacy=0x7f4d7c1d1710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 monclient: found mon.a 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c125320 con 0x7f4d7c09ac90 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient: _finish_auth 0 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 5 monclient: authenticate success, global_id 5499 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 10 monclient: _renew_subs 2024-09-16T05:38:14.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:14.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c125600 con 0x7f4d7c09ac90 2024-09-16T05:38:14.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 10 monclient: _renew_subs 2024-09-16T05:38:14.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:14.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4d7c135320 con 0x7f4d7c09ac90 2024-09-16T05:38:14.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d70001880 con 0x7f4d7c09ac90 2024-09-16T05:38:14.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d70004c40 con 0x7f4d7c09ac90 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d79ffb640 4 set_mon_vals no callback set 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 10 monclient: dump: 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:14.671 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d5c005090 con 0x7f4d7c09ac90 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4d70004ea0 con 0x7f4d7c09ac90 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(123..123 src has 1..123) ==== 5759+0+0 (unknown 1080377195 0 0) 0x7f4d700448f0 con 0x7f4d7c09ac90 2024-09-16T05:38:14.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.650+0000 7f4d7b7fe640 10 monclient: get_auth_request con 0x7f4d68036990 auth_method 0 2024-09-16T05:38:14.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.662+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4d700153b0 con 0x7f4d7c09ac90 2024-09-16T05:38:14.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.662+0000 7f4d63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:14.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.662+0000 7f4d63fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.834+0000 7f4d82061640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.834+0000 7f4d82061640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.834+0000 7f4d82061640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:14.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.834+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) -- 0x7f4d5c005380 con 0x7f4d7c09ac90 2024-09-16T05:38:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.834+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v123) ==== 73+0+3 (unknown 2894706596 0 2423995997) 0x7f4d7001d800 con 0x7f4d7c09ac90 2024-09-16T05:38:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.834+0000 7f4d63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:14.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.834+0000 7f4d63fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:14.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:14.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:14.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:14.945 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) -- 0x7f4d5c005ac0 con 0x7f4d7c09ac90 2024-09-16T05:38:14.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d63fff640 1 -- 172.21.15.63:0/3929507854 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v123) ==== 73+0+3 (unknown 2894706596 0 2423995997) 0x7f4d70040380 con 0x7f4d7c09ac90 2024-09-16T05:38:14.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d63fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:14.946 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d63fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:14.946 INFO:tasks.workunit.client.0.smithi063.stdout:{} 2024-09-16T05:38:14.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4d68036990 legacy=0x7f4d68038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:14.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 10 monclient: shutdown 2024-09-16T05:38:14.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:14.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 >> v1:172.21.15.6:6789/0 conn(0x7f4d7c09ac90 legacy=0x7f4d7c1cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:14.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 shutdown_connections 2024-09-16T05:38:14.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 >> 172.21.15.63:0/3929507854 conn(0x7f4d7c0a1c90 msgr2=0x7f4d7c11b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:14.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 shutdown_connections 2024-09-16T05:38:14.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:14.942+0000 7f4d82061640 1 -- 172.21.15.63:0/3929507854 wait complete. 2024-09-16T05:38:14.963 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set ls 2024-09-16T05:38:15.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient: get_monmap_and_config 2024-09-16T05:38:15.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient: build_initial_monmap 2024-09-16T05:38:15.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient: monmap: 2024-09-16T05:38:15.078 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:15.078 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.078 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:15.077548+0000 2024-09-16T05:38:15.078 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:15.077548+0000 2024-09-16T05:38:15.079 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:15.079 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.079 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:15.079 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:15.079 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:15.079 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 1 Processor -- start 2024-09-16T05:38:15.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 1 -- start start 2024-09-16T05:38:15.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient: init 2024-09-16T05:38:15.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:15.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:15.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient(hunting): picked mon.noname-b con 0x7f9b780a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:15.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7809b450 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:15.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7809b8a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:15.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b7809e970 con 0x7f9b7809b450 2024-09-16T05:38:15.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b7809c370 con 0x7f9b7809b8a0 2024-09-16T05:38:15.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b7809d620 con 0x7f9b780a08a0 2024-09-16T05:38:15.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:15.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b7d55e640 10 monclient(hunting): authenticate will time out at 1151.855390s 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b777fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9b7809b8a0 legacy=0x7f9b7813c0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b767fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9b7809b450 0x7f9b781389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48078/0 (socket says 172.21.15.63:48078) 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b76ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9b780a08a0 0x7f9b7809ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40772/0 (socket says 172.21.15.63:40772) 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.074+0000 7f9b767fc640 1 -- 172.21.15.63:0/3362415940 learned_addr learned my addr 172.21.15.63:0/3362415940 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2681880280 0 0) 0x7f9b7809e970 con 0x7f9b7809b450 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient(hunting): my global_id is 5502 2024-09-16T05:38:15.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b60003530 con 0x7f9b7809b450 2024-09-16T05:38:15.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761739289 0 0) 0x7f9b7809d620 con 0x7f9b780a08a0 2024-09-16T05:38:15.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient(hunting): my global_id is 5489 2024-09-16T05:38:15.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b7809e970 con 0x7f9b780a08a0 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4036118514 0 0) 0x7f9b7809e970 con 0x7f9b780a08a0 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 >> v1:172.21.15.63:6789/0 conn(0x7f9b7809b8a0 legacy=0x7f9b7813c0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 >> v1:172.21.15.6:6789/0 conn(0x7f9b7809b450 legacy=0x7f9b781389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 monclient: found mon.noname-b 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7809d900 con 0x7f9b780a08a0 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:15.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9b68001600 con 0x7f9b780a08a0 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: dump: 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.093 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 5 monclient: authenticate success, global_id 5489 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9b680018f0 con 0x7f9b780a08a0 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 4 set_mon_vals no callback set 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 1 -- 172.21.15.63:0/3362415940 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9b68001cc0 con 0x7f9b780a08a0 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b75ffb640 10 monclient: dump: 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.094 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient: shutdown 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3362415940 >> v1:172.21.15.6:6790/0 conn(0x7f9b780a08a0 legacy=0x7f9b7809ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3362415940 shutdown_connections 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3362415940 >> 172.21.15.63:0/3362415940 conn(0x7f9b780a1c90 msgr2=0x7f9b780a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3362415940 shutdown_connections 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3362415940 wait complete. 2024-09-16T05:38:15.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient: build_initial_monmap 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient: monmap: 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 Processor -- start 2024-09-16T05:38:15.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- start start 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient: init 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7809b450 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7809b8a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient(hunting): picked mon.noname-b con 0x7f9b780a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b78122b20 con 0x7f9b7809b8a0 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b78122e00 con 0x7f9b780a08a0 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b7809c0f0 con 0x7f9b7809b450 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b7d55e640 10 monclient(hunting): authenticate will time out at 1151.859097s 2024-09-16T05:38:15.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b777fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9b780a08a0 legacy=0x7f9b781cf8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b76ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9b7809b450 0x7f9b781c7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40778/0 (socket says 172.21.15.63:40778) 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b76ffd640 1 -- 172.21.15.63:0/3072189137 learned_addr learned my addr 172.21.15.63:0/3072189137 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2296636254 0 0) 0x7f9b7809c0f0 con 0x7f9b7809b450 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 10 monclient(hunting): my global_id is 5492 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b64003530 con 0x7f9b7809b450 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057358024 0 0) 0x7f9b78122b20 con 0x7f9b7809b8a0 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 10 monclient(hunting): my global_id is 5505 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.078+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b7809c0f0 con 0x7f9b7809b8a0 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1467546265 0 0) 0x7f9b64003530 con 0x7f9b7809b450 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b78122b20 con 0x7f9b7809b450 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9b68001680 con 0x7f9b7809b450 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:15.099 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.100 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.100 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.100 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.100 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.100 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.100 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3117547121 0 0) 0x7f9b7809c0f0 con 0x7f9b7809b8a0 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b64003530 con 0x7f9b7809b8a0 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9b6c0017a0 con 0x7f9b7809b8a0 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:15.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 959150287 0 0) 0x7f9b78122b20 con 0x7f9b7809b450 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 >> v1:172.21.15.63:6789/0 conn(0x7f9b780a08a0 legacy=0x7f9b781cf8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 >> v1:172.21.15.6:6789/0 conn(0x7f9b7809b8a0 legacy=0x7f9b781cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 monclient: found mon.c 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b780a3de0 con 0x7f9b7809b450 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 5 monclient: authenticate success, global_id 5492 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 10 monclient: _renew_subs 2024-09-16T05:38:15.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9b78120b10 con 0x7f9b7809b450 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 10 monclient: _renew_subs 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9b7811fed0 con 0x7f9b7809b450 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9b680018f0 con 0x7f9b7809b450 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9b68004d30 con 0x7f9b7809b450 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 10 monclient: dump: 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.104 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b757fa640 4 set_mon_vals no callback set 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b58005090 con 0x7f9b7809b450 2024-09-16T05:38:15.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9b68004ed0 con 0x7f9b7809b450 2024-09-16T05:38:15.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(123..123 src has 1..123) ==== 5759+0+0 (unknown 1080377195 0 0) 0x7f9b68044a30 con 0x7f9b7809b450 2024-09-16T05:38:15.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.082+0000 7f9b767fc640 10 monclient: get_auth_request con 0x7f9b64036a70 auth_method 0 2024-09-16T05:38:15.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.094+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9b68015490 con 0x7f9b7809b450 2024-09-16T05:38:15.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.094+0000 7f9b5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.094+0000 7f9b5f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:15.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.258+0000 7f9b7d55e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:15.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.258+0000 7f9b7d55e640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:15.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.258+0000 7f9b7d55e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.258+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f9b58005380 con 0x7f9b7809b450 2024-09-16T05:38:15.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.262+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v123) ==== 71+0+0 (unknown 2612544506 0 0) 0x7f9b68014d90 con 0x7f9b7809b450 2024-09-16T05:38:15.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.262+0000 7f9b5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:15.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.262+0000 7f9b5f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:15.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.366+0000 7f9b7d55e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:15.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.366+0000 7f9b7d55e640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:15.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.366+0000 7f9b7d55e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.366+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f9b58005ac0 con 0x7f9b7809b450 2024-09-16T05:38:15.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.366+0000 7f9b5f7fe640 1 -- 172.21.15.63:0/3072189137 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v123) ==== 71+0+0 (unknown 2612544506 0 0) 0x7f9b6801d940 con 0x7f9b7809b450 2024-09-16T05:38:15.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.366+0000 7f9b5f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:15.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.366+0000 7f9b5f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:15.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.366+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9b64036a70 legacy=0x7f9b64038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.370+0000 7f9b7d55e640 10 monclient: shutdown 2024-09-16T05:38:15.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.370+0000 7f9b7d55e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:15.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.370+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 >> v1:172.21.15.6:6790/0 conn(0x7f9b7809b450 legacy=0x7f9b781c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.370+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 shutdown_connections 2024-09-16T05:38:15.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.370+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 >> 172.21.15.63:0/3072189137 conn(0x7f9b780a1c90 msgr2=0x7f9b7809f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:15.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.370+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 shutdown_connections 2024-09-16T05:38:15.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.370+0000 7f9b7d55e640 1 -- 172.21.15.63:0/3072189137 wait complete. 2024-09-16T05:38:15.386 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T05:38:15.386 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:38:15.386 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T05:38:15.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.506+0000 7f8a18fc8640 10 monclient: get_monmap_and_config 2024-09-16T05:38:15.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.506+0000 7f8a18fc8640 10 monclient: build_initial_monmap 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.506+0000 7f8a18fc8640 10 monclient: monmap: 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:15.511249+0000 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:15.511249+0000 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:15.512 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.506+0000 7f8a18fc8640 1 Processor -- start 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- start start 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient: init 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f8a1409c300 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f8a140a09c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f8a1409d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a140a0ed0 con 0x7f8a140a09c0 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a1409d950 con 0x7f8a1409c300 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a1409ebd0 con 0x7f8a1409d3a0 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:15.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient(hunting): authenticate will time out at 1152.288761s 2024-09-16T05:38:15.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8a1409c300 legacy=0x7f8a1409c6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:15.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a11d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8a140a09c0 0x7f8a14138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48100/0 (socket says 172.21.15.63:48100) 2024-09-16T05:38:15.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a11d9b640 1 -- 172.21.15.63:0/742489116 learned_addr learned my addr 172.21.15.63:0/742489116 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:15.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 694626669 0 0) 0x7f8a140a0ed0 con 0x7f8a140a09c0 2024-09-16T05:38:15.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient(hunting): my global_id is 5508 2024-09-16T05:38:15.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89fc003530 con 0x7f8a140a09c0 2024-09-16T05:38:15.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3610375074 0 0) 0x7f8a1409ebd0 con 0x7f8a1409d3a0 2024-09-16T05:38:15.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient(hunting): my global_id is 5495 2024-09-16T05:38:15.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a140a0ed0 con 0x7f8a1409d3a0 2024-09-16T05:38:15.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2341101972 0 0) 0x7f89fc003530 con 0x7f8a140a09c0 2024-09-16T05:38:15.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 >> v1:172.21.15.6:6790/0 conn(0x7f8a1409d3a0 legacy=0x7f8a1413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 >> v1:172.21.15.63:6789/0 conn(0x7f8a1409c300 legacy=0x7f8a1409c6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:15.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:15.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 monclient: found mon.noname-a 2024-09-16T05:38:15.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:15.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:15.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a140a2470 con 0x7f8a140a09c0 2024-09-16T05:38:15.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: _finish_auth 0 2024-09-16T05:38:15.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:15.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:15.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8a08001620 con 0x7f8a140a09c0 2024-09-16T05:38:15.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 5 monclient: authenticate success, global_id 5508 2024-09-16T05:38:15.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:15.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:15.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: dump: 2024-09-16T05:38:15.525 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.525 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.525 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.525 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.525 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.525 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.526 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.526 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.526 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.526 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:15.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a08001880 con 0x7f8a140a09c0 2024-09-16T05:38:15.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:15.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 1 -- 172.21.15.63:0/742489116 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8a08001c50 con 0x7f8a140a09c0 2024-09-16T05:38:15.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:15.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 4 set_mon_vals no callback set 2024-09-16T05:38:15.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:15.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a1159a640 10 monclient: dump: 2024-09-16T05:38:15.527 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.527 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.527 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.528 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.528 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.528 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.528 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.528 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.528 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.528 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient: shutdown 2024-09-16T05:38:15.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:15.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- 172.21.15.63:0/742489116 >> v1:172.21.15.6:6789/0 conn(0x7f8a140a09c0 legacy=0x7f8a14138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- 172.21.15.63:0/742489116 shutdown_connections 2024-09-16T05:38:15.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- 172.21.15.63:0/742489116 >> 172.21.15.63:0/742489116 conn(0x7f8a140a1e90 msgr2=0x7f8a14075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:15.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- 172.21.15.63:0/742489116 shutdown_connections 2024-09-16T05:38:15.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 1 -- 172.21.15.63:0/742489116 wait complete. 2024-09-16T05:38:15.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient: build_initial_monmap 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.510+0000 7f8a18fc8640 10 monclient: monmap: 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:15.530 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:15.531 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:15.531 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 1 Processor -- start 2024-09-16T05:38:15.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 1 -- start start 2024-09-16T05:38:15.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: init 2024-09-16T05:38:15.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:15.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:15.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f8a1409c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1409d3a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f8a141cba20 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a140a35b0 con 0x7f8a1409d3a0 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a14123210 con 0x7f8a141cba20 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a140a4150 con 0x7f8a1409c300 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient(hunting): authenticate will time out at 1152.292894s 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a12d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8a141cba20 legacy=0x7f8a141d3a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:15.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a11d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8a1409d3a0 0x7f8a141d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48106/0 (socket says 172.21.15.63:48106) 2024-09-16T05:38:15.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a1259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8a1409c300 0x7f8a14133c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40796/0 (socket says 172.21.15.63:40796) 2024-09-16T05:38:15.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a11d9b640 1 -- 172.21.15.63:0/451561610 learned_addr learned my addr 172.21.15.63:0/451561610 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:15.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653964615 0 0) 0x7f8a140a35b0 con 0x7f8a1409d3a0 2024-09-16T05:38:15.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): my global_id is 5511 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89f4003530 con 0x7f8a1409d3a0 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1661232922 0 0) 0x7f8a140a4150 con 0x7f8a1409c300 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): my global_id is 5498 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a140a35b0 con 0x7f8a1409c300 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3613528382 0 0) 0x7f8a140a35b0 con 0x7f8a1409c300 2024-09-16T05:38:15.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a140a4150 con 0x7f8a1409c300 2024-09-16T05:38:15.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2621757634 0 0) 0x7f89f4003530 con 0x7f8a1409d3a0 2024-09-16T05:38:15.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a140a35b0 con 0x7f8a1409d3a0 2024-09-16T05:38:15.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8a08001bd0 con 0x7f8a1409d3a0 2024-09-16T05:38:15.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:15.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:15.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:15.535 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8a04004110 con 0x7f8a1409c300 2024-09-16T05:38:15.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:15.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:15.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:15.537 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.537 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.537 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.537 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.537 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 432443530 0 0) 0x7f8a140a4150 con 0x7f8a1409c300 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 >> v1:172.21.15.63:6789/0 conn(0x7f8a141cba20 legacy=0x7f8a141d3a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 >> v1:172.21.15.6:6789/0 conn(0x7f8a1409d3a0 legacy=0x7f8a141d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 monclient: found mon.c 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:15.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1409c0f0 con 0x7f8a1409c300 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 5 monclient: authenticate success, global_id 5498 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: _renew_subs 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8a1406f500 con 0x7f8a1409c300 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: _renew_subs 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8a14124660 con 0x7f8a1409c300 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a04004610 con 0x7f8a1409c300 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8a04004960 con 0x7f8a1409c300 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a10d99640 4 set_mon_vals no callback set 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a02ffd640 10 monclient: dump: 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.540 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.514+0000 7f8a18fc8640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89f8005090 con 0x7f8a1409c300 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.518+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8a04004b80 con 0x7f8a1409c300 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.518+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(123..123 src has 1..123) ==== 5759+0+0 (unknown 1080377195 0 0) 0x7f8a04044990 con 0x7f8a1409c300 2024-09-16T05:38:15.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.518+0000 7f8a11d9b640 10 monclient: get_auth_request con 0x7f89f40369c0 auth_method 0 2024-09-16T05:38:15.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.530+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8a040153f0 con 0x7f8a1409c300 2024-09-16T05:38:15.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.530+0000 7f8a02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.530+0000 7f8a02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:15.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.702+0000 7f8a18fc8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:15.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.702+0000 7f8a18fc8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:15.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.702+0000 7f8a18fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.702+0000 7f8a18fc8640 1 -- 172.21.15.63:0/451561610 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7f89f8005380 con 0x7f8a1409c300 2024-09-16T05:38:15.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.706+0000 7f8a02ffd640 1 -- 172.21.15.63:0/451561610 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 pool 'fooset' not found v123) ==== 152+0+0 (unknown 958680232 0 0) 0x7f8a04014cf0 con 0x7f8a1409c300 2024-09-16T05:38:15.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.706+0000 7f8a02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:15.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.706+0000 7f8a02ffd640 10 monclient: _finish_command 2 = mon:2 pool 'fooset' not found 2024-09-16T05:38:15.710 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOENT: pool 'fooset' not found 2024-09-16T05:38:15.715 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:38:15.715 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create fooset 8 2024-09-16T05:38:15.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.830+0000 7f6c231c6640 10 monclient: get_monmap_and_config 2024-09-16T05:38:15.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.830+0000 7f6c231c6640 10 monclient: build_initial_monmap 2024-09-16T05:38:15.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.830+0000 7f6c231c6640 10 monclient: monmap: 2024-09-16T05:38:15.836 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:15.836 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.836 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:15.835933+0000 2024-09-16T05:38:15.837 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:15.835933+0000 2024-09-16T05:38:15.837 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:15.837 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.837 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:15.837 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:15.837 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:15.837 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 Processor -- start 2024-09-16T05:38:15.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- start start 2024-09-16T05:38:15.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient: init 2024-09-16T05:38:15.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient(hunting): picked mon.noname-a con 0x7f6c1c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient(hunting): picked mon.noname-c con 0x7f6c1c09b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6c1c139fc0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c0a14f0 con 0x7f6c1c09ac90 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c09c680 con 0x7f6c1c09b780 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c09e930 con 0x7f6c1c139fc0 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient(hunting): authenticate will time out at 1152.612857s 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6c1c09b780 legacy=0x7f6c1c1388a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:15.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c21763640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6c1c139fc0 0x7f6c1c13c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40798/0 (socket says 172.21.15.63:40798) 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c21763640 1 -- 172.21.15.63:0/2886460794 learned_addr learned my addr 172.21.15.63:0/2886460794 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c20f62640 1 --1- 172.21.15.63:0/2886460794 >> v1:172.21.15.6:6789/0 conn(0x7f6c1c09ac90 0x7f6c1c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48116/0 (socket says 172.21.15.63:48116) 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1351388160 0 0) 0x7f6c1c09e930 con 0x7f6c1c139fc0 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient(hunting): my global_id is 5501 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c0c003530 con 0x7f6c1c139fc0 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228298860 0 0) 0x7f6c1c0a14f0 con 0x7f6c1c09ac90 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient(hunting): my global_id is 5514 2024-09-16T05:38:15.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c1c09e930 con 0x7f6c1c09ac90 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 161775339 0 0) 0x7f6c0c003530 con 0x7f6c1c139fc0 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 >> v1:172.21.15.63:6789/0 conn(0x7f6c1c09b780 legacy=0x7f6c1c1388a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 >> v1:172.21.15.6:6789/0 conn(0x7f6c1c09ac90 legacy=0x7f6c1c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 monclient: found mon.noname-b 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c09c400 con 0x7f6c1c139fc0 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:15.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c140019a0 con 0x7f6c1c139fc0 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: dump: 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.844 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 5 monclient: authenticate success, global_id 5501 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c14001c50 con 0x7f6c1c139fc0 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 4 set_mon_vals no callback set 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 1 -- 172.21.15.63:0/2886460794 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c14002020 con 0x7f6c1c139fc0 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c1b7fe640 10 monclient: dump: 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.845 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient: shutdown 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- 172.21.15.63:0/2886460794 >> v1:172.21.15.6:6790/0 conn(0x7f6c1c139fc0 legacy=0x7f6c1c13c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- 172.21.15.63:0/2886460794 shutdown_connections 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- 172.21.15.63:0/2886460794 >> 172.21.15.63:0/2886460794 conn(0x7f6c1c0a5290 msgr2=0x7f6c1c0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- 172.21.15.63:0/2886460794 shutdown_connections 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 1 -- 172.21.15.63:0/2886460794 wait complete. 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient: build_initial_monmap 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.834+0000 7f6c231c6640 10 monclient: monmap: 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:15.847 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:15.848 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:15.848 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 1 Processor -- start 2024-09-16T05:38:15.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 1 -- start start 2024-09-16T05:38:15.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: init 2024-09-16T05:38:15.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:15.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:15.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient(hunting): picked mon.noname-c con 0x7f6c1c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:15.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient(hunting): picked mon.noname-a con 0x7f6c1c09b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:15.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6c1c139fc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:15.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c11d410 con 0x7f6c1c09b780 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c11d6f0 con 0x7f6c1c139fc0 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c11c3f0 con 0x7f6c1c09ac90 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient(hunting): authenticate will time out at 1152.616576s 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c21763640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6c1c139fc0 legacy=0x7f6c1c1cf180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c1bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6c1c09b780 0x7f6c1c138840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48132/0 (socket says 172.21.15.63:48132) 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c20f62640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6c1c09ac90 0x7f6c1c138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40804/0 (socket says 172.21.15.63:40804) 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c1bfff640 1 -- 172.21.15.63:0/4266804048 learned_addr learned my addr 172.21.15.63:0/4266804048 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 207301343 0 0) 0x7f6c1c11d410 con 0x7f6c1c09b780 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): my global_id is 5517 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c00003530 con 0x7f6c1c09b780 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3092443986 0 0) 0x7f6c1c11c3f0 con 0x7f6c1c09ac90 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): my global_id is 5504 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c1c11d410 con 0x7f6c1c09ac90 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2278977792 0 0) 0x7f6c00003530 con 0x7f6c1c09b780 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c1c11c3f0 con 0x7f6c1c09b780 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1603913047 0 0) 0x7f6c1c11d410 con 0x7f6c1c09ac90 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c00003530 con 0x7f6c1c09ac90 2024-09-16T05:38:15.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c100018c0 con 0x7f6c1c09ac90 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): dump: 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.854 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c08004160 con 0x7f6c1c09b780 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient(hunting): dump: 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.855 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2048422194 0 0) 0x7f6c1c11c3f0 con 0x7f6c1c09b780 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 >> v1:172.21.15.6:6790/0 conn(0x7f6c1c09ac90 legacy=0x7f6c1c138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 >> v1:172.21.15.63:6789/0 conn(0x7f6c1c139fc0 legacy=0x7f6c1c1cf180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient: _finish_hunting 0 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 monclient: found mon.a 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c11b670 con 0x7f6c1c09b780 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient: _finish_auth 0 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient: _check_auth_tickets 2024-09-16T05:38:15.856 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 5 monclient: authenticate success, global_id 5517 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: _renew_subs 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1c11d9d0 con 0x7f6c1c09b780 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: _renew_subs 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c1c139120 con 0x7f6c1c09b780 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c08004390 con 0x7f6c1c09b780 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6c08004760 con 0x7f6c1c09b780 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:15.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c197fa640 10 monclient: dump: 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c1affd640 4 set_mon_vals no callback set 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:15.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.838+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bfc005090 con 0x7f6c1c09b780 2024-09-16T05:38:15.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.842+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6c08004980 con 0x7f6c1c09b780 2024-09-16T05:38:15.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.842+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(123..123 src has 1..123) ==== 5759+0+0 (unknown 1080377195 0 0) 0x7f6c080449c0 con 0x7f6c1c09b780 2024-09-16T05:38:15.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.842+0000 7f6c20f62640 10 monclient: get_auth_request con 0x7f6c000369c0 auth_method 0 2024-09-16T05:38:15.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.854+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6c080153c0 con 0x7f6c1c09b780 2024-09-16T05:38:15.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.854+0000 7f6c197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:15.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:15.854+0000 7f6c197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:16.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.018+0000 7f6c231c6640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T05:38:16.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.018+0000 7f6c231c6640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T05:38:16.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.018+0000 7f6c231c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:16.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.018+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) -- 0x7f6bfc005b90 con 0x7f6c1c09b780 2024-09-16T05:38:16.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.514+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' created v124) ==== 115+0+0 (unknown 1318649273 0 0) 0x7f6c08014cc0 con 0x7f6c1c09b780 2024-09-16T05:38:16.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.514+0000 7f6c197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T05:38:16.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.514+0000 7f6c197fa640 10 monclient: _finish_command 2 = system:0 pool 'fooset' created 2024-09-16T05:38:16.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.614+0000 7f6c231c6640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T05:38:16.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.614+0000 7f6c231c6640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T05:38:16.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.614+0000 7f6c231c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:16.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.614+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) -- 0x7f6bfc005610 con 0x7f6c1c09b780 2024-09-16T05:38:16.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c197fa640 1 -- 172.21.15.63:0/4266804048 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' already exists v124) ==== 122+0+0 (unknown 675487933 0 0) 0x7f6c08019010 con 0x7f6c1c09b780 2024-09-16T05:38:16.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T05:38:16.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c197fa640 10 monclient: _finish_command 3 = system:0 pool 'fooset' already exists 2024-09-16T05:38:16.621 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'fooset' already exists 2024-09-16T05:38:16.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6c000369c0 legacy=0x7f6c00038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:16.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c231c6640 10 monclient: shutdown 2024-09-16T05:38:16.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c231c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:16.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 >> v1:172.21.15.6:6789/0 conn(0x7f6c1c09b780 legacy=0x7f6c1c138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:16.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 shutdown_connections 2024-09-16T05:38:16.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 >> 172.21.15.63:0/4266804048 conn(0x7f6c1c0a5290 msgr2=0x7f6c1c0a4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:16.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 shutdown_connections 2024-09-16T05:38:16.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.618+0000 7f6c231c6640 1 -- 172.21.15.63:0/4266804048 wait complete. 2024-09-16T05:38:16.640 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create barset 8 2024-09-16T05:38:16.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient: get_monmap_and_config 2024-09-16T05:38:16.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient: build_initial_monmap 2024-09-16T05:38:16.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient: monmap: 2024-09-16T05:38:16.761 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:16.761 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:16.762 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:16.761250+0000 2024-09-16T05:38:16.762 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:16.761250+0000 2024-09-16T05:38:16.762 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:16.762 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:16.762 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:16.762 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:16.762 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:16.762 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:16.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 1 Processor -- start 2024-09-16T05:38:16.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 1 -- start start 2024-09-16T05:38:16.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient: init 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a4134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a4135420 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a40a2a80 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a40a36d0 con 0x7fc5a4135420 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a409b370 con 0x7fc5a4134930 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a409c3a0 con 0x7fc5a40a2a80 2024-09-16T05:38:16.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:16.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5aa868640 10 monclient(hunting): authenticate will time out at 1153.539062s 2024-09-16T05:38:16.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5a3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc5a4134930 legacy=0x7fc5a4134d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:16.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5a37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc5a4135420 0x7fc5a4139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48136/0 (socket says 172.21.15.63:48136) 2024-09-16T05:38:16.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5a37fe640 1 -- 172.21.15.63:0/3789032825 learned_addr learned my addr 172.21.15.63:0/3789032825 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:16.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.758+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322649882 0 0) 0x7fc5a40a36d0 con 0x7fc5a4135420 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient(hunting): my global_id is 5520 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc58c003530 con 0x7fc5a4135420 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234178034 0 0) 0x7fc5a409c3a0 con 0x7fc5a40a2a80 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient(hunting): my global_id is 5507 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a40a36d0 con 0x7fc5a40a2a80 2024-09-16T05:38:16.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 929801954 0 0) 0x7fc5a40a36d0 con 0x7fc5a40a2a80 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 >> v1:172.21.15.63:6789/0 conn(0x7fc5a4134930 legacy=0x7fc5a4134d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 >> v1:172.21.15.6:6789/0 conn(0x7fc5a4135420 legacy=0x7fc5a4139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 monclient: found mon.noname-b 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a409cc40 con 0x7fc5a40a2a80 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:16.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc59c0018b0 con 0x7fc5a40a2a80 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: dump: 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:16.769 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 5 monclient: authenticate success, global_id 5507 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc59c001b10 con 0x7fc5a40a2a80 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 1 -- 172.21.15.63:0/3789032825 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc59c001ee0 con 0x7fc5a40a2a80 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a2ffd640 10 monclient: dump: 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:16.770 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 4 set_mon_vals no callback set 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient: shutdown 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- 172.21.15.63:0/3789032825 >> v1:172.21.15.6:6790/0 conn(0x7fc5a40a2a80 legacy=0x7fc5a413d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- 172.21.15.63:0/3789032825 shutdown_connections 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- 172.21.15.63:0/3789032825 >> 172.21.15.63:0/3789032825 conn(0x7fc5a40a7340 msgr2=0x7fc5a40a7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- 172.21.15.63:0/3789032825 shutdown_connections 2024-09-16T05:38:16.771 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- 172.21.15.63:0/3789032825 wait complete. 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient: build_initial_monmap 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient: monmap: 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:16.772 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:16.773 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:16.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 Processor -- start 2024-09-16T05:38:16.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- start start 2024-09-16T05:38:16.773 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient: init 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a40a2a80 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a4134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a4135420 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a4124390 con 0x7fc5a4134930 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a4122820 con 0x7fc5a4135420 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a4072f80 con 0x7fc5a40a2a80 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5aa868640 10 monclient(hunting): authenticate will time out at 1153.542916s 2024-09-16T05:38:16.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a8e05640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc5a4135420 legacy=0x7fc5a41d1890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc5a40a2a80 0x7fc5a4133c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40816/0 (socket says 172.21.15.63:40816) 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a3fff640 1 -- 172.21.15.63:0/1066954438 learned_addr learned my addr 172.21.15.63:0/1066954438 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a3fff640 1 -- 172.21.15.63:0/1066954438 >> v1:172.21.15.6:6790/0 conn(0x7fc5a40a2a80 legacy=0x7fc5a4133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a3fff640 1 -- 172.21.15.63:0/1066954438 >> v1:172.21.15.6:6790/0 conn(0x7fc5a40a2a80 legacy=0x7fc5a4133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a3fff640 1 --1- 172.21.15.63:0/1066954438 >> v1:172.21.15.6:6790/0 conn(0x7fc5a40a2a80 0x7fc5a4133c80 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a3fff640 1 --1- 172.21.15.63:0/1066954438 >> v1:172.21.15.6:6790/0 conn(0x7fc5a40a2a80 0x7fc5a4133c80 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a0ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6790/0 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2158850203 0 0) 0x7fc5a4124390 con 0x7fc5a4134930 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a0ff9640 10 monclient(hunting): my global_id is 5523 2024-09-16T05:38:16.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.762+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc590003530 con 0x7fc5a4134930 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3917360356 0 0) 0x7fc590003530 con 0x7fc5a4134930 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5a4124390 con 0x7fc5a4134930 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc598001a40 con 0x7fc5a4134930 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:16.778 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1138146988 0 0) 0x7fc5a4124390 con 0x7fc5a4134930 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 >> v1:172.21.15.6:6790/0 conn(0x7fc5a40a2a80 legacy=0x7fc5a4133c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 >> v1:172.21.15.63:6789/0 conn(0x7fc5a4135420 legacy=0x7fc5a41d1890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 monclient: found mon.a 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a40a23f0 con 0x7fc5a4134930 2024-09-16T05:38:16.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 5 monclient: authenticate success, global_id 5523 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 10 monclient: _renew_subs 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a41240b0 con 0x7fc5a4134930 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 10 monclient: _renew_subs 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5a4134390 con 0x7fc5a4134930 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc598004aa0 con 0x7fc5a4134930 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc598004c50 con 0x7fc5a4134930 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:16.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a27fc640 4 set_mon_vals no callback set 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 10 monclient: dump: 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:16.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:16.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:16.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:16.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc584005090 con 0x7fc5a4134930 2024-09-16T05:38:16.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc59800f9f0 con 0x7fc5a4134930 2024-09-16T05:38:16.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(124..124 src has 1..124) ==== 6104+0+0 (unknown 391995788 0 0) 0x7fc598044c90 con 0x7fc5a4134930 2024-09-16T05:38:16.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.766+0000 7fc5a3fff640 10 monclient: get_auth_request con 0x7fc590036980 auth_method 0 2024-09-16T05:38:16.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.778+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc598015530 con 0x7fc5a4134930 2024-09-16T05:38:16.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.778+0000 7fc5a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:16.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.778+0000 7fc5a0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:16.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.946+0000 7fc5aa868640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T05:38:16.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.946+0000 7fc5aa868640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T05:38:16.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.946+0000 7fc5aa868640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:16.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:16.946+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) -- 0x7fc584005b90 con 0x7fc5a4134930 2024-09-16T05:38:17.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.522+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' created v125) ==== 115+0+0 (unknown 3450295504 0 0) 0x7fc598014e30 con 0x7fc5a4134930 2024-09-16T05:38:17.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.526+0000 7fc5a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T05:38:17.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.526+0000 7fc5a0ff9640 10 monclient: _finish_command 2 = system:0 pool 'barset' created 2024-09-16T05:38:17.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5aa868640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T05:38:17.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5aa868640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T05:38:17.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5aa868640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:17.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) -- 0x7fc584005610 con 0x7fc5a4134930 2024-09-16T05:38:17.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5a0ff9640 1 -- 172.21.15.63:0/1066954438 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' already exists v125) ==== 122+0+0 (unknown 186601214 0 0) 0x7fc598019180 con 0x7fc5a4134930 2024-09-16T05:38:17.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5a0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T05:38:17.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5a0ff9640 10 monclient: _finish_command 3 = system:0 pool 'barset' already exists 2024-09-16T05:38:17.638 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'barset' already exists 2024-09-16T05:38:17.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc590036980 legacy=0x7fc590038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:17.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5aa868640 10 monclient: shutdown 2024-09-16T05:38:17.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5aa868640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:17.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.634+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 >> v1:172.21.15.6:6789/0 conn(0x7fc5a4134930 legacy=0x7fc5a41ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:17.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.638+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 shutdown_connections 2024-09-16T05:38:17.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.638+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 >> 172.21.15.63:0/1066954438 conn(0x7fc5a40a7340 msgr2=0x7fc5a40a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:17.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.638+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 shutdown_connections 2024-09-16T05:38:17.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.638+0000 7fc5aa868640 1 -- 172.21.15.63:0/1066954438 wait complete. 2024-09-16T05:38:17.655 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set barset size 3 2024-09-16T05:38:17.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient: get_monmap_and_config 2024-09-16T05:38:17.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient: build_initial_monmap 2024-09-16T05:38:17.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient: monmap: 2024-09-16T05:38:17.781 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:17.781 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:17.781 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:17.781017+0000 2024-09-16T05:38:17.782 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:17.781017+0000 2024-09-16T05:38:17.782 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:17.782 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:17.782 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:17.782 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:17.782 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:17.782 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:17.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 1 Processor -- start 2024-09-16T05:38:17.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 1 -- start start 2024-09-16T05:38:17.783 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient: init 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient(hunting): picked mon.noname-b con 0x7fa54409c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient(hunting): picked mon.noname-c con 0x7fa5440a09c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient(hunting): picked mon.noname-a con 0x7fa54409d3a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5440a0ed0 con 0x7fa54409d3a0 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54409d950 con 0x7fa5440a09c0 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54409ebd0 con 0x7fa54409c300 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 10 monclient(hunting): authenticate will time out at 1154.558554s 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa5437fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa5440a09c0 legacy=0x7fa544138a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa543fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa54409c300 0x7fa54409c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40832/0 (socket says 172.21.15.63:40832) 2024-09-16T05:38:17.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa543fff640 1 -- 172.21.15.63:0/1423578154 learned_addr learned my addr 172.21.15.63:0/1423578154 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:17.785 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa548d32640 1 --1- 172.21.15.63:0/1423578154 >> v1:172.21.15.6:6789/0 conn(0x7fa54409d3a0 0x7fa54413c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48158/0 (socket says 172.21.15.63:48158) 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2565839355 0 0) 0x7fa5440a0ed0 con 0x7fa54409d3a0 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient(hunting): my global_id is 5526 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa52c003530 con 0x7fa54409d3a0 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3752517752 0 0) 0x7fa54409ebd0 con 0x7fa54409c300 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:17.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient(hunting): my global_id is 5510 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5440a0ed0 con 0x7fa54409c300 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2712750376 0 0) 0x7fa52c003530 con 0x7fa54409d3a0 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 >> v1:172.21.15.6:6790/0 conn(0x7fa54409c300 legacy=0x7fa54409c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 >> v1:172.21.15.63:6789/0 conn(0x7fa5440a09c0 legacy=0x7fa544138a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 monclient: found mon.noname-a 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5440a2470 con 0x7fa54409d3a0 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:17.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 5 monclient: authenticate success, global_id 5526 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa53c0019a0 con 0x7fa54409d3a0 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa542ffd640 10 monclient: dump: 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.792 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.778+0000 7fa54a795640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa53c001c00 con 0x7fa54409d3a0 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa542ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa542ffd640 1 -- 172.21.15.63:0/1423578154 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa53c001fd0 con 0x7fa54409d3a0 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa542ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa542ffd640 10 monclient: dump: 2024-09-16T05:38:17.793 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 4 set_mon_vals no callback set 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient: shutdown 2024-09-16T05:38:17.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- 172.21.15.63:0/1423578154 >> v1:172.21.15.6:6789/0 conn(0x7fa54409d3a0 legacy=0x7fa54413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- 172.21.15.63:0/1423578154 shutdown_connections 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- 172.21.15.63:0/1423578154 >> 172.21.15.63:0/1423578154 conn(0x7fa5440a1e90 msgr2=0x7fa544075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- 172.21.15.63:0/1423578154 shutdown_connections 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- 172.21.15.63:0/1423578154 wait complete. 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient: build_initial_monmap 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient: monmap: 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:17.795 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 Processor -- start 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- start start 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient: init 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient(hunting): picked mon.noname-c con 0x7fa54409c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient(hunting): picked mon.noname-b con 0x7fa54409d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:17.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient(hunting): picked mon.noname-a con 0x7fa544132070 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa544120330 con 0x7fa544132070 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5440a42c0 con 0x7fa54409d3a0 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54409c0f0 con 0x7fa54409c300 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa54a795640 10 monclient(hunting): authenticate will time out at 1154.562713s 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa5437fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa54409d3a0 legacy=0x7fa5441cc010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa543fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa54409c300 0x7fa544131960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40836/0 (socket says 172.21.15.63:40836) 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa543fff640 1 -- 172.21.15.63:0/3078940466 learned_addr learned my addr 172.21.15.63:0/3078940466 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 501811592 0 0) 0x7fa54409c0f0 con 0x7fa54409c300 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:17.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 10 monclient(hunting): my global_id is 5513 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa530003530 con 0x7fa54409c300 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306207216 0 0) 0x7fa544120330 con 0x7fa544132070 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 10 monclient(hunting): my global_id is 5529 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.782+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa54409c0f0 con 0x7fa544132070 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 475426092 0 0) 0x7fa530003530 con 0x7fa54409c300 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa544120330 con 0x7fa54409c300 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa5340018a0 con 0x7fa54409c300 2024-09-16T05:38:17.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:17.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2489728909 0 0) 0x7fa54409c0f0 con 0x7fa544132070 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa530003530 con 0x7fa544132070 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa53c001f50 con 0x7fa544132070 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:17.800 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3049862731 0 0) 0x7fa544120330 con 0x7fa54409c300 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 >> v1:172.21.15.63:6789/0 conn(0x7fa54409d3a0 legacy=0x7fa5441cc010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 >> v1:172.21.15.6:6789/0 conn(0x7fa544132070 legacy=0x7fa5441cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 monclient: found mon.c 2024-09-16T05:38:17.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa544132550 con 0x7fa54409c300 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient: _finish_auth 0 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 5 monclient: authenticate success, global_id 5513 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 10 monclient: _renew_subs 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa54411fd70 con 0x7fa54409c300 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 10 monclient: _renew_subs 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa54411e920 con 0x7fa54409c300 2024-09-16T05:38:17.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5340047f0 con 0x7fa54409c300 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa534004b40 con 0x7fa54409c300 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 10 monclient: dump: 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:17.803 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:17.804 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:17.804 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:17.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa5427fc640 4 set_mon_vals no callback set 2024-09-16T05:38:17.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:17.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:17.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:17.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa524005090 con 0x7fa54409c300 2024-09-16T05:38:17.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa53400f8e0 con 0x7fa54409c300 2024-09-16T05:38:17.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(125..125 src has 1..125) ==== 6449+0+0 (unknown 1861013614 0 0) 0x7fa534044c00 con 0x7fa54409c300 2024-09-16T05:38:17.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.786+0000 7fa5437fe640 10 monclient: get_auth_request con 0x7fa530036a50 auth_method 0 2024-09-16T05:38:17.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.798+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa534015340 con 0x7fa54409c300 2024-09-16T05:38:17.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.798+0000 7fa540ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:17.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.798+0000 7fa540ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:17.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.966+0000 7fa54a795640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T05:38:17.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.966+0000 7fa54a795640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T05:38:17.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.966+0000 7fa54a795640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:17.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:17.966+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) -- 0x7fa524005b90 con 0x7fa54409c300 2024-09-16T05:38:18.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:18.538+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v126) ==== 126+0+0 (unknown 1027552326 0 0) 0x7fa534014c40 con 0x7fa54409c300 2024-09-16T05:38:18.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:18.538+0000 7fa540ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T05:38:18.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:18.538+0000 7fa540ff9640 10 monclient: _finish_command 2 = system:0 set pool 10 size to 3 2024-09-16T05:38:18.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:18.642+0000 7fa54a795640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T05:38:18.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:18.642+0000 7fa54a795640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T05:38:18.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:18.642+0000 7fa54a795640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:18.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:18.642+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) -- 0x7fa524005610 con 0x7fa54409c300 2024-09-16T05:38:19.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.546+0000 7fa540ff9640 1 -- 172.21.15.63:0/3078940466 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v127) ==== 126+0+0 (unknown 3215428444 0 0) 0x7fa534018f90 con 0x7fa54409c300 2024-09-16T05:38:19.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.546+0000 7fa540ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T05:38:19.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.546+0000 7fa540ff9640 10 monclient: _finish_command 3 = system:0 set pool 10 size to 3 2024-09-16T05:38:19.552 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 10 size to 3 2024-09-16T05:38:19.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.550+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa530036a50 legacy=0x7fa530038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:19.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.550+0000 7fa54a795640 10 monclient: shutdown 2024-09-16T05:38:19.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.550+0000 7fa54a795640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:19.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.550+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 >> v1:172.21.15.6:6790/0 conn(0x7fa54409c300 legacy=0x7fa544131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:19.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.550+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 shutdown_connections 2024-09-16T05:38:19.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.550+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 >> 172.21.15.63:0/3078940466 conn(0x7fa5440a1e90 msgr2=0x7fa5440a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:19.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.554+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 shutdown_connections 2024-09-16T05:38:19.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.554+0000 7fa54a795640 1 -- 172.21.15.63:0/3078940466 wait complete. 2024-09-16T05:38:19.581 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T05:38:19.581 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:38:19.581 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient: get_monmap_and_config 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient: build_initial_monmap 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient: monmap: 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:19.702680+0000 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:19.702680+0000 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:19.703 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:19.704 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:19.704 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 1 Processor -- start 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 1 -- start start 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient: init 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient(hunting): picked mon.noname-b con 0x7f1934134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient(hunting): picked mon.noname-a con 0x7f19340a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient(hunting): picked mon.noname-c con 0x7f193406dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1934135420 con 0x7f19340a0810 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f193409e830 con 0x7f193406dfa0 2024-09-16T05:38:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19340a0dc0 con 0x7f1934134930 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f193977b640 10 monclient(hunting): authenticate will time out at 1156.479631s 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.698+0000 7f19337fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f193406dfa0 legacy=0x7f193413cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1932ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1934134930 0x7f1934134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40838/0 (socket says 172.21.15.63:40838) 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1932ffd640 1 -- 172.21.15.63:0/536154570 learned_addr learned my addr 172.21.15.63:0/536154570 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711939903 0 0) 0x7f1934135420 con 0x7f19340a0810 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient(hunting): my global_id is 5532 2024-09-16T05:38:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f191c003530 con 0x7f19340a0810 2024-09-16T05:38:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778950442 0 0) 0x7f19340a0dc0 con 0x7f1934134930 2024-09-16T05:38:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:19.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient(hunting): my global_id is 5516 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1934135420 con 0x7f1934134930 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2845239835 0 0) 0x7f191c003530 con 0x7f19340a0810 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 >> v1:172.21.15.6:6790/0 conn(0x7f1934134930 legacy=0x7f1934134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 >> v1:172.21.15.63:6789/0 conn(0x7f193406dfa0 legacy=0x7f193413cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 monclient: found mon.noname-a 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19340a2500 con 0x7f19340a0810 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:19.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1928001550 con 0x7f19340a0810 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: dump: 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:19.713 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 5 monclient: authenticate success, global_id 5532 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f19280017b0 con 0x7f19340a0810 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 1 -- 172.21.15.63:0/536154570 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1928001b80 con 0x7f19340a0810 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f1931ffb640 10 monclient: dump: 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.714 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 4 set_mon_vals no callback set 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 10 monclient: shutdown 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 1 -- 172.21.15.63:0/536154570 >> v1:172.21.15.6:6789/0 conn(0x7f19340a0810 legacy=0x7f1934139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 1 -- 172.21.15.63:0/536154570 shutdown_connections 2024-09-16T05:38:19.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 1 -- 172.21.15.63:0/536154570 >> 172.21.15.63:0/536154570 conn(0x7f19340a1e90 msgr2=0x7f1934075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 1 -- 172.21.15.63:0/536154570 shutdown_connections 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 1 -- 172.21.15.63:0/536154570 wait complete. 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 10 monclient: build_initial_monmap 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 10 monclient: monmap: 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:19.716 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 1 Processor -- start 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 1 -- start start 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.702+0000 7f193977b640 10 monclient: init 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient(hunting): picked mon.noname-a con 0x7f193406dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient(hunting): picked mon.noname-b con 0x7f19340a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient(hunting): picked mon.noname-c con 0x7f1934134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f193409d8b0 con 0x7f193406dfa0 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19340a40b0 con 0x7f19340a0810 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f193413d410 con 0x7f1934134930 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:19.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient(hunting): authenticate will time out at 1156.484229s 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f19327fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f19340a0810 legacy=0x7f193409c800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f1932ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f193406dfa0 0x7f193409c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48192/0 (socket says 172.21.15.63:48192) 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f1932ffd640 1 -- 172.21.15.63:0/739536524 learned_addr learned my addr 172.21.15.63:0/739536524 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2382973155 0 0) 0x7f193409d8b0 con 0x7f193406dfa0 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): my global_id is 5535 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1920003560 con 0x7f193406dfa0 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190108534 0 0) 0x7f193413d410 con 0x7f1934134930 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:19.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): my global_id is 5519 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f193409d8b0 con 0x7f1934134930 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3083797095 0 0) 0x7f193409d8b0 con 0x7f1934134930 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f193413d410 con 0x7f1934134930 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f192c004300 con 0x7f1934134930 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.719 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3899374436 0 0) 0x7f1920003560 con 0x7f193406dfa0 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f193409d8b0 con 0x7f193406dfa0 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f19240018c0 con 0x7f193406dfa0 2024-09-16T05:38:19.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:19.721 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2385845640 0 0) 0x7f193413d410 con 0x7f1934134930 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 >> v1:172.21.15.63:6789/0 conn(0x7f19340a0810 legacy=0x7f193409c800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 >> v1:172.21.15.6:6789/0 conn(0x7f193406dfa0 legacy=0x7f193409c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 monclient: found mon.c 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19341369f0 con 0x7f1934134930 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:19.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 5 monclient: authenticate success, global_id 5519 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient: _renew_subs 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f193409df90 con 0x7f1934134930 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient: _renew_subs 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f19341355f0 con 0x7f1934134930 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f192c004560 con 0x7f1934134930 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f19317fa640 4 set_mon_vals no callback set 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f192c0048b0 con 0x7f1934134930 2024-09-16T05:38:19.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 10 monclient: dump: 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:19.724 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:19.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:19.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:19.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:19.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f193977b640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1914005090 con 0x7f1934134930 2024-09-16T05:38:19.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f192c004a70 con 0x7f1934134930 2024-09-16T05:38:19.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(127..127 src has 1..127) ==== 6449+0+0 (unknown 1371578255 0 0) 0x7f192c044d40 con 0x7f1934134930 2024-09-16T05:38:19.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.706+0000 7f1932ffd640 10 monclient: get_auth_request con 0x7f1920036ad0 auth_method 0 2024-09-16T05:38:19.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.718+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f192c0190d0 con 0x7f1934134930 2024-09-16T05:38:19.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.722+0000 7f191b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:19.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.722+0000 7f191b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:19.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.890+0000 7f193977b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:19.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.890+0000 7f193977b640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:19.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.890+0000 7f193977b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:19.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.890+0000 7f193977b640 1 -- 172.21.15.63:0/739536524 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7f1914005380 con 0x7f1934134930 2024-09-16T05:38:19.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.894+0000 7f191b7fe640 1 -- 172.21.15.63:0/739536524 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 no weight-set for pool 'fooset' v127) ==== 160+0+0 (unknown 2399672111 0 0) 0x7f192c01d990 con 0x7f1934134930 2024-09-16T05:38:19.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.894+0000 7f191b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:19.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:19.894+0000 7f191b7fe640 10 monclient: _finish_command 2 = mon:2 no weight-set for pool 'fooset' 2024-09-16T05:38:19.899 INFO:tasks.workunit.client.0.smithi063.stderr:Error ENOENT: no weight-set for pool 'fooset' 2024-09-16T05:38:19.903 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:38:19.903 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set create fooset flat 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient: get_monmap_and_config 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient: build_initial_monmap 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient: monmap: 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:20.016564+0000 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:20.016564+0000 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:20.017 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:20.018 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:20.018 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 Processor -- start 2024-09-16T05:38:20.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- start start 2024-09-16T05:38:20.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient: init 2024-09-16T05:38:20.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:20.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:20.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient(hunting): picked mon.noname-b con 0x7fbfa40a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:20.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient(hunting): picked mon.noname-a con 0x7fbfa40a4d90 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient(hunting): picked mon.noname-c con 0x7fbfa40a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa40a51e0 con 0x7fbfa40a4d90 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa40a0d60 con 0x7fbfa40a4790 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa40a1040 con 0x7fbfa40a3ca0 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient(hunting): authenticate will time out at 1156.793492s 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa2d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbfa40a4790 legacy=0x7fbfa413f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa1d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbfa40a4d90 0x7fbfa413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48206/0 (socket says 172.21.15.63:48206) 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbfa40a3ca0 0x7fbfa40a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40852/0 (socket says 172.21.15.63:40852) 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa1d9b640 1 -- 172.21.15.63:0/2296880214 learned_addr learned my addr 172.21.15.63:0/2296880214 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225554740 0 0) 0x7fbfa40a51e0 con 0x7fbfa40a4d90 2024-09-16T05:38:20.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient(hunting): my global_id is 5538 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf90003530 con 0x7fbfa40a4d90 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2899200857 0 0) 0x7fbfa40a1040 con 0x7fbfa40a3ca0 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient(hunting): my global_id is 5522 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfa40a51e0 con 0x7fbfa40a3ca0 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1194782431 0 0) 0x7fbf90003530 con 0x7fbfa40a4d90 2024-09-16T05:38:20.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 >> v1:172.21.15.6:6790/0 conn(0x7fbfa40a3ca0 legacy=0x7fbfa40a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 >> v1:172.21.15.63:6789/0 conn(0x7fbfa40a4790 legacy=0x7fbfa413f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 monclient: found mon.noname-a 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa40a0810 con 0x7fbfa40a4d90 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: _finish_auth 0 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf8c0018d0 con 0x7fbfa40a4d90 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:20.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: dump: 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 5 monclient: authenticate success, global_id 5538 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:20.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf8c001b30 con 0x7fbfa40a4d90 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 1 -- 172.21.15.63:0/2296880214 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf8c001e80 con 0x7fbfa40a4d90 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa159a640 10 monclient: dump: 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.027 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 4 set_mon_vals no callback set 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 10 monclient: shutdown 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- 172.21.15.63:0/2296880214 >> v1:172.21.15.6:6789/0 conn(0x7fbfa40a4d90 legacy=0x7fbfa413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- 172.21.15.63:0/2296880214 shutdown_connections 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- 172.21.15.63:0/2296880214 >> 172.21.15.63:0/2296880214 conn(0x7fbfa4075470 msgr2=0x7fbfa4075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- 172.21.15.63:0/2296880214 shutdown_connections 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.014+0000 7fbfa8e29640 1 -- 172.21.15.63:0/2296880214 wait complete. 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: build_initial_monmap 2024-09-16T05:38:20.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: monmap: 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 1 Processor -- start 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 1 -- start start 2024-09-16T05:38:20.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: init 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient(hunting): picked mon.noname-b con 0x7fbfa40a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient(hunting): picked mon.noname-a con 0x7fbfa40a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient(hunting): picked mon.noname-c con 0x7fbfa40a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa41368d0 con 0x7fbfa40a4790 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa4137ed0 con 0x7fbfa40a3ca0 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa40a8910 con 0x7fbfa40a4d90 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient(hunting): authenticate will time out at 1156.796905s 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbfa40a3ca0 legacy=0x7fbfa4134080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:20.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa2d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbfa40a4d90 0x7fbfa4134ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40868/0 (socket says 172.21.15.63:40868) 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa1d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbfa40a4790 0x7fbfa4134790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48216/0 (socket says 172.21.15.63:48216) 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa2d9d640 1 -- 172.21.15.63:0/1881304989 learned_addr learned my addr 172.21.15.63:0/1881304989 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269152218 0 0) 0x7fbfa40a8910 con 0x7fbfa40a4d90 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): my global_id is 5525 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf84003530 con 0x7fbfa40a4d90 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516126375 0 0) 0x7fbfa40a0810 con 0x7fbfa40a4790 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:20.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): my global_id is 5541 2024-09-16T05:38:20.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfa40a8910 con 0x7fbfa40a4790 2024-09-16T05:38:20.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 135232643 0 0) 0x7fbf84003530 con 0x7fbfa40a4d90 2024-09-16T05:38:20.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfa40a0810 con 0x7fbfa40a4d90 2024-09-16T05:38:20.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf94004040 con 0x7fbfa40a4d90 2024-09-16T05:38:20.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:20.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:20.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): dump: 2024-09-16T05:38:20.034 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1927579125 0 0) 0x7fbfa40a8910 con 0x7fbfa40a4790 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf84003530 con 0x7fbfa40a4790 2024-09-16T05:38:20.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf8c0017d0 con 0x7fbfa40a4790 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient(hunting): dump: 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.036 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 395680171 0 0) 0x7fbf84003530 con 0x7fbfa40a4790 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 >> v1:172.21.15.6:6790/0 conn(0x7fbfa40a4d90 legacy=0x7fbfa4134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 >> v1:172.21.15.63:6789/0 conn(0x7fbfa40a3ca0 legacy=0x7fbfa4134080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 monclient: found mon.a 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa40a83e0 con 0x7fbfa40a4790 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient: _finish_auth 0 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:20.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 5 monclient: authenticate success, global_id 5541 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: _renew_subs 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa40a0500 con 0x7fbfa40a4790 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: _renew_subs 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbfa41381b0 con 0x7fbfa40a4790 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf8c001a80 con 0x7fbfa40a4790 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa0d99640 4 set_mon_vals no callback set 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf8c004fc0 con 0x7fbfa40a4790 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:20.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbf9affd640 10 monclient: dump: 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:20.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:20.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.018+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf88005090 con 0x7fbfa40a4790 2024-09-16T05:38:20.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.022+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbf8c00fd60 con 0x7fbfa40a4790 2024-09-16T05:38:20.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.022+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(127..127 src has 1..127) ==== 6449+0+0 (unknown 1371578255 0 0) 0x7fbf8c044fc0 con 0x7fbfa40a4790 2024-09-16T05:38:20.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.022+0000 7fbfa259c640 10 monclient: get_auth_request con 0x7fbf840369d0 auth_method 0 2024-09-16T05:38:20.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.030+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbfa41368d0 con 0x7fbfa40a4790 2024-09-16T05:38:20.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.030+0000 7fbf9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:20.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.030+0000 7fbf9affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:20.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.198+0000 7fbfa8e29640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T05:38:20.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.198+0000 7fbfa8e29640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T05:38:20.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.198+0000 7fbfa8e29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:20.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.198+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) -- 0x7fbf88005380 con 0x7fbfa40a4790 2024-09-16T05:38:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.550+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 v128) ==== 109+0+0 (unknown 3200147039 0 0) 0x7fbf8c015820 con 0x7fbfa40a4790 2024-09-16T05:38:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.550+0000 7fbf9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T05:38:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.550+0000 7fbf9affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.658+0000 7fbfa8e29640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T05:38:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.658+0000 7fbfa8e29640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T05:38:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.658+0000 7fbfa8e29640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:20.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.658+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) -- 0x7fbf88005be0 con 0x7fbfa40a4790 2024-09-16T05:38:20.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbf9affd640 1 -- 172.21.15.63:0/1881304989 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 weight-set for pool 'fooset' already created v128) ==== 153+0+0 (unknown 1458925521 0 0) 0x7fbf8c015120 con 0x7fbfa40a4790 2024-09-16T05:38:20.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbf9affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T05:38:20.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbf9affd640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'fooset' already created 2024-09-16T05:38:20.665 INFO:tasks.workunit.client.0.smithi063.stderr:weight-set for pool 'fooset' already created 2024-09-16T05:38:20.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbf840369d0 legacy=0x7fbf84038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:20.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbfa8e29640 10 monclient: shutdown 2024-09-16T05:38:20.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbfa8e29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:20.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 >> v1:172.21.15.6:6789/0 conn(0x7fbfa40a4790 legacy=0x7fbfa4134790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:20.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 shutdown_connections 2024-09-16T05:38:20.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 >> 172.21.15.63:0/1881304989 conn(0x7fbfa4075470 msgr2=0x7fbfa40a70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:20.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 shutdown_connections 2024-09-16T05:38:20.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.662+0000 7fbfa8e29640 1 -- 172.21.15.63:0/1881304989 wait complete. 2024-09-16T05:38:20.682 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set create barset positional 2024-09-16T05:38:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient: get_monmap_and_config 2024-09-16T05:38:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient: build_initial_monmap 2024-09-16T05:38:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient: monmap: 2024-09-16T05:38:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.805 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:20.805046+0000 2024-09-16T05:38:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:20.805046+0000 2024-09-16T05:38:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:20.806 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 1 Processor -- start 2024-09-16T05:38:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 1 -- start start 2024-09-16T05:38:20.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient: init 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b14098500 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b14075770 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b1413a9b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b14135250 con 0x7f0b1413a9b0 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1409c940 con 0x7f0b14075770 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1409cc20 con 0x7f0b14098500 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b19d3d640 10 monclient(hunting): authenticate will time out at 1157.582894s 2024-09-16T05:38:20.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b12ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0b14075770 legacy=0x7f0b14134b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b137fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0b14098500 0x7f0b140988e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40874/0 (socket says 172.21.15.63:40874) 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b137fe640 1 -- 172.21.15.63:0/2146931479 learned_addr learned my addr 172.21.15.63:0/2146931479 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3396403757 0 0) 0x7f0b14135250 con 0x7f0b1413a9b0 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 10 monclient(hunting): my global_id is 5544 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0afc003530 con 0x7f0b1413a9b0 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135975530 0 0) 0x7f0b1409cc20 con 0x7f0b14098500 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:20.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 10 monclient(hunting): my global_id is 5528 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.802+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b14135250 con 0x7f0b14098500 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2858751007 0 0) 0x7f0afc003530 con 0x7f0b1413a9b0 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 >> v1:172.21.15.6:6790/0 conn(0x7f0b14098500 legacy=0x7f0b140988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 >> v1:172.21.15.63:6789/0 conn(0x7f0b14075770 legacy=0x7f0b14134b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 1 monclient: found mon.noname-a 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1409cf00 con 0x7f0b1413a9b0 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:20.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 5 monclient: authenticate success, global_id 5544 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b0c0019a0 con 0x7f0b1413a9b0 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: dump: 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.813 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b0c001cc0 con 0x7f0b1413a9b0 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 1 -- 172.21.15.63:0/2146931479 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b0c002090 con 0x7f0b1413a9b0 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 4 set_mon_vals no callback set 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b127fc640 10 monclient: dump: 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.814 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient: shutdown 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- 172.21.15.63:0/2146931479 >> v1:172.21.15.6:6789/0 conn(0x7f0b1413a9b0 legacy=0x7f0b1413cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- 172.21.15.63:0/2146931479 shutdown_connections 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- 172.21.15.63:0/2146931479 >> 172.21.15.63:0/2146931479 conn(0x7f0b140a2110 msgr2=0x7f0b140a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- 172.21.15.63:0/2146931479 shutdown_connections 2024-09-16T05:38:20.815 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- 172.21.15.63:0/2146931479 wait complete. 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient: build_initial_monmap 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient: monmap: 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:20.816 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 Processor -- start 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- start start 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient: init 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b14075770 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b14098500 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b1413a9b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b14071240 con 0x7f0b1413a9b0 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b140a1970 con 0x7f0b14098500 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b14073530 con 0x7f0b14075770 2024-09-16T05:38:20.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b19d3d640 10 monclient(hunting): authenticate will time out at 1157.586443s 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b12ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0b14098500 legacy=0x7f0b141d09a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b13fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b1413a9b0 0x7f0b141d40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48228/0 (socket says 172.21.15.63:48228) 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b13fff640 1 -- 172.21.15.63:0/3503011751 learned_addr learned my addr 172.21.15.63:0/3503011751 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0b137fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0b14075770 0x7f0b14097300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40886/0 (socket says 172.21.15.63:40886) 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3461357292 0 0) 0x7f0b14073530 con 0x7f0b14075770 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): my global_id is 5531 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b00003530 con 0x7f0b14075770 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677519035 0 0) 0x7f0b14071240 con 0x7f0b1413a9b0 2024-09-16T05:38:20.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): my global_id is 5547 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b14073530 con 0x7f0b1413a9b0 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2498454043 0 0) 0x7f0b00003530 con 0x7f0b14075770 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b14071240 con 0x7f0b14075770 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b040018c0 con 0x7f0b14075770 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): dump: 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.822 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2020135448 0 0) 0x7f0b14073530 con 0x7f0b1413a9b0 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b00003530 con 0x7f0b1413a9b0 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b0c0017c0 con 0x7f0b1413a9b0 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.806+0000 7f0afbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:20.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient(hunting): dump: 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 39658075 0 0) 0x7f0b14071240 con 0x7f0b14075770 2024-09-16T05:38:20.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 >> v1:172.21.15.63:6789/0 conn(0x7f0b14098500 legacy=0x7f0b141d09a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 >> v1:172.21.15.6:6789/0 conn(0x7f0b1413a9b0 legacy=0x7f0b141d40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 monclient: found mon.c 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b140716e0 con 0x7f0b14075770 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient: _finish_auth 0 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 5 monclient: authenticate success, global_id 5531 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 10 monclient: _renew_subs 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b14099130 con 0x7f0b14075770 2024-09-16T05:38:20.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 10 monclient: _renew_subs 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b1413dd80 con 0x7f0b14075770 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b040047f0 con 0x7f0b14075770 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b04004b40 con 0x7f0b14075770 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 10 monclient: dump: 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:20.826 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b11ffb640 4 set_mon_vals no callback set 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0af4005090 con 0x7f0b14075770 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0b0400f8e0 con 0x7f0b14075770 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(128..128 src has 1..128) ==== 6781+0+0 (unknown 2132386453 0 0) 0x7f0b04044d60 con 0x7f0b14075770 2024-09-16T05:38:20.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.810+0000 7f0b12ffd640 10 monclient: get_auth_request con 0x7f0b00036aa0 auth_method 0 2024-09-16T05:38:20.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.822+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0b040153c0 con 0x7f0b14075770 2024-09-16T05:38:20.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.822+0000 7f0afbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:20.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.822+0000 7f0afbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:20.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.986+0000 7f0b19d3d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T05:38:20.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.986+0000 7f0b19d3d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T05:38:20.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.986+0000 7f0b19d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:20.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:20.986+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) -- 0x7f0af4005380 con 0x7f0b14075770 2024-09-16T05:38:21.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.010+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 v129) ==== 115+0+0 (unknown 298627773 0 0) 0x7f0b04014cc0 con 0x7f0b14075770 2024-09-16T05:38:21.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.010+0000 7f0afbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T05:38:21.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.010+0000 7f0afbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.110+0000 7f0b19d3d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T05:38:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.110+0000 7f0b19d3d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T05:38:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.110+0000 7f0b19d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.110+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) -- 0x7f0af4005ac0 con 0x7f0b14075770 2024-09-16T05:38:21.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0afbfff640 1 -- 172.21.15.63:0/3503011751 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 weight-set for pool 'barset' already created v129) ==== 159+0+0 (unknown 1792178642 0 0) 0x7f0b0401d870 con 0x7f0b14075770 2024-09-16T05:38:21.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0afbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T05:38:21.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0afbfff640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'barset' already created 2024-09-16T05:38:21.117 INFO:tasks.workunit.client.0.smithi063.stderr:weight-set for pool 'barset' already created 2024-09-16T05:38:21.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0b00036aa0 legacy=0x7f0b00038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0b19d3d640 10 monclient: shutdown 2024-09-16T05:38:21.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0b19d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:21.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 >> v1:172.21.15.6:6790/0 conn(0x7f0b14075770 legacy=0x7f0b14097300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 shutdown_connections 2024-09-16T05:38:21.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 >> 172.21.15.63:0/3503011751 conn(0x7f0b140a2110 msgr2=0x7f0b1409efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 shutdown_connections 2024-09-16T05:38:21.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.114+0000 7f0b19d3d640 1 -- 172.21.15.63:0/3503011751 wait complete. 2024-09-16T05:38:21.133 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set ls 2024-09-16T05:38:21.133 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep fooset 2024-09-16T05:38:21.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.246+0000 7f2025042640 10 monclient: get_monmap_and_config 2024-09-16T05:38:21.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.246+0000 7f2025042640 10 monclient: build_initial_monmap 2024-09-16T05:38:21.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.246+0000 7f2025042640 10 monclient: monmap: 2024-09-16T05:38:21.252 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:21.252 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.252 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:21.251732+0000 2024-09-16T05:38:21.252 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:21.251732+0000 2024-09-16T05:38:21.253 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:21.253 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.253 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:21.253 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:21.253 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:21.253 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 Processor -- start 2024-09-16T05:38:21.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- start start 2024-09-16T05:38:21.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient: init 2024-09-16T05:38:21.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:21.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:21.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient(hunting): picked mon.noname-a con 0x7f20200a12b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:21.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient(hunting): picked mon.noname-b con 0x7f2020072fa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:21.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient(hunting): picked mon.noname-c con 0x7f20201395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20200a05b0 con 0x7f20200a12b0 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202009c510 con 0x7f20201395a0 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202009ce10 con 0x7f2020072fa0 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient(hunting): authenticate will time out at 1158.028855s 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201f59e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f20201395a0 legacy=0x7f202013b990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f20200a12b0 0x7f202009f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48230/0 (socket says 172.21.15.63:48230) 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201ed9d640 1 -- 172.21.15.63:0/2781309865 learned_addr learned my addr 172.21.15.63:0/2781309865 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3099243396 0 0) 0x7f20200a05b0 con 0x7f20200a12b0 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:21.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient(hunting): my global_id is 5550 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2008003530 con 0x7f20200a12b0 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3630332711 0 0) 0x7f202009ce10 con 0x7f2020072fa0 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient(hunting): my global_id is 5534 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20200a05b0 con 0x7f2020072fa0 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1682591751 0 0) 0x7f20200a05b0 con 0x7f2020072fa0 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 >> v1:172.21.15.63:6789/0 conn(0x7f20201395a0 legacy=0x7f202013b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 >> v1:172.21.15.6:6789/0 conn(0x7f20200a12b0 legacy=0x7f202009f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 monclient: found mon.noname-b 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202009d0f0 con 0x7f2020072fa0 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: _finish_auth 0 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2014001600 con 0x7f2020072fa0 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:21.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: dump: 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 5 monclient: authenticate success, global_id 5534 2024-09-16T05:38:21.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20140018f0 con 0x7f2020072fa0 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 1 -- 172.21.15.63:0/2781309865 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2014001d00 con 0x7f2020072fa0 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f201dd9b640 10 monclient: dump: 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.261 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 4 set_mon_vals no callback set 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient: shutdown 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- 172.21.15.63:0/2781309865 >> v1:172.21.15.6:6790/0 conn(0x7f2020072fa0 legacy=0x7f202009fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- 172.21.15.63:0/2781309865 shutdown_connections 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- 172.21.15.63:0/2781309865 >> 172.21.15.63:0/2781309865 conn(0x7f20200a1c90 msgr2=0x7f20200a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- 172.21.15.63:0/2781309865 shutdown_connections 2024-09-16T05:38:21.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 1 -- 172.21.15.63:0/2781309865 wait complete. 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient: build_initial_monmap 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.250+0000 7f2025042640 10 monclient: monmap: 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:21.267 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 1 Processor -- start 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 1 -- start start 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: init 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient(hunting): picked mon.noname-a con 0x7f2020072fa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient(hunting): picked mon.noname-c con 0x7f20200a12b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient(hunting): picked mon.noname-b con 0x7f20201395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2020123440 con 0x7f2020072fa0 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2020123160 con 0x7f20201395a0 2024-09-16T05:38:21.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2020123720 con 0x7f20200a12b0 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient(hunting): authenticate will time out at 1158.032993s 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f201f59e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f20201395a0 legacy=0x7f20201209e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f201ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2020072fa0 0x7f202011fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48232/0 (socket says 172.21.15.63:48232) 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f201e59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f20200a12b0 0x7f20201202d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40896/0 (socket says 172.21.15.63:40896) 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f201ed9d640 1 -- 172.21.15.63:0/3941901563 learned_addr learned my addr 172.21.15.63:0/3941901563 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4192947217 0 0) 0x7f2020123720 con 0x7f20200a12b0 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): my global_id is 5537 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f200c003560 con 0x7f20200a12b0 2024-09-16T05:38:21.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2503297124 0 0) 0x7f2020123440 con 0x7f2020072fa0 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): my global_id is 5553 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2020123720 con 0x7f2020072fa0 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 658571166 0 0) 0x7f200c003560 con 0x7f20200a12b0 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2020123440 con 0x7f20200a12b0 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20140015c0 con 0x7f20200a12b0 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): dump: 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.270 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2086294925 0 0) 0x7f2020123720 con 0x7f2020072fa0 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f200c003560 con 0x7f2020072fa0 2024-09-16T05:38:21.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2010001a40 con 0x7f2020072fa0 2024-09-16T05:38:21.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:21.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:21.272 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient(hunting): dump: 2024-09-16T05:38:21.272 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.272 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3647886423 0 0) 0x7f2020123440 con 0x7f20200a12b0 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 >> v1:172.21.15.63:6789/0 conn(0x7f20201395a0 legacy=0x7f20201209e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 >> v1:172.21.15.6:6789/0 conn(0x7f2020072fa0 legacy=0x7f202011fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 monclient: found mon.c 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2020122270 con 0x7f20200a12b0 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 5 monclient: authenticate success, global_id 5537 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: _renew_subs 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20201d47c0 con 0x7f20200a12b0 2024-09-16T05:38:21.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: _renew_subs 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f20201d4b60 con 0x7f20200a12b0 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20140018f0 con 0x7f20200a12b0 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2014003ce0 con 0x7f20200a12b0 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f201d59a640 4 set_mon_vals no callback set 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f20077fe640 10 monclient: dump: 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.275 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.254+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2000005090 con 0x7f20200a12b0 2024-09-16T05:38:21.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.258+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2014003f80 con 0x7f20200a12b0 2024-09-16T05:38:21.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.258+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(129..129 src has 1..129) ==== 7393+0+0 (unknown 2053469478 0 0) 0x7f20140450f0 con 0x7f20200a12b0 2024-09-16T05:38:21.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.258+0000 7f201ed9d640 10 monclient: get_auth_request con 0x7f200c036a50 auth_method 0 2024-09-16T05:38:21.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.270+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2014015540 con 0x7f20200a12b0 2024-09-16T05:38:21.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.270+0000 7f20077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:21.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.270+0000 7f20077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.442+0000 7f2025042640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.442+0000 7f2025042640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.442+0000 7f2025042640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.442+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f2000005380 con 0x7f20200a12b0 2024-09-16T05:38:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.442+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v129) ==== 71+0+14 (unknown 417425193 0 3003798303) 0x7f2014014e40 con 0x7f20200a12b0 2024-09-16T05:38:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.442+0000 7f20077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.442+0000 7f20077fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:21.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.546+0000 7f2025042640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.546+0000 7f2025042640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.546+0000 7f2025042640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.546+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f2000005ac0 con 0x7f20200a12b0 2024-09-16T05:38:21.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.546+0000 7f20077fe640 1 -- 172.21.15.63:0/3941901563 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v129) ==== 71+0+14 (unknown 417425193 0 3003798303) 0x7f201401d9f0 con 0x7f20200a12b0 2024-09-16T05:38:21.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.546+0000 7f20077fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.546+0000 7f20077fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:21.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.550+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 >> v1:172.21.15.63:6812/3643207211 conn(0x7f200c036a50 legacy=0x7f200c038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.550+0000 7f2025042640 10 monclient: shutdown 2024-09-16T05:38:21.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.550+0000 7f2025042640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:21.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.550+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 >> v1:172.21.15.6:6790/0 conn(0x7f20200a12b0 legacy=0x7f20201202d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.550+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 shutdown_connections 2024-09-16T05:38:21.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.550+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 >> 172.21.15.63:0/3941901563 conn(0x7f20200a1c90 msgr2=0x7f2020075410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:21.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.550+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 shutdown_connections 2024-09-16T05:38:21.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.550+0000 7f2025042640 1 -- 172.21.15.63:0/3941901563 wait complete. 2024-09-16T05:38:21.569 INFO:tasks.workunit.client.0.smithi063.stdout:fooset 2024-09-16T05:38:21.570 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set ls 2024-09-16T05:38:21.570 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep barset 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.686+0000 7fccf8908640 10 monclient: get_monmap_and_config 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.686+0000 7fccf8908640 10 monclient: build_initial_monmap 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.686+0000 7fccf8908640 10 monclient: monmap: 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:21.690798+0000 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:21.690798+0000 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:21.691 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.692 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:21.692 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:21.692 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:21.692 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.686+0000 7fccf8908640 1 Processor -- start 2024-09-16T05:38:21.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.686+0000 7fccf8908640 1 -- start start 2024-09-16T05:38:21.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.686+0000 7fccf8908640 10 monclient: init 2024-09-16T05:38:21.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:21.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:21.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient(hunting): picked mon.noname-a con 0x7fccf40a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient(hunting): picked mon.noname-b con 0x7fccf409b450 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient(hunting): picked mon.noname-c con 0x7fccf409b8a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf409e970 con 0x7fccf40a08a0 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf409c370 con 0x7fccf409b8a0 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf409d620 con 0x7fccf409b450 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient(hunting): authenticate will time out at 1158.468343s 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf2d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fccf409b8a0 legacy=0x7fccf413c0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf1d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fccf409b450 0x7fccf41389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40898/0 (socket says 172.21.15.63:40898) 2024-09-16T05:38:21.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf1d9b640 1 -- 172.21.15.63:0/838269138 learned_addr learned my addr 172.21.15.63:0/838269138 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:21.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3873862801 0 0) 0x7fccf409e970 con 0x7fccf40a08a0 2024-09-16T05:38:21.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:21.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:21.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:21.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient(hunting): my global_id is 5556 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccdc003530 con 0x7fccf40a08a0 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341378659 0 0) 0x7fccf409d620 con 0x7fccf409b450 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient(hunting): my global_id is 5540 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccf409e970 con 0x7fccf409b450 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 293987446 0 0) 0x7fccdc003530 con 0x7fccf40a08a0 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 >> v1:172.21.15.6:6790/0 conn(0x7fccf409b450 legacy=0x7fccf41389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 >> v1:172.21.15.63:6789/0 conn(0x7fccf409b8a0 legacy=0x7fccf413c0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 monclient: found mon.noname-a 2024-09-16T05:38:21.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf409d900 con 0x7fccf40a08a0 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: _finish_auth 0 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcce40019a0 con 0x7fccf40a08a0 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: dump: 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.704 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 5 monclient: authenticate success, global_id 5556 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcce4001c00 con 0x7fccf40a08a0 2024-09-16T05:38:21.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 1 -- 172.21.15.63:0/838269138 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcce4001f90 con 0x7fccf40a08a0 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf159a640 10 monclient: dump: 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.706 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 4 set_mon_vals no callback set 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient: shutdown 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- 172.21.15.63:0/838269138 >> v1:172.21.15.6:6789/0 conn(0x7fccf40a08a0 legacy=0x7fccf409ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- 172.21.15.63:0/838269138 shutdown_connections 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- 172.21.15.63:0/838269138 >> 172.21.15.63:0/838269138 conn(0x7fccf40a1c90 msgr2=0x7fccf40a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- 172.21.15.63:0/838269138 shutdown_connections 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- 172.21.15.63:0/838269138 wait complete. 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient: build_initial_monmap 2024-09-16T05:38:21.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient: monmap: 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 Processor -- start 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 1 -- start start 2024-09-16T05:38:21.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.690+0000 7fccf8908640 10 monclient: init 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient(hunting): picked mon.noname-b con 0x7fccf409b450 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient(hunting): picked mon.noname-c con 0x7fccf409b8a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient(hunting): picked mon.noname-a con 0x7fccf40a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf4122500 con 0x7fccf40a08a0 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf40a3ed0 con 0x7fccf409b450 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf40a43e0 con 0x7fccf409b8a0 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fccf409b450 legacy=0x7fccf4133b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient(hunting): authenticate will time out at 1158.472394s 2024-09-16T05:38:21.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf1d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fccf409b8a0 0x7fccf41cbef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40906/0 (socket says 172.21.15.63:40906) 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf1d9b640 1 -- 172.21.15.63:0/1287181112 learned_addr learned my addr 172.21.15.63:0/1287181112 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2905092137 0 0) 0x7fccf40a43e0 con 0x7fccf409b8a0 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): my global_id is 5543 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd4003560 con 0x7fccf409b8a0 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2412988958 0 0) 0x7fccf4122500 con 0x7fccf40a08a0 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): my global_id is 5559 2024-09-16T05:38:21.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccf40a43e0 con 0x7fccf40a08a0 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2954094537 0 0) 0x7fccd4003560 con 0x7fccf409b8a0 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fccf4122500 con 0x7fccf409b8a0 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3144978854 0 0) 0x7fccf40a43e0 con 0x7fccf40a08a0 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fccd4003560 con 0x7fccf40a08a0 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcce8001860 con 0x7fccf409b8a0 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.711 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fccec003f40 con 0x7fccf40a08a0 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.712 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1364140430 0 0) 0x7fccf4122500 con 0x7fccf409b8a0 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 >> v1:172.21.15.63:6789/0 conn(0x7fccf409b450 legacy=0x7fccf4133b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 >> v1:172.21.15.6:6789/0 conn(0x7fccf40a08a0 legacy=0x7fccf41cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 monclient: found mon.c 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf4121f40 con 0x7fccf409b8a0 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 5 monclient: authenticate success, global_id 5543 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient: _renew_subs 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fccf4122220 con 0x7fccf409b8a0 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient: _renew_subs 2024-09-16T05:38:21.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fccf41c6e30 con 0x7fccf409b8a0 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcce80047f0 con 0x7fccf409b8a0 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcce8004b40 con 0x7fccf409b8a0 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf0d99640 4 set_mon_vals no callback set 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 10 monclient: dump: 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:21.715 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccd8005090 con 0x7fccf409b8a0 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcce800f8e0 con 0x7fccf409b8a0 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.694+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(129..129 src has 1..129) ==== 7393+0+0 (unknown 2053469478 0 0) 0x7fcce8044fa0 con 0x7fccf409b8a0 2024-09-16T05:38:21.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.698+0000 7fccf259c640 10 monclient: get_auth_request con 0x7fccd40369f0 auth_method 0 2024-09-16T05:38:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.706+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fcce8018f80 con 0x7fccf409b8a0 2024-09-16T05:38:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.706+0000 7fcce2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:21.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.706+0000 7fcce2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:21.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.882+0000 7fccf8908640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.882+0000 7fccf8908640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.882+0000 7fccf8908640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.882+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fccd8005380 con 0x7fccf409b8a0 2024-09-16T05:38:21.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.882+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v129) ==== 71+0+14 (unknown 417425193 0 3003798303) 0x7fcce801d840 con 0x7fccf409b8a0 2024-09-16T05:38:21.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.882+0000 7fcce2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.882+0000 7fcce2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:21.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.986+0000 7fccf8908640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.986+0000 7fccf8908640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.986+0000 7fccf8908640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:21.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.986+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fccd8005ac0 con 0x7fccf409b8a0 2024-09-16T05:38:21.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fcce2ffd640 1 -- 172.21.15.63:0/1287181112 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v129) ==== 71+0+14 (unknown 417425193 0 3003798303) 0x7fcce80403c0 con 0x7fccf409b8a0 2024-09-16T05:38:21.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fcce2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:21.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fcce2ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:21.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 >> v1:172.21.15.63:6812/3643207211 conn(0x7fccd40369f0 legacy=0x7fccd4038eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fccf8908640 10 monclient: shutdown 2024-09-16T05:38:21.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fccf8908640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:21.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 >> v1:172.21.15.6:6790/0 conn(0x7fccf409b8a0 legacy=0x7fccf41cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:21.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 shutdown_connections 2024-09-16T05:38:21.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 >> 172.21.15.63:0/1287181112 conn(0x7fccf40a1c90 msgr2=0x7fccf41b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:21.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 shutdown_connections 2024-09-16T05:38:21.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:21.990+0000 7fccf8908640 1 -- 172.21.15.63:0/1287181112 wait complete. 2024-09-16T05:38:22.010 INFO:tasks.workunit.client.0.smithi063.stdout:barset 2024-09-16T05:38:22.010 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set dump 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient: get_monmap_and_config 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient: build_initial_monmap 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient: monmap: 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:22.130679+0000 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:22.130679+0000 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:22.131 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.132 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:22.132 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:22.132 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:22.132 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 1 Processor -- start 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 1 -- start start 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient: init 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc4f809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc4f806dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc4f80a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:22.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f809f800 con 0x7fc4f80a0210 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f80a0ab0 con 0x7fc4f809ac90 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f80a0d90 con 0x7fc4f806dd30 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.126+0000 7fc4fedc6640 10 monclient(hunting): authenticate will time out at 1158.907970s 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fcb62640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc4f809ac90 legacy=0x7fc4f809b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc4f806dd30 0x7fc4f81399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40918/0 (socket says 172.21.15.63:40918) 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f7fff640 1 -- 172.21.15.63:0/3199673550 learned_addr learned my addr 172.21.15.63:0/3199673550 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1005935673 0 0) 0x7fc4f80a0d90 con 0x7fc4f806dd30 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient(hunting): my global_id is 5546 2024-09-16T05:38:22.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4e4003530 con 0x7fc4f806dd30 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3881782307 0 0) 0x7fc4f809f800 con 0x7fc4f80a0210 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient(hunting): my global_id is 5562 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4f80a0d90 con 0x7fc4f80a0210 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4047679499 0 0) 0x7fc4e4003530 con 0x7fc4f806dd30 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 >> v1:172.21.15.63:6789/0 conn(0x7fc4f809ac90 legacy=0x7fc4f809b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 >> v1:172.21.15.6:6789/0 conn(0x7fc4f80a0210 legacy=0x7fc4f813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 monclient: found mon.noname-b 2024-09-16T05:38:22.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f809e770 con 0x7fc4f806dd30 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4e0001620 con 0x7fc4f806dd30 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: dump: 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.138 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 5 monclient: authenticate success, global_id 5546 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc4e0001880 con 0x7fc4f806dd30 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 1 -- 172.21.15.63:0/3199673550 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4e0001c90 con 0x7fc4f806dd30 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:22.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4f77fe640 10 monclient: dump: 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:22.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 4 set_mon_vals no callback set 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient: shutdown 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- 172.21.15.63:0/3199673550 >> v1:172.21.15.6:6790/0 conn(0x7fc4f806dd30 legacy=0x7fc4f81399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- 172.21.15.63:0/3199673550 shutdown_connections 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- 172.21.15.63:0/3199673550 >> 172.21.15.63:0/3199673550 conn(0x7fc4f80a1c90 msgr2=0x7fc4f80a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- 172.21.15.63:0/3199673550 shutdown_connections 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- 172.21.15.63:0/3199673550 wait complete. 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient: build_initial_monmap 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient: monmap: 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:22.141 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:22.142 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:22.142 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:22.142 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.142 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:22.142 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:22.142 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:22.142 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 Processor -- start 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- start start 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient: init 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc4f806dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc4f809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc4f80a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fcb62640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc4f806dd30 legacy=0x7fc4f81c8bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f8125250 con 0x7fc4f809ac90 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f81236e0 con 0x7fc4f806dd30 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f80761e0 con 0x7fc4f80a0210 2024-09-16T05:38:22.146 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.130+0000 7fc4fedc6640 10 monclient(hunting): authenticate will time out at 1158.911821s 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fd363640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc4f80a0210 0x7fc4f81d0f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40934/0 (socket says 172.21.15.63:40934) 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fd363640 1 -- 172.21.15.63:0/285709444 learned_addr learned my addr 172.21.15.63:0/285709444 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618165258 0 0) 0x7fc4f8125250 con 0x7fc4f809ac90 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): my global_id is 5565 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4d8003530 con 0x7fc4f809ac90 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3928996984 0 0) 0x7fc4f80761e0 con 0x7fc4f80a0210 2024-09-16T05:38:22.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): my global_id is 5549 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4f8125250 con 0x7fc4f80a0210 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 452174746 0 0) 0x7fc4d8003530 con 0x7fc4f809ac90 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc4f80761e0 con 0x7fc4f809ac90 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4e0001c10 con 0x7fc4f809ac90 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): dump: 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.148 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2189882876 0 0) 0x7fc4f8125250 con 0x7fc4f80a0210 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc4d8003530 con 0x7fc4f80a0210 2024-09-16T05:38:22.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4e8001a20 con 0x7fc4f80a0210 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient(hunting): dump: 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.150 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2419472081 0 0) 0x7fc4f80761e0 con 0x7fc4f809ac90 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 >> v1:172.21.15.6:6790/0 conn(0x7fc4f80a0210 legacy=0x7fc4f81d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 >> v1:172.21.15.63:6789/0 conn(0x7fc4f806dd30 legacy=0x7fc4f81c8bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient: _finish_hunting 0 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 monclient: found mon.a 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f809c0f0 con 0x7fc4f809ac90 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient: _finish_auth 0 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 5 monclient: authenticate success, global_id 5565 2024-09-16T05:38:22.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 10 monclient: _renew_subs 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f806e3f0 con 0x7fc4f809ac90 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 10 monclient: _renew_subs 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc4f8135250 con 0x7fc4f809ac90 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc4e0001840 con 0x7fc4f809ac90 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f6ffd640 4 set_mon_vals no callback set 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4e0004c40 con 0x7fc4f809ac90 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 10 monclient: dump: 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.152 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4dc005090 con 0x7fc4f809ac90 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc4e0004ea0 con 0x7fc4f809ac90 2024-09-16T05:38:22.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(129..129 src has 1..129) ==== 7393+0+0 (unknown 2053469478 0 0) 0x7fc4e0044f30 con 0x7fc4f809ac90 2024-09-16T05:38:22.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.134+0000 7fc4fcb62640 10 monclient: get_auth_request con 0x7fc4d8036980 auth_method 0 2024-09-16T05:38:22.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.146+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc4e00154a0 con 0x7fc4f809ac90 2024-09-16T05:38:22.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.146+0000 7fc4f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:22.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.146+0000 7fc4f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:22.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.318+0000 7fc4fedc6640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:22.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.318+0000 7fc4fedc6640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:22.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.318+0000 7fc4fedc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:22.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.318+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) -- 0x7fc4dc005380 con 0x7fc4f809ac90 2024-09-16T05:38:22.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.318+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v129) ==== 73+0+6748 (unknown 1146636505 0 3407865348) 0x7fc4e0014da0 con 0x7fc4f809ac90 2024-09-16T05:38:22.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.318+0000 7fc4f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:22.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.318+0000 7fc4f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:22.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.422+0000 7fc4fedc6640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:22.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.422+0000 7fc4fedc6640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:22.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.422+0000 7fc4fedc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:22.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.422+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) -- 0x7fc4dc0071d0 con 0x7fc4f809ac90 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4f57fa640 1 -- 172.21.15.63:0/285709444 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v129) ==== 73+0+6748 (unknown 1146636505 0 3407865348) 0x7fc4e00403b0 con 0x7fc4f809ac90 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4f57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4f57fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stdout:{ 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stdout: "9": [ 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -1, 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.429 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: 0.263671875, 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: 0.263671875, 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -2, 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.430 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -3, 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.431 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -4, 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -5, 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -6, 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.433 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -7, 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.434 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -8, 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -9, 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.435 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -10, 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.436 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -11, 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -12, 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.437 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -13, 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -14, 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.438 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -15, 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: "10": [ 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -1, 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.439 INFO:tasks.workunit.client.0.smithi063.stdout: 0.263671875, 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0.263671875, 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0.263671875, 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0.263671875, 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0.263671875, 2024-09-16T05:38:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 0.263671875, 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -2, 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.441 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -3, 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.442 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.443 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -4, 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.444 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -5, 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.445 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.446 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -6, 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.448 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -7, 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.449 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -8, 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -9, 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.450 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: ], 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: [ 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: 0.087890625, 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: 0, 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: 0 2024-09-16T05:38:22.451 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -10, 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -11, 2024-09-16T05:38:22.452 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -12, 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.453 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -13, 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -14, 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.454 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: }, 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: { 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: "bucket_id": -15, 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: "weight_set": [ 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: [], 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: [] 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: } 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: ] 2024-09-16T05:38:22.455 INFO:tasks.workunit.client.0.smithi063.stdout:} 2024-09-16T05:38:22.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc4d8036980 legacy=0x7fc4d8038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:22.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4fedc6640 10 monclient: shutdown 2024-09-16T05:38:22.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4fedc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:22.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 >> v1:172.21.15.6:6789/0 conn(0x7fc4f809ac90 legacy=0x7fc4f8134b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:22.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 shutdown_connections 2024-09-16T05:38:22.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 >> 172.21.15.63:0/285709444 conn(0x7fc4f80a1c90 msgr2=0x7fc4f806e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:22.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 shutdown_connections 2024-09-16T05:38:22.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.426+0000 7fc4fedc6640 1 -- 172.21.15.63:0/285709444 wait complete. 2024-09-16T05:38:22.457 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T05:38:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient: get_monmap_and_config 2024-09-16T05:38:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient: build_initial_monmap 2024-09-16T05:38:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient: monmap: 2024-09-16T05:38:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:22.574044+0000 2024-09-16T05:38:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:22.574044+0000 2024-09-16T05:38:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:22.575 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:22.575 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 1 Processor -- start 2024-09-16T05:38:22.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 1 -- start start 2024-09-16T05:38:22.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient: init 2024-09-16T05:38:22.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:22.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:22.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient(hunting): picked mon.noname-c con 0x7f894c0a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient(hunting): picked mon.noname-a con 0x7f894c0a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient(hunting): picked mon.noname-b con 0x7f894c0a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894c0a53b0 con 0x7f894c0a07a0 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894c0a0290 con 0x7f894c0a3b10 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894c0a0dc0 con 0x7f894c0a4600 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f895276c640 10 monclient(hunting): authenticate will time out at 1159.351880s 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.570+0000 7f894bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f894c0a3b10 legacy=0x7f894c0a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f8950d09640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f894c0a4600 0x7f894c13f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40940/0 (socket says 172.21.15.63:40940) 2024-09-16T05:38:22.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f894c0a07a0 0x7f894c13be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:48284/0 (socket says 172.21.15.63:48284) 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f8950d09640 1 -- 172.21.15.63:0/2225180044 learned_addr learned my addr 172.21.15.63:0/2225180044 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 145877658 0 0) 0x7f894c0a0dc0 con 0x7f894c0a4600 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient(hunting): my global_id is 5552 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8934003530 con 0x7f894c0a4600 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529819042 0 0) 0x7f894c0a53b0 con 0x7f894c0a07a0 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:22.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient(hunting): my global_id is 5568 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f894c0a0dc0 con 0x7f894c0a07a0 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2594496311 0 0) 0x7f8934003530 con 0x7f894c0a4600 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 >> v1:172.21.15.63:6789/0 conn(0x7f894c0a3b10 legacy=0x7f894c0a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 >> v1:172.21.15.6:6789/0 conn(0x7f894c0a07a0 legacy=0x7f894c13be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 monclient: found mon.noname-b 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894c0a10a0 con 0x7f894c0a4600 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: _finish_auth 0 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:22.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f89440019a0 con 0x7f894c0a4600 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: dump: 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.582 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 5 monclient: authenticate success, global_id 5552 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8944001c50 con 0x7f894c0a4600 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 1 -- 172.21.15.63:0/2225180044 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8944002060 con 0x7f894c0a4600 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 4 set_mon_vals no callback set 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f894affd640 10 monclient: dump: 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.583 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient: shutdown 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- 172.21.15.63:0/2225180044 >> v1:172.21.15.6:6790/0 conn(0x7f894c0a4600 legacy=0x7f894c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- 172.21.15.63:0/2225180044 shutdown_connections 2024-09-16T05:38:22.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- 172.21.15.63:0/2225180044 >> 172.21.15.63:0/2225180044 conn(0x7f894c075470 msgr2=0x7f894c075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:22.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- 172.21.15.63:0/2225180044 shutdown_connections 2024-09-16T05:38:22.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- 172.21.15.63:0/2225180044 wait complete. 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient: build_initial_monmap 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient: monmap: 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:22.589 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 Processor -- start 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- start start 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient: init 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient(hunting): picked mon.noname-c con 0x7f894c0a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient(hunting): picked mon.noname-a con 0x7f894c0a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient(hunting): picked mon.noname-b con 0x7f894c0a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894c1259e0 con 0x7f894c0a3b10 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894c0a87f0 con 0x7f894c0a4600 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894c125540 con 0x7f894c0a07a0 2024-09-16T05:38:22.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f895276c640 10 monclient(hunting): authenticate will time out at 1159.355834s 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.574+0000 7f8950d09640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f894c0a4600 legacy=0x7f894c1d3840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f894bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f894c0a07a0 0x7f894c136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:40948/0 (socket says 172.21.15.63:40948) 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f894bfff640 1 -- 172.21.15.63:0/3414425402 learned_addr learned my addr 172.21.15.63:0/3414425402 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893137025 0 0) 0x7f894c125540 con 0x7f894c0a07a0 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): my global_id is 5555 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8938003530 con 0x7f894c0a07a0 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2059256377 0 0) 0x7f894c1259e0 con 0x7f894c0a3b10 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:22.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): my global_id is 5571 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f894c125540 con 0x7f894c0a3b10 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 425272192 0 0) 0x7f8938003530 con 0x7f894c0a07a0 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f894c1259e0 con 0x7f894c0a07a0 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f893c004160 con 0x7f894c0a07a0 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.592 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3036783155 0 0) 0x7f894c125540 con 0x7f894c0a3b10 2024-09-16T05:38:22.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8938003530 con 0x7f894c0a3b10 2024-09-16T05:38:22.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8940001a20 con 0x7f894c0a3b10 2024-09-16T05:38:22.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:22.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:22.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:22.594 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.594 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.594 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.594 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1869935493 0 0) 0x7f894c1259e0 con 0x7f894c0a07a0 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 >> v1:172.21.15.63:6789/0 conn(0x7f894c0a4600 legacy=0x7f894c1d3840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 >> v1:172.21.15.6:6789/0 conn(0x7f894c0a3b10 legacy=0x7f894c1d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 monclient: found mon.c 2024-09-16T05:38:22.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894c127830 con 0x7f894c0a07a0 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient: _finish_auth 0 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 5 monclient: authenticate success, global_id 5555 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 10 monclient: _renew_subs 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f894c0a82a0 con 0x7f894c0a07a0 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 10 monclient: _renew_subs 2024-09-16T05:38:22.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f894c137720 con 0x7f894c0a07a0 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f893c0043c0 con 0x7f894c0a07a0 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f893c004710 con 0x7f894c0a07a0 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 10 monclient: dump: 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:22.597 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f894a7fc640 4 set_mon_vals no callback set 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f892c005090 con 0x7f894c0a07a0 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f893c004930 con 0x7f894c0a07a0 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(129..129 src has 1..129) ==== 7393+0+0 (unknown 2053469478 0 0) 0x7f893c0450f0 con 0x7f894c0a07a0 2024-09-16T05:38:22.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.578+0000 7f894b7fe640 10 monclient: get_auth_request con 0x7f8938036a20 auth_method 0 2024-09-16T05:38:22.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.590+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f893c015480 con 0x7f894c0a07a0 2024-09-16T05:38:22.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.590+0000 7f8948ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:22.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.590+0000 7f8948ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:22.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.762+0000 7f895276c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:22.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.762+0000 7f895276c640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:22.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.762+0000 7f895276c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:22.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:22.762+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7f892c005380 con 0x7f894c0a07a0 2024-09-16T05:38:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:23.574+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v130) ==== 129+0+0 (unknown 743800201 0 0) 0x7f893c014d80 con 0x7f894c0a07a0 2024-09-16T05:38:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:23.574+0000 7f8948ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:23.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:23.574+0000 7f8948ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:23.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:23.686+0000 7f895276c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:23.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:23.686+0000 7f895276c640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:23.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:23.686+0000 7f895276c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:23.691 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:23.686+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7f892c001150 con 0x7f894c0a07a0 2024-09-16T05:38:24.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.582+0000 7f8948ff9640 1 -- 172.21.15.63:0/3414425402 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v131) ==== 129+0+0 (unknown 1867262486 0 0) 0x7f893c0190d0 con 0x7f894c0a07a0 2024-09-16T05:38:24.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.582+0000 7f8948ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:24.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.582+0000 7f8948ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.586+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8938036a20 legacy=0x7f8938038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.586+0000 7f895276c640 10 monclient: shutdown 2024-09-16T05:38:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.586+0000 7f895276c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.586+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 >> v1:172.21.15.6:6790/0 conn(0x7f894c0a07a0 legacy=0x7f894c136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.586+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 shutdown_connections 2024-09-16T05:38:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.586+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 >> 172.21.15.63:0/3414425402 conn(0x7f894c075470 msgr2=0x7f894c0a7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.586+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 shutdown_connections 2024-09-16T05:38:24.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.586+0000 7f895276c640 1 -- 172.21.15.63:0/3414425402 wait complete. 2024-09-16T05:38:24.609 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-16T05:38:24.609 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:38:24.609 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-16T05:38:24.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.734+0000 7faa1ef82640 10 monclient: get_monmap_and_config 2024-09-16T05:38:24.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.734+0000 7faa1ef82640 10 monclient: build_initial_monmap 2024-09-16T05:38:24.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.734+0000 7faa1ef82640 10 monclient: monmap: 2024-09-16T05:38:24.740 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:24.740 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:24.741 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:24.740006+0000 2024-09-16T05:38:24.741 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:24.740006+0000 2024-09-16T05:38:24.741 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:24.741 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:24.741 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:24.741 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:24.741 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:24.741 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:24.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 Processor -- start 2024-09-16T05:38:24.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- start start 2024-09-16T05:38:24.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient: init 2024-09-16T05:38:24.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:24.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient(hunting): picked mon.noname-a con 0x7faa180a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient(hunting): picked mon.noname-c con 0x7faa180a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient(hunting): picked mon.noname-b con 0x7faa180a1f10 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa180a0f40 con 0x7faa180a3b10 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa180a1220 con 0x7faa180a4600 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa180a1500 con 0x7faa180a1f10 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient(hunting): authenticate will time out at 1161.517270s 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa17fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7faa180a4600 legacy=0x7faa1813bc80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1d51f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7faa180a1f10 0x7faa1813f3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59872/0 (socket says 172.21.15.63:59872) 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1d51f640 1 -- 172.21.15.63:0/534720122 learned_addr learned my addr 172.21.15.63:0/534720122 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:24.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765593476 0 0) 0x7faa180a1500 con 0x7faa180a1f10 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient(hunting): my global_id is 5558 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa00003530 con 0x7faa180a1f10 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4266095505 0 0) 0x7faa180a0f40 con 0x7faa180a3b10 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient(hunting): my global_id is 5574 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa180a1500 con 0x7faa180a3b10 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 389983357 0 0) 0x7faa00003530 con 0x7faa180a1f10 2024-09-16T05:38:24.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 >> v1:172.21.15.63:6789/0 conn(0x7faa180a4600 legacy=0x7faa1813bc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 >> v1:172.21.15.6:6789/0 conn(0x7faa180a3b10 legacy=0x7faa180a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 monclient: found mon.noname-b 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa180a51c0 con 0x7faa180a1f10 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa10001620 con 0x7faa180a1f10 2024-09-16T05:38:24.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: dump: 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 5 monclient: authenticate success, global_id 5558 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa100018d0 con 0x7faa180a1f10 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 1 -- 172.21.15.63:0/534720122 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa10001ce0 con 0x7faa180a1f10 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa177fe640 10 monclient: dump: 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:24.748 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:24.749 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:24.749 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:24.749 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:24.749 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:24.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:24.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 4 set_mon_vals no callback set 2024-09-16T05:38:24.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 10 monclient: shutdown 2024-09-16T05:38:24.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:24.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- 172.21.15.63:0/534720122 >> v1:172.21.15.6:6790/0 conn(0x7faa180a1f10 legacy=0x7faa1813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- 172.21.15.63:0/534720122 shutdown_connections 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- 172.21.15.63:0/534720122 >> 172.21.15.63:0/534720122 conn(0x7faa18075470 msgr2=0x7faa18075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- 172.21.15.63:0/534720122 shutdown_connections 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.738+0000 7faa1ef82640 1 -- 172.21.15.63:0/534720122 wait complete. 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: build_initial_monmap 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: monmap: 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:24.754 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 1 Processor -- start 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 1 -- start start 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: init 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient(hunting): picked mon.noname-a con 0x7faa180a1f10 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient(hunting): picked mon.noname-c con 0x7faa180a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient(hunting): picked mon.noname-b con 0x7faa180a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:24.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa180a2f70 con 0x7faa180a1f10 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa180a0500 con 0x7faa180a4600 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa18138a30 con 0x7faa180a3b10 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient(hunting): authenticate will time out at 1161.521106s 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1d51f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7faa180a4600 legacy=0x7faa181d3950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa17fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7faa180a3b10 0x7faa181d0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59884/0 (socket says 172.21.15.63:59884) 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1cd1e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa180a1f10 0x7faa1813f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36732/0 (socket says 172.21.15.63:36732) 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa17fff640 1 -- 172.21.15.63:0/1147498323 learned_addr learned my addr 172.21.15.63:0/1147498323 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994409335 0 0) 0x7faa08001560 con 0x7faa180a1f10 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:24.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): my global_id is 5577 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa04003530 con 0x7faa180a1f10 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418707381 0 0) 0x7faa18138a30 con 0x7faa180a3b10 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): my global_id is 5561 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa08001560 con 0x7faa180a3b10 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1634504365 0 0) 0x7faa04003530 con 0x7faa180a1f10 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa18138a30 con 0x7faa180a1f10 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa08001e40 con 0x7faa180a1f10 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:24.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): dump: 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3309299790 0 0) 0x7faa08001560 con 0x7faa180a3b10 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa04003530 con 0x7faa180a3b10 2024-09-16T05:38:24.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa0c004160 con 0x7faa180a3b10 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient(hunting): dump: 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:24.759 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2650763755 0 0) 0x7faa18138a30 con 0x7faa180a1f10 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 >> v1:172.21.15.6:6790/0 conn(0x7faa180a3b10 legacy=0x7faa181d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 >> v1:172.21.15.63:6789/0 conn(0x7faa180a4600 legacy=0x7faa181d3950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient: _finish_hunting 0 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 monclient: found mon.a 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa18138d10 con 0x7faa180a1f10 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient: _finish_auth 0 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient: _check_auth_tickets 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:24.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 5 monclient: authenticate success, global_id 5577 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: _renew_subs 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa18137c60 con 0x7faa180a1f10 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: _renew_subs 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa1ef82640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faa18137640 con 0x7faa180a1f10 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa08003c30 con 0x7faa180a1f10 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faa08004000 con 0x7faa180a1f10 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa16ffd640 4 set_mon_vals no callback set 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:24.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.742+0000 7faa157fa640 10 monclient: dump: 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.746+0000 7faa1ef82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:24.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.746+0000 7faa1ef82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:24.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.746+0000 7faa1ef82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:24.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.746+0000 7faa1ef82640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f8005090 con 0x7faa180a1f10 2024-09-16T05:38:24.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.746+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7faa08004220 con 0x7faa180a1f10 2024-09-16T05:38:24.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.746+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(131..131 src has 1..131) ==== 7393+0+0 (unknown 319718428 0 0) 0x7faa080455f0 con 0x7faa180a1f10 2024-09-16T05:38:24.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.746+0000 7faa17fff640 10 monclient: get_auth_request con 0x7faa040369e0 auth_method 0 2024-09-16T05:38:24.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.758+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7faa180a2f70 con 0x7faa180a1f10 2024-09-16T05:38:24.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.758+0000 7faa157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:24.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.758+0000 7faa157fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:24.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.926+0000 7faa1ef82640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T05:38:24.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.926+0000 7faa1ef82640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T05:38:24.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.926+0000 7faa1ef82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:24.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.926+0000 7faa1ef82640 1 -- 172.21.15.63:0/1147498323 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]} v 0) -- 0x7fa9f8005380 con 0x7faa180a1f10 2024-09-16T05:38:24.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.926+0000 7faa157fa640 1 -- 172.21.15.63:0/1147498323 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}]=-22 must specify exact 1 weight values v131) ==== 168+0+0 (unknown 1105226602 0 0) 0x7faa080159e0 con 0x7faa180a1f10 2024-09-16T05:38:24.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.926+0000 7faa157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T05:38:24.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:24.926+0000 7faa157fa640 10 monclient: _finish_command 2 = mon:22 must specify exact 1 weight values 2024-09-16T05:38:24.931 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: must specify exact 1 weight values 2024-09-16T05:38:24.935 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:38:24.935 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect_false ceph osd crush weight-set reweight barset osd.0 .9 2024-09-16T05:38:24.935 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -x 2024-09-16T05:38:24.935 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 2024-09-16T05:38:25.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient: get_monmap_and_config 2024-09-16T05:38:25.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient: build_initial_monmap 2024-09-16T05:38:25.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient: monmap: 2024-09-16T05:38:25.058 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:25.058114+0000 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:25.058114+0000 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:25.059 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 1 Processor -- start 2024-09-16T05:38:25.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 1 -- start start 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient: init 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient(hunting): picked mon.noname-c con 0x7fa28809c300 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient(hunting): picked mon.noname-a con 0x7fa2880a09c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient(hunting): picked mon.noname-b con 0x7fa28809d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2880a0ed0 con 0x7fa2880a09c0 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa28809d950 con 0x7fa28809c300 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa28809ebd0 con 0x7fa28809d3a0 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa28d3f5640 10 monclient(hunting): authenticate will time out at 1161.835928s 2024-09-16T05:38:25.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.054+0000 7fa286ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa28809c300 legacy=0x7fa28809c6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:25.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa2867fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa2880a09c0 0x7fa288138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36746/0 (socket says 172.21.15.63:36746) 2024-09-16T05:38:25.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa2867fc640 1 -- 172.21.15.63:0/601278305 learned_addr learned my addr 172.21.15.63:0/601278305 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:25.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2543283932 0 0) 0x7fa2880a0ed0 con 0x7fa2880a09c0 2024-09-16T05:38:25.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:25.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:25.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:25.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient(hunting): my global_id is 5580 2024-09-16T05:38:25.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa270005740 con 0x7fa2880a09c0 2024-09-16T05:38:25.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553682297 0 0) 0x7fa28809ebd0 con 0x7fa28809d3a0 2024-09-16T05:38:25.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient(hunting): my global_id is 5564 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2880a0ed0 con 0x7fa28809d3a0 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1473747256 0 0) 0x7fa270005740 con 0x7fa2880a09c0 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 >> v1:172.21.15.6:6790/0 conn(0x7fa28809d3a0 legacy=0x7fa28813c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 >> v1:172.21.15.63:6789/0 conn(0x7fa28809c300 legacy=0x7fa28809c6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 monclient: found mon.noname-a 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2880a2470 con 0x7fa2880a09c0 2024-09-16T05:38:25.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 5 monclient: authenticate success, global_id 5580 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa27c001550 con 0x7fa2880a09c0 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: dump: 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.070 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa27c0017b0 con 0x7fa2880a09c0 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:25.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 1 -- 172.21.15.63:0/601278305 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa27c001bc0 con 0x7fa2880a09c0 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 4 set_mon_vals no callback set 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa285ffb640 10 monclient: dump: 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.072 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient: shutdown 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- 172.21.15.63:0/601278305 >> v1:172.21.15.6:6789/0 conn(0x7fa2880a09c0 legacy=0x7fa288138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- 172.21.15.63:0/601278305 shutdown_connections 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- 172.21.15.63:0/601278305 >> 172.21.15.63:0/601278305 conn(0x7fa2880a1e90 msgr2=0x7fa288075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- 172.21.15.63:0/601278305 shutdown_connections 2024-09-16T05:38:25.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- 172.21.15.63:0/601278305 wait complete. 2024-09-16T05:38:25.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient: build_initial_monmap 2024-09-16T05:38:25.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient: monmap: 2024-09-16T05:38:25.074 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:25.074 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:25.074 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:25.074 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 Processor -- start 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- start start 2024-09-16T05:38:25.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient: init 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient(hunting): picked mon.noname-c con 0x7fa28809c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient(hunting): picked mon.noname-a con 0x7fa28809d3a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient(hunting): picked mon.noname-b con 0x7fa2881cd210 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa288122410 con 0x7fa28809d3a0 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2881226f0 con 0x7fa2881cd210 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa28811f970 con 0x7fa28809c300 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa28d3f5640 10 monclient(hunting): authenticate will time out at 1161.839727s 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.058+0000 7fa2877fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa2881cd210 legacy=0x7fa2881cf600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:25.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa286ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa28809c300 0x7fa28811eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59900/0 (socket says 172.21.15.63:59900) 2024-09-16T05:38:25.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa2867fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa28809d3a0 0x7fa28811f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36748/0 (socket says 172.21.15.63:36748) 2024-09-16T05:38:25.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa286ffd640 1 -- 172.21.15.63:0/1193011512 learned_addr learned my addr 172.21.15.63:0/1193011512 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:25.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 218837772 0 0) 0x7fa288122410 con 0x7fa28809d3a0 2024-09-16T05:38:25.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:25.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): my global_id is 5583 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa274003530 con 0x7fa28809d3a0 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3654457049 0 0) 0x7fa28811f970 con 0x7fa28809c300 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): my global_id is 5567 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa288122410 con 0x7fa28809c300 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3262880673 0 0) 0x7fa274003530 con 0x7fa28809d3a0 2024-09-16T05:38:25.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa28811f970 con 0x7fa28809d3a0 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa27c001b00 con 0x7fa28809d3a0 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.079 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4203037529 0 0) 0x7fa288122410 con 0x7fa28809c300 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa274003530 con 0x7fa28809c300 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa2780040e0 con 0x7fa28809c300 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:25.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 790221988 0 0) 0x7fa28811f970 con 0x7fa28809d3a0 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 >> v1:172.21.15.6:6790/0 conn(0x7fa28809c300 legacy=0x7fa28811eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 >> v1:172.21.15.63:6789/0 conn(0x7fa2881cd210 legacy=0x7fa2881cf600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 monclient: found mon.a 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2880a4010 con 0x7fa28809d3a0 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:25.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 5 monclient: authenticate success, global_id 5583 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 10 monclient: _renew_subs 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa28809c0f0 con 0x7fa28809d3a0 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 10 monclient: _renew_subs 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa2881229d0 con 0x7fa28809d3a0 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa27c0017b0 con 0x7fa28809d3a0 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa27c004c40 con 0x7fa28809d3a0 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa284ff9640 4 set_mon_vals no callback set 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:25.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 10 monclient: dump: 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.084 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:25.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:25.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa28d3f5640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa268005090 con 0x7fa28809d3a0 2024-09-16T05:38:25.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa27c004e60 con 0x7fa28809d3a0 2024-09-16T05:38:25.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(131..131 src has 1..131) ==== 7393+0+0 (unknown 319718428 0 0) 0x7fa27c044f70 con 0x7fa28809d3a0 2024-09-16T05:38:25.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.062+0000 7fa286ffd640 10 monclient: get_auth_request con 0x7fa274036990 auth_method 0 2024-09-16T05:38:25.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.074+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa27c015480 con 0x7fa28809d3a0 2024-09-16T05:38:25.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.074+0000 7fa26f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:25.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.074+0000 7fa26f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:25.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.246+0000 7fa28d3f5640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:25.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.246+0000 7fa28d3f5640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:25.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.246+0000 7fa28d3f5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.246+0000 7fa28d3f5640 1 -- 172.21.15.63:0/1193011512 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7fa268005380 con 0x7fa28809d3a0 2024-09-16T05:38:25.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.246+0000 7fa26f7fe640 1 -- 172.21.15.63:0/1193011512 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}]=-22 must specify exact 3 weight values v131) ==== 163+0+0 (unknown 3341478217 0 0) 0x7fa27c014d80 con 0x7fa28809d3a0 2024-09-16T05:38:25.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.246+0000 7fa26f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T05:38:25.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.246+0000 7fa26f7fe640 10 monclient: _finish_command 2 = mon:22 must specify exact 3 weight values 2024-09-16T05:38:25.252 INFO:tasks.workunit.client.0.smithi063.stderr:Error EINVAL: must specify exact 3 weight values 2024-09-16T05:38:25.258 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:38:25.258 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 .9 .9 2024-09-16T05:38:25.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient: get_monmap_and_config 2024-09-16T05:38:25.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient: build_initial_monmap 2024-09-16T05:38:25.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient: monmap: 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:25.381237+0000 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:25.381237+0000 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:25.382 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 1 Processor -- start 2024-09-16T05:38:25.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 1 -- start start 2024-09-16T05:38:25.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient: init 2024-09-16T05:38:25.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient(hunting): picked mon.noname-c con 0x7f086809ba70 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient(hunting): picked mon.noname-b con 0x7f08680a08c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient(hunting): picked mon.noname-a con 0x7f08680a0400 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f086809f8a0 con 0x7f08680a0400 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0868072f20 con 0x7f086809ba70 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f086809eb50 con 0x7f08680a08c0 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient(hunting): authenticate will time out at 1162.158193s 2024-09-16T05:38:25.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f086809ba70 legacy=0x7f086809fcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f0865d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f08680a08c0 0x7f0868139900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59926/0 (socket says 172.21.15.63:59926) 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f0865d9b640 1 -- 172.21.15.63:0/453216692 learned_addr learned my addr 172.21.15.63:0/453216692 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431765198 0 0) 0x7f086809eb50 con 0x7f08680a08c0 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient(hunting): my global_id is 5570 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0850003530 con 0x7f08680a08c0 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3671860277 0 0) 0x7f086809f8a0 con 0x7f08680a0400 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:25.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient(hunting): my global_id is 5586 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f086809eb50 con 0x7f08680a0400 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2889244710 0 0) 0x7f086809eb50 con 0x7f08680a0400 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 >> v1:172.21.15.6:6790/0 conn(0x7f08680a08c0 legacy=0x7f0868139900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 >> v1:172.21.15.63:6789/0 conn(0x7f086809ba70 legacy=0x7f086809fcf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 monclient: found mon.noname-a 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086809e6c0 con 0x7f08680a0400 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: _finish_auth 0 2024-09-16T05:38:25.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0860001900 con 0x7f08680a0400 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: dump: 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.389 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 5 monclient: authenticate success, global_id 5586 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0860001b60 con 0x7f08680a0400 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 1 -- 172.21.15.63:0/453216692 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0860001fb0 con 0x7f08680a0400 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086559a640 10 monclient: dump: 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.390 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.378+0000 7f086cd7d640 4 set_mon_vals no callback set 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient: shutdown 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- 172.21.15.63:0/453216692 >> v1:172.21.15.6:6789/0 conn(0x7f08680a0400 legacy=0x7f086813d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- 172.21.15.63:0/453216692 shutdown_connections 2024-09-16T05:38:25.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- 172.21.15.63:0/453216692 >> 172.21.15.63:0/453216692 conn(0x7f0868075470 msgr2=0x7f0868075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- 172.21.15.63:0/453216692 shutdown_connections 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- 172.21.15.63:0/453216692 wait complete. 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient: build_initial_monmap 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient: monmap: 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:25.392 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:25.393 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 Processor -- start 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- start start 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient: init 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient(hunting): picked mon.noname-a con 0x7f086809ba70 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient(hunting): picked mon.noname-c con 0x7f08680a0400 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient(hunting): picked mon.noname-b con 0x7f08680a08c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08680a2cb0 con 0x7f086809ba70 2024-09-16T05:38:25.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0868121160 con 0x7f08680a08c0 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08680a3250 con 0x7f08680a0400 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086cd7d640 10 monclient(hunting): authenticate will time out at 1162.162150s 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0866d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f08680a08c0 legacy=0x7f08681cf5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f086809ba70 0x7f0868133a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36778/0 (socket says 172.21.15.63:36778) 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f086659c640 1 -- 172.21.15.63:0/105645907 learned_addr learned my addr 172.21.15.63:0/105645907 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744234771 0 0) 0x7f0858003aa0 con 0x7f086809ba70 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): my global_id is 5589 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0848003530 con 0x7f086809ba70 2024-09-16T05:38:25.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1009872295 0 0) 0x7f08680a3250 con 0x7f08680a0400 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): my global_id is 5573 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0858003aa0 con 0x7f08680a0400 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 855826189 0 0) 0x7f0848003530 con 0x7f086809ba70 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08680a3250 con 0x7f086809ba70 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0858004330 con 0x7f086809ba70 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:25.399 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 478060060 0 0) 0x7f0858003aa0 con 0x7f08680a0400 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0848003530 con 0x7f08680a0400 2024-09-16T05:38:25.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f085c001220 con 0x7f08680a0400 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.401 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.382+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3131726965 0 0) 0x7f08680a3250 con 0x7f086809ba70 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 >> v1:172.21.15.6:6790/0 conn(0x7f08680a0400 legacy=0x7f08681cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 >> v1:172.21.15.63:6789/0 conn(0x7f08680a08c0 legacy=0x7f08681cf5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 1 monclient: found mon.a 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086809c0f0 con 0x7f086809ba70 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 5 monclient: authenticate success, global_id 5589 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 10 monclient: _renew_subs 2024-09-16T05:38:25.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0868120e80 con 0x7f086809ba70 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 10 monclient: _renew_subs 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0868134190 con 0x7f086809ba70 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f08580044d0 con 0x7f086809ba70 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f08580048a0 con 0x7f086809ba70 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0864d99640 4 set_mon_vals no callback set 2024-09-16T05:38:25.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 10 monclient: dump: 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:25.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084c005090 con 0x7f086809ba70 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0858004ac0 con 0x7f086809ba70 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(131..131 src has 1..131) ==== 7393+0+0 (unknown 319718428 0 0) 0x7f0858045190 con 0x7f086809ba70 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.386+0000 7f0865d9b640 10 monclient: get_auth_request con 0x7f0848036a30 auth_method 0 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.398+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f08680a2cb0 con 0x7f086809ba70 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.398+0000 7f0856ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:25.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.398+0000 7f0856ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:25.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.566+0000 7f086cd7d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T05:38:25.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.566+0000 7f086cd7d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T05:38:25.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.566+0000 7f086cd7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.566+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) -- 0x7f084c005380 con 0x7f086809ba70 2024-09-16T05:38:25.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.578+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v132) ==== 139+0+0 (unknown 26582975 0 0) 0x7f0858015520 con 0x7f086809ba70 2024-09-16T05:38:25.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.578+0000 7f0856ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T05:38:25.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.578+0000 7f0856ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:25.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.682+0000 7f086cd7d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T05:38:25.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.682+0000 7f086cd7d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T05:38:25.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.682+0000 7f086cd7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:25.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:25.682+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) -- 0x7f084c005ac0 con 0x7f086809ba70 2024-09-16T05:38:26.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.006+0000 7f0856ffd640 1 -- 172.21.15.63:0/105645907 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v133) ==== 139+0+0 (unknown 3857981491 0 0) 0x7f0858014e20 con 0x7f086809ba70 2024-09-16T05:38:26.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.006+0000 7f0856ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T05:38:26.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.006+0000 7f0856ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:26.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.010+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0848036a30 legacy=0x7f0848038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.010+0000 7f086cd7d640 10 monclient: shutdown 2024-09-16T05:38:26.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.010+0000 7f086cd7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:26.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.010+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 >> v1:172.21.15.6:6789/0 conn(0x7f086809ba70 legacy=0x7f0868133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.010+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 shutdown_connections 2024-09-16T05:38:26.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.010+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 >> 172.21.15.63:0/105645907 conn(0x7f0868075470 msgr2=0x7f0868134790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:26.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.010+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 shutdown_connections 2024-09-16T05:38:26.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.010+0000 7f086cd7d640 1 -- 172.21.15.63:0/105645907 wait complete. 2024-09-16T05:38:26.034 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set ls 2024-09-16T05:38:26.034 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c fooset 2024-09-16T05:38:26.034 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 1 2024-09-16T05:38:26.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.150+0000 7f5a225ce640 10 monclient: get_monmap_and_config 2024-09-16T05:38:26.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.150+0000 7f5a225ce640 10 monclient: build_initial_monmap 2024-09-16T05:38:26.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.150+0000 7f5a225ce640 10 monclient: monmap: 2024-09-16T05:38:26.156 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:26.155893+0000 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:26.155893+0000 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:26.157 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 1 Processor -- start 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 1 -- start start 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient: init 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient(hunting): picked mon.noname-b con 0x7f5a1c0a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient(hunting): picked mon.noname-c con 0x7f5a1c0a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient(hunting): picked mon.noname-a con 0x7f5a1c0a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1c0a4fd0 con 0x7f5a1c0a4790 2024-09-16T05:38:26.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1c0a5b40 con 0x7f5a1c0a08a0 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1c0a1a70 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5a1c0a08a0 legacy=0x7f5a1c13beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient(hunting): authenticate will time out at 1162.933446s 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5a1c0a3ca0 0x7f5a1c0a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59940/0 (socket says 172.21.15.63:59940) 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a20b6b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5a1c0a4790 0x7f5a1c13f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36788/0 (socket says 172.21.15.63:36788) 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1bfff640 1 -- 172.21.15.63:0/1412333500 learned_addr learned my addr 172.21.15.63:0/1412333500 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354381009 0 0) 0x7f5a1c0a1a70 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:26.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient(hunting): my global_id is 5576 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a04003530 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2575785152 0 0) 0x7f5a1c0a4fd0 con 0x7f5a1c0a4790 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient(hunting): my global_id is 5592 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a1c0a1a70 con 0x7f5a1c0a4790 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3163604519 0 0) 0x7f5a1c0a1a70 con 0x7f5a1c0a4790 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 >> v1:172.21.15.6:6790/0 conn(0x7f5a1c0a3ca0 legacy=0x7f5a1c0a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 >> v1:172.21.15.63:6789/0 conn(0x7f5a1c0a08a0 legacy=0x7f5a1c13beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 monclient: found mon.noname-a 2024-09-16T05:38:26.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c0a53e0 con 0x7f5a1c0a4790 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: _finish_auth 0 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a14001980 con 0x7f5a1c0a4790 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: dump: 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.163 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 5 monclient: authenticate success, global_id 5592 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a14001c70 con 0x7f5a1c0a4790 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 1 -- 172.21.15.63:0/1412333500 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a14002080 con 0x7f5a1c0a4790 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:26.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a1affd640 10 monclient: dump: 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:26.165 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 4 set_mon_vals no callback set 2024-09-16T05:38:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 10 monclient: shutdown 2024-09-16T05:38:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 1 -- 172.21.15.63:0/1412333500 >> v1:172.21.15.6:6789/0 conn(0x7f5a1c0a4790 legacy=0x7f5a1c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.154+0000 7f5a225ce640 1 -- 172.21.15.63:0/1412333500 shutdown_connections 2024-09-16T05:38:26.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 -- 172.21.15.63:0/1412333500 >> 172.21.15.63:0/1412333500 conn(0x7f5a1c0a67c0 msgr2=0x7f5a1c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 -- 172.21.15.63:0/1412333500 shutdown_connections 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 -- 172.21.15.63:0/1412333500 wait complete. 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient: build_initial_monmap 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient: monmap: 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:26.171 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 Processor -- start 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 -- start start 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient: init 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient(hunting): picked mon.noname-b con 0x7f5a1c0a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient(hunting): picked mon.noname-c con 0x7f5a1c0a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient(hunting): picked mon.noname-a con 0x7f5a1c0a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1c125350 con 0x7f5a1c0a4790 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1c125630 con 0x7f5a1c0a08a0 2024-09-16T05:38:26.172 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a1bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5a1c0a08a0 legacy=0x7f5a1c136c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1c0a7f70 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient(hunting): authenticate will time out at 1162.938047s 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a1b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5a1c0a3ca0 0x7f5a1c1d0110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59942/0 (socket says 172.21.15.63:59942) 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a1b7fe640 1 -- 172.21.15.63:0/1151722014 learned_addr learned my addr 172.21.15.63:0/1151722014 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509672737 0 0) 0x7f5a1c0a7f70 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): my global_id is 5579 2024-09-16T05:38:26.173 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a08003530 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 928565595 0 0) 0x7f5a1c125350 con 0x7f5a1c0a4790 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): my global_id is 5595 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a1c0a7f70 con 0x7f5a1c0a4790 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 370765304 0 0) 0x7f5a1c0a7f70 con 0x7f5a1c0a4790 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a1c125350 con 0x7f5a1c0a4790 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1482853239 0 0) 0x7f5a08003530 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a1c0a7f70 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a14001940 con 0x7f5a1c0a4790 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:26.174 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a100040e0 con 0x7f5a1c0a3ca0 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient(hunting): dump: 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.176 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4201095820 0 0) 0x7f5a1c125350 con 0x7f5a1c0a4790 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 >> v1:172.21.15.6:6790/0 conn(0x7f5a1c0a3ca0 legacy=0x7f5a1c1d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 >> v1:172.21.15.63:6789/0 conn(0x7f5a1c0a08a0 legacy=0x7f5a1c136c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 monclient: found mon.a 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c0a0500 con 0x7f5a1c0a4790 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient: _finish_auth 0 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:26.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 5 monclient: authenticate success, global_id 5595 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient: _renew_subs 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.158+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1c125070 con 0x7f5a1c0a4790 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a225ce640 10 monclient: _renew_subs 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a225ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a1c1cbc50 con 0x7f5a1c0a4790 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a14004f00 con 0x7f5a1c0a4790 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a18ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a1a7fc640 4 set_mon_vals no callback set 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a140052d0 con 0x7f5a1c0a4790 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:26.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a18ff9640 10 monclient: dump: 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.179 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.180 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a225ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:26.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a225ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:26.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a225ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:26.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59fc005090 con 0x7f5a1c0a4790 2024-09-16T05:38:26.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5a140054f0 con 0x7f5a1c0a4790 2024-09-16T05:38:26.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(133..133 src has 1..133) ==== 7393+0+0 (unknown 2830276434 0 0) 0x7f5a14045590 con 0x7f5a1c0a4790 2024-09-16T05:38:26.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.162+0000 7f5a1bfff640 10 monclient: get_auth_request con 0x7f5a08036a50 auth_method 0 2024-09-16T05:38:26.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.174+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5a14015920 con 0x7f5a1c0a4790 2024-09-16T05:38:26.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.174+0000 7f5a18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:26.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.174+0000 7f5a18ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:26.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.350+0000 7f5a225ce640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:26.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.350+0000 7f5a225ce640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:26.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.350+0000 7f5a225ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:26.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.350+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f59fc005380 con 0x7f5a1c0a4790 2024-09-16T05:38:26.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.354+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) ==== 71+0+14 (unknown 1899167245 0 3003798303) 0x7f5a14019570 con 0x7f5a1c0a4790 2024-09-16T05:38:26.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.354+0000 7f5a18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:26.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.354+0000 7f5a18ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f59fc005ac0 con 0x7f5a1c0a4790 2024-09-16T05:38:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a18ff9640 1 -- 172.21.15.63:0/1151722014 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) ==== 71+0+14 (unknown 1899167245 0 3003798303) 0x7f5a1401de30 con 0x7f5a1c0a4790 2024-09-16T05:38:26.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a18ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:26.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a18ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5a08036a50 legacy=0x7f5a08038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 10 monclient: shutdown 2024-09-16T05:38:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 >> v1:172.21.15.6:6789/0 conn(0x7f5a1c0a4790 legacy=0x7f5a1c1d3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 shutdown_connections 2024-09-16T05:38:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 >> 172.21.15.63:0/1151722014 conn(0x7f5a1c0a67c0 msgr2=0x7f5a1c0a8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 shutdown_connections 2024-09-16T05:38:26.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.458+0000 7f5a225ce640 1 -- 172.21.15.63:0/1151722014 wait complete. 2024-09-16T05:38:26.479 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set rm fooset 2024-09-16T05:38:26.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient: get_monmap_and_config 2024-09-16T05:38:26.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient: build_initial_monmap 2024-09-16T05:38:26.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient: monmap: 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:26.601302+0000 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:26.601302+0000 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:26.602 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 1 Processor -- start 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 1 -- start start 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient: init 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient(hunting): picked mon.noname-c con 0x7fa858136960 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient(hunting): picked mon.noname-b con 0x7fa8580a1c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient(hunting): picked mon.noname-a con 0x7fa858137450 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:26.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8580a09c0 con 0x7fa858137450 2024-09-16T05:38:26.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8580a0f80 con 0x7fa858136960 2024-09-16T05:38:26.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8580a4bb0 con 0x7fa8580a1c40 2024-09-16T05:38:26.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85dbba640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa858136960 legacy=0x7fa858136d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:26.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:26.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85fe1e640 10 monclient(hunting): authenticate will time out at 1163.378380s 2024-09-16T05:38:26.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85e3bb640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa858137450 0x7fa85813f4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36808/0 (socket says 172.21.15.63:36808) 2024-09-16T05:38:26.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85d3b9640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa8580a1c40 0x7fa85813bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59956/0 (socket says 172.21.15.63:59956) 2024-09-16T05:38:26.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.598+0000 7fa85e3bb640 1 -- 172.21.15.63:0/1793051590 learned_addr learned my addr 172.21.15.63:0/1793051590 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:26.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713610024 0 0) 0x7fa8580a4bb0 con 0x7fa8580a1c40 2024-09-16T05:38:26.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient(hunting): _negotiate 2024-09-16T05:38:26.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:26.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:26.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient(hunting): my global_id is 5582 2024-09-16T05:38:26.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa844003530 con 0x7fa8580a1c40 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525783823 0 0) 0x7fa8580a09c0 con 0x7fa858137450 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient(hunting): _negotiate 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient(hunting): my global_id is 5598 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8580a4bb0 con 0x7fa858137450 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1142727953 0 0) 0x7fa844003530 con 0x7fa8580a1c40 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 >> v1:172.21.15.63:6789/0 conn(0x7fa858136960 legacy=0x7fa858136d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 >> v1:172.21.15.6:6789/0 conn(0x7fa858137450 legacy=0x7fa85813f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: _finish_hunting 0 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 monclient: found mon.noname-b 2024-09-16T05:38:26.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8580a4e60 con 0x7fa8580a1c40 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: _finish_auth 0 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: _check_auth_tickets 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa848001620 con 0x7fa8580a1c40 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:26.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: dump: 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 5 monclient: authenticate success, global_id 5582 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:26.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa848001880 con 0x7fa8580a1c40 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 1 -- 172.21.15.63:0/1793051590 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa848001c10 con 0x7fa8580a1c40 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85cbb8640 10 monclient: dump: 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.614 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.615 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.615 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.615 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:26.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 4 set_mon_vals no callback set 2024-09-16T05:38:26.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 10 monclient: shutdown 2024-09-16T05:38:26.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:26.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 1 -- 172.21.15.63:0/1793051590 >> v1:172.21.15.6:6790/0 conn(0x7fa8580a1c40 legacy=0x7fa85813bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 1 -- 172.21.15.63:0/1793051590 shutdown_connections 2024-09-16T05:38:26.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 1 -- 172.21.15.63:0/1793051590 >> 172.21.15.63:0/1793051590 conn(0x7fa8580a60a0 msgr2=0x7fa8580a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:26.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 1 -- 172.21.15.63:0/1793051590 shutdown_connections 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 1 -- 172.21.15.63:0/1793051590 wait complete. 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 10 monclient: build_initial_monmap 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 10 monclient: monmap: 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:26.620 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 1 Processor -- start 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 1 -- start start 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.606+0000 7fa85fe1e640 10 monclient: init 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient(hunting): picked mon.noname-a con 0x7fa8580a1c40 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient(hunting): picked mon.noname-b con 0x7fa858136960 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient(hunting): picked mon.noname-c con 0x7fa858137450 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8580a0500 con 0x7fa8580a1c40 2024-09-16T05:38:26.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa858124180 con 0x7fa858136960 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85d3b9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa858136960 legacy=0x7fa8581d0190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8580a72c0 con 0x7fa858137450 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient(hunting): authenticate will time out at 1163.388439s 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85dbba640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8580a1c40 0x7fa858135d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36818/0 (socket says 172.21.15.63:36818) 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85e3bb640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa858137450 0x7fa8581d38c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59972/0 (socket says 172.21.15.63:59972) 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85dbba640 1 -- 172.21.15.63:0/86300465 learned_addr learned my addr 172.21.15.63:0/86300465 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262649743 0 0) 0x7fa8580a72c0 con 0x7fa858137450 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:26.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): my global_id is 5585 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa83c003530 con 0x7fa858137450 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 226396931 0 0) 0x7fa854003aa0 con 0x7fa8580a1c40 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): my global_id is 5601 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8580a72c0 con 0x7fa8580a1c40 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3913874204 0 0) 0x7fa83c003530 con 0x7fa858137450 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa854003aa0 con 0x7fa858137450 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa850001ae0 con 0x7fa858137450 2024-09-16T05:38:26.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2585118117 0 0) 0x7fa8580a72c0 con 0x7fa8580a1c40 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa83c003530 con 0x7fa8580a1c40 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa8540042b0 con 0x7fa8580a1c40 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.625 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 450614272 0 0) 0x7fa854003aa0 con 0x7fa858137450 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 >> v1:172.21.15.63:6789/0 conn(0x7fa858136960 legacy=0x7fa8581d0190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 >> v1:172.21.15.6:6789/0 conn(0x7fa8580a1c40 legacy=0x7fa858135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 monclient: found mon.c 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa858124460 con 0x7fa858137450 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:26.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 5 monclient: authenticate success, global_id 5585 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient: _renew_subs 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa858122d30 con 0x7fa858137450 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient: _renew_subs 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa858136650 con 0x7fa858137450 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa850002090 con 0x7fa858137450 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa850004d10 con 0x7fa858137450 2024-09-16T05:38:26.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 10 monclient: dump: 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:26.628 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:26.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84ffff640 4 set_mon_vals no callback set 2024-09-16T05:38:26.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:26.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:26.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:26.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa840005090 con 0x7fa858137450 2024-09-16T05:38:26.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa85000fab0 con 0x7fa858137450 2024-09-16T05:38:26.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.610+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(133..133 src has 1..133) ==== 7393+0+0 (unknown 2830276434 0 0) 0x7fa8500451b0 con 0x7fa858137450 2024-09-16T05:38:26.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.614+0000 7fa85dbba640 10 monclient: get_auth_request con 0x7fa83c036ac0 auth_method 0 2024-09-16T05:38:26.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.626+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa8500155e0 con 0x7fa858137450 2024-09-16T05:38:26.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.626+0000 7fa84e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:26.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.626+0000 7fa84e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:26.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.798+0000 7fa85fe1e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T05:38:26.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.798+0000 7fa85fe1e640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T05:38:26.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.798+0000 7fa85fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:26.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:26.798+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) -- 0x7fa840005380 con 0x7fa858137450 2024-09-16T05:38:27.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:27.618+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v134) ==== 89+0+0 (unknown 3371209887 0 0) 0x7fa850019290 con 0x7fa858137450 2024-09-16T05:38:27.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:27.618+0000 7fa84e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T05:38:27.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:27.618+0000 7fa84e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:27.730+0000 7fa85fe1e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T05:38:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:27.730+0000 7fa85fe1e640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T05:38:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:27.730+0000 7fa85fe1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:27.730+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) -- 0x7fa840005770 con 0x7fa858137450 2024-09-16T05:38:28.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.622+0000 7fa84e7fc640 1 -- 172.21.15.63:0/86300465 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v135) ==== 89+0+0 (unknown 2323818614 0 0) 0x7fa85001db50 con 0x7fa858137450 2024-09-16T05:38:28.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.622+0000 7fa84e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T05:38:28.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.622+0000 7fa84e7fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:28.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.626+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa83c036ac0 legacy=0x7fa83c038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:28.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.626+0000 7fa85fe1e640 10 monclient: shutdown 2024-09-16T05:38:28.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.626+0000 7fa85fe1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:28.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.626+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 >> v1:172.21.15.6:6790/0 conn(0x7fa858137450 legacy=0x7fa8581d38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:28.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.626+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 shutdown_connections 2024-09-16T05:38:28.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.626+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 >> 172.21.15.63:0/86300465 conn(0x7fa8580a60a0 msgr2=0x7fa8580a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:28.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.626+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 shutdown_connections 2024-09-16T05:38:28.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.626+0000 7fa85fe1e640 1 -- 172.21.15.63:0/86300465 wait complete. 2024-09-16T05:38:28.656 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set ls 2024-09-16T05:38:28.656 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c fooset 2024-09-16T05:38:28.656 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 0 2024-09-16T05:38:28.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient: get_monmap_and_config 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient: build_initial_monmap 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient: monmap: 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:28.777596+0000 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:28.777596+0000 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:28.778 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 1 Processor -- start 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 1 -- start start 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient: init 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient(hunting): picked mon.noname-c con 0x7f715409ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient(hunting): picked mon.noname-a con 0x7f715406dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient(hunting): picked mon.noname-b con 0x7f71540a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:28.780 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715409f800 con 0x7f715406dd30 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71540a0ab0 con 0x7f715409ac90 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71540a0d90 con 0x7f71540a0210 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715b9e1640 10 monclient(hunting): authenticate will time out at 1165.554543s 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f715977d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f715409ac90 legacy=0x7f715409b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f7158f7c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f715406dd30 0x7f71541399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36832/0 (socket says 172.21.15.63:36832) 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f7159f7e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f71540a0210 0x7f715413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59980/0 (socket says 172.21.15.63:59980) 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f7158f7c640 1 -- 172.21.15.63:0/2341629336 learned_addr learned my addr 172.21.15.63:0/2341629336 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2004497302 0 0) 0x7f71540a0d90 con 0x7f71540a0210 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:28.781 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 10 monclient(hunting): my global_id is 5588 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7140003530 con 0x7f71540a0210 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2709240957 0 0) 0x7f715409f800 con 0x7f715406dd30 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 10 monclient(hunting): my global_id is 5604 2024-09-16T05:38:28.782 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.774+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71540a0d90 con 0x7f715406dd30 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2005294555 0 0) 0x7f7140003530 con 0x7f71540a0210 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 >> v1:172.21.15.63:6789/0 conn(0x7f715409ac90 legacy=0x7f715409b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 >> v1:172.21.15.6:6789/0 conn(0x7f715406dd30 legacy=0x7f71541399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 1 monclient: found mon.noname-b 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715409e770 con 0x7f71540a0210 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: _finish_auth 0 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:28.787 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f714c0019a0 con 0x7f71540a0210 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: dump: 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:28.788 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 5 monclient: authenticate success, global_id 5588 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f714c001c00 con 0x7f71540a0210 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 1 -- 172.21.15.63:0/2341629336 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f714c001fd0 con 0x7f71540a0210 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f714bfff640 10 monclient: dump: 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.789 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 4 set_mon_vals no callback set 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient: shutdown 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:28.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- 172.21.15.63:0/2341629336 >> v1:172.21.15.6:6790/0 conn(0x7f71540a0210 legacy=0x7f715413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- 172.21.15.63:0/2341629336 shutdown_connections 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- 172.21.15.63:0/2341629336 >> 172.21.15.63:0/2341629336 conn(0x7f71540a1c90 msgr2=0x7f71540a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- 172.21.15.63:0/2341629336 shutdown_connections 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- 172.21.15.63:0/2341629336 wait complete. 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient: build_initial_monmap 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient: monmap: 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:28.791 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 Processor -- start 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- start start 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient: init 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient(hunting): picked mon.noname-b con 0x7f715406dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:28.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient(hunting): picked mon.noname-c con 0x7f715409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient(hunting): picked mon.noname-a con 0x7f71540a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715411f0f0 con 0x7f71540a0210 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715409c0f0 con 0x7f715406dd30 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7154121100 con 0x7f715409ac90 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715b9e1640 10 monclient(hunting): authenticate will time out at 1165.559095s 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f715977d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f715406dd30 legacy=0x7f7154133c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7158f7c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f715409ac90 0x7f71541ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59988/0 (socket says 172.21.15.63:59988) 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7158f7c640 1 -- 172.21.15.63:0/668183550 learned_addr learned my addr 172.21.15.63:0/668183550 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631138383 0 0) 0x7f715411f0f0 con 0x7f71540a0210 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:28.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 10 monclient(hunting): my global_id is 5607 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7138003530 con 0x7f71540a0210 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1448027695 0 0) 0x7f7154121100 con 0x7f715409ac90 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 10 monclient(hunting): my global_id is 5591 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.778+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f715411f0f0 con 0x7f715409ac90 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 826159459 0 0) 0x7f7138003530 con 0x7f71540a0210 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7154121100 con 0x7f71540a0210 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f714c001f50 con 0x7f71540a0210 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:28.794 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1596233859 0 0) 0x7f715411f0f0 con 0x7f715409ac90 2024-09-16T05:38:28.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7138003530 con 0x7f715409ac90 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f71440017e0 con 0x7f715409ac90 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:28.796 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1159252412 0 0) 0x7f7154121100 con 0x7f71540a0210 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 >> v1:172.21.15.6:6790/0 conn(0x7f715409ac90 legacy=0x7f71541ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 >> v1:172.21.15.63:6789/0 conn(0x7f715406dd30 legacy=0x7f7154133c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 monclient: found mon.a 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71541213e0 con 0x7f71540a0210 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:28.797 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 5 monclient: authenticate success, global_id 5607 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 10 monclient: _renew_subs 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f715411f9d0 con 0x7f71540a0210 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 10 monclient: _renew_subs 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7154134530 con 0x7f71540a0210 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f714c001c00 con 0x7f71540a0210 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f714c005060 con 0x7f71540a0210 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:28.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 10 monclient: dump: 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:28.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f714b7fe640 4 set_mon_vals no callback set 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7134005090 con 0x7f71540a0210 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f714c00fe00 con 0x7f71540a0210 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(135..135 src has 1..135) ==== 7061+0+0 (unknown 2614354796 0 0) 0x7f714c045120 con 0x7f71540a0210 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.782+0000 7f715977d640 10 monclient: get_auth_request con 0x7f7138036a20 auth_method 0 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.794+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f714c01db10 con 0x7f71540a0210 2024-09-16T05:38:28.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.794+0000 7f7149ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:28.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.794+0000 7f7149ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:28.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.966+0000 7f715b9e1640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:28.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.966+0000 7f715b9e1640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.966+0000 7f715b9e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.966+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f7134005380 con 0x7f71540a0210 2024-09-16T05:38:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.966+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v135) ==== 71+0+7 (unknown 1171993247 0 1352219128) 0x7f714c040690 con 0x7f71540a0210 2024-09-16T05:38:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.966+0000 7f7149ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:28.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:28.966+0000 7f7149ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:29.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.070+0000 7f715b9e1640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.070+0000 7f715b9e1640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.070+0000 7f715b9e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.070+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f7134005ac0 con 0x7f71540a0210 2024-09-16T05:38:29.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f7149ffb640 1 -- 172.21.15.63:0/668183550 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v135) ==== 71+0+7 (unknown 1171993247 0 1352219128) 0x7f714c0408d0 con 0x7f71540a0210 2024-09-16T05:38:29.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f7149ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f7149ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:29.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7138036a20 legacy=0x7f7138038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f715b9e1640 10 monclient: shutdown 2024-09-16T05:38:29.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f715b9e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:29.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 >> v1:172.21.15.6:6789/0 conn(0x7f71540a0210 legacy=0x7f71541d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 shutdown_connections 2024-09-16T05:38:29.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 >> 172.21.15.63:0/668183550 conn(0x7f71540a1c90 msgr2=0x7f71540a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:29.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.074+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 shutdown_connections 2024-09-16T05:38:29.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.078+0000 7f715b9e1640 1 -- 172.21.15.63:0/668183550 wait complete. 2024-09-16T05:38:29.096 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set ls 2024-09-16T05:38:29.096 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep barset 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient: get_monmap_and_config 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient: build_initial_monmap 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient: monmap: 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:29.213703+0000 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:29.213703+0000 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.214 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:29.215 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:29.215 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:29.215 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 1 Processor -- start 2024-09-16T05:38:29.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 1 -- start start 2024-09-16T05:38:29.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient: init 2024-09-16T05:38:29.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:29.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:29.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient(hunting): picked mon.noname-c con 0x7faab80a3c10 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient(hunting): picked mon.noname-b con 0x7faab80a4700 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient(hunting): picked mon.noname-a con 0x7faab80a5710 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab8072e80 con 0x7faab80a5710 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab80a11b0 con 0x7faab80a3c10 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab80a00e0 con 0x7faab80a4700 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faabcc23640 10 monclient(hunting): authenticate will time out at 1165.990617s 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7faab80a3c10 legacy=0x7faab80a3ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab5d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7faab80a4700 0x7faab813be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60004/0 (socket says 172.21.15.63:60004) 2024-09-16T05:38:29.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab6d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faab80a5710 0x7faab813f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36852/0 (socket says 172.21.15.63:36852) 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab5d9b640 1 -- 172.21.15.63:0/2811034094 learned_addr learned my addr 172.21.15.63:0/2811034094 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484225356 0 0) 0x7faab8072e80 con 0x7faab80a5710 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 10 monclient(hunting): my global_id is 5610 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaa0003530 con 0x7faab80a5710 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3167114013 0 0) 0x7faab80a00e0 con 0x7faab80a4700 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:29.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 10 monclient(hunting): my global_id is 5594 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faab8072e80 con 0x7faab80a4700 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 344520917 0 0) 0x7faaa0003530 con 0x7faab80a5710 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 >> v1:172.21.15.6:6790/0 conn(0x7faab80a4700 legacy=0x7faab813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.210+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 >> v1:172.21.15.63:6789/0 conn(0x7faab80a3c10 legacy=0x7faab80a3ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 1 monclient: found mon.noname-a 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab80a1e70 con 0x7faab80a5710 2024-09-16T05:38:29.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: _finish_auth 0 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faab00019a0 con 0x7faab80a5710 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 5 monclient: authenticate success, global_id 5610 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: dump: 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.222 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faab0001cc0 con 0x7faab80a5710 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 1 -- 172.21.15.63:0/2811034094 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faab0002090 con 0x7faab80a5710 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 4 set_mon_vals no callback set 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:29.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab559a640 10 monclient: dump: 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient: shutdown 2024-09-16T05:38:29.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- 172.21.15.63:0/2811034094 >> v1:172.21.15.6:6789/0 conn(0x7faab80a5710 legacy=0x7faab813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- 172.21.15.63:0/2811034094 shutdown_connections 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- 172.21.15.63:0/2811034094 >> 172.21.15.63:0/2811034094 conn(0x7faab8075470 msgr2=0x7faab8075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- 172.21.15.63:0/2811034094 shutdown_connections 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- 172.21.15.63:0/2811034094 wait complete. 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient: build_initial_monmap 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient: monmap: 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:29.225 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:29.226 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.226 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:29.226 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:29.226 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:29.226 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 Processor -- start 2024-09-16T05:38:29.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- start start 2024-09-16T05:38:29.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient: init 2024-09-16T05:38:29.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:29.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:29.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient(hunting): picked mon.noname-c con 0x7faab80a3c10 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:29.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient(hunting): picked mon.noname-a con 0x7faab80a4700 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:29.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient(hunting): picked mon.noname-b con 0x7faab80a5710 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:29.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab81237a0 con 0x7faab80a4700 2024-09-16T05:38:29.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab80a8440 con 0x7faab80a5710 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab81231e0 con 0x7faab80a3c10 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faabcc23640 10 monclient(hunting): authenticate will time out at 1165.994434s 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab6d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7faab80a5710 legacy=0x7faab81d3c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7faab80a3c10 0x7faab8135de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60008/0 (socket says 172.21.15.63:60008) 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faab659c640 1 -- 172.21.15.63:0/3692853273 learned_addr learned my addr 172.21.15.63:0/3692853273 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1703438249 0 0) 0x7faab81231e0 con 0x7faab80a3c10 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): my global_id is 5597 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa98003530 con 0x7faab80a3c10 2024-09-16T05:38:29.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1985093811 0 0) 0x7faab81237a0 con 0x7faab80a4700 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): my global_id is 5613 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faab81231e0 con 0x7faab80a4700 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 606950361 0 0) 0x7faab81231e0 con 0x7faab80a4700 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faab81237a0 con 0x7faab80a4700 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faaac0017f0 con 0x7faab80a4700 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:29.232 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2542890154 0 0) 0x7faa98003530 con 0x7faab80a3c10 2024-09-16T05:38:29.233 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faab81231e0 con 0x7faab80a3c10 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faaa80041a0 con 0x7faab80a3c10 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.214+0000 7faaa6ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.234 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2066620512 0 0) 0x7faab81237a0 con 0x7faab80a4700 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 >> v1:172.21.15.6:6790/0 conn(0x7faab80a3c10 legacy=0x7faab8135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 >> v1:172.21.15.63:6789/0 conn(0x7faab80a5710 legacy=0x7faab81d3c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 monclient: found mon.a 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab81234c0 con 0x7faab80a4700 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:29.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 5 monclient: authenticate success, global_id 5613 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 10 monclient: _renew_subs 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faab8121d90 con 0x7faab80a4700 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 10 monclient: _renew_subs 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faab81366c0 con 0x7faab80a4700 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faaac0047f0 con 0x7faab80a4700 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faaac004b40 con 0x7faab80a4700 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faab4d99640 4 set_mon_vals no callback set 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:29.236 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 10 monclient: dump: 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa9c005090 con 0x7faab80a4700 2024-09-16T05:38:29.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7faaac00f8e0 con 0x7faab80a4700 2024-09-16T05:38:29.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(135..135 src has 1..135) ==== 7061+0+0 (unknown 2614354796 0 0) 0x7faaac044d30 con 0x7faab80a4700 2024-09-16T05:38:29.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.218+0000 7faab659c640 10 monclient: get_auth_request con 0x7faa98036aa0 auth_method 0 2024-09-16T05:38:29.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.230+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7faaac018e60 con 0x7faab80a4700 2024-09-16T05:38:29.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.230+0000 7faaa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:29.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.230+0000 7faaa6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:29.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.394+0000 7faabcc23640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.394+0000 7faabcc23640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.394+0000 7faabcc23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.394+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7faa9c005380 con 0x7faab80a4700 2024-09-16T05:38:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.394+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v135) ==== 71+0+7 (unknown 1171993247 0 1352219128) 0x7faaac01d720 con 0x7faab80a4700 2024-09-16T05:38:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.394+0000 7faaa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.394+0000 7faaa6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:29.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.490+0000 7faabcc23640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.490+0000 7faabcc23640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.490+0000 7faabcc23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.490+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7faa9c005ac0 con 0x7faab80a4700 2024-09-16T05:38:29.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faaa6ffd640 1 -- 172.21.15.63:0/3692853273 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v135) ==== 71+0+7 (unknown 1171993247 0 1352219128) 0x7faaac0402a0 con 0x7faab80a4700 2024-09-16T05:38:29.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faaa6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:29.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faaa6ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:29.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 >> v1:172.21.15.63:6812/3643207211 conn(0x7faa98036aa0 legacy=0x7faa98038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faabcc23640 10 monclient: shutdown 2024-09-16T05:38:29.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faabcc23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:29.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 >> v1:172.21.15.6:6789/0 conn(0x7faab80a4700 legacy=0x7faab81d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 shutdown_connections 2024-09-16T05:38:29.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 >> 172.21.15.63:0/3692853273 conn(0x7faab8075470 msgr2=0x7faab80a7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:29.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 shutdown_connections 2024-09-16T05:38:29.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.494+0000 7faabcc23640 1 -- 172.21.15.63:0/3692853273 wait complete. 2024-09-16T05:38:29.512 INFO:tasks.workunit.client.0.smithi063.stdout:barset 2024-09-16T05:38:29.513 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set rm barset 2024-09-16T05:38:29.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.626+0000 7fea3d03d640 10 monclient: get_monmap_and_config 2024-09-16T05:38:29.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.626+0000 7fea3d03d640 10 monclient: build_initial_monmap 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.626+0000 7fea3d03d640 10 monclient: monmap: 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:29.632393+0000 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:29.632393+0000 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:29.633 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 Processor -- start 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- start start 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient: init 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient(hunting): picked mon.noname-c con 0x7fea3809b540 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient(hunting): picked mon.noname-b con 0x7fea380a1520 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient(hunting): picked mon.noname-a con 0x7fea38138d70 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea3809f890 con 0x7fea38138d70 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea38072e80 con 0x7fea3809b540 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea3809ce30 con 0x7fea380a1520 2024-09-16T05:38:29.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient(hunting): authenticate will time out at 1166.409239s 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea36d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fea3809b540 legacy=0x7fea3809fb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3759e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fea38138d70 0x7fea3813b160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36872/0 (socket says 172.21.15.63:36872) 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3759e640 1 -- 172.21.15.63:0/3170660497 learned_addr learned my addr 172.21.15.63:0/3170660497 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3659c640 1 --1- 172.21.15.63:0/3170660497 >> v1:172.21.15.6:6790/0 conn(0x7fea380a1520 0x7fea380a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60010/0 (socket says 172.21.15.63:60010) 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1428491502 0 0) 0x7fea3809f890 con 0x7fea38138d70 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): my global_id is 5616 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea24003530 con 0x7fea38138d70 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430911368 0 0) 0x7fea3809ce30 con 0x7fea380a1520 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:38:29.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): my global_id is 5600 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea3809f890 con 0x7fea380a1520 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3659c640 1 -- 172.21.15.63:0/3170660497 >> v1:172.21.15.6:6790/0 conn(0x7fea380a1520 legacy=0x7fea380a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3659c640 1 -- 172.21.15.63:0/3170660497 >> v1:172.21.15.6:6790/0 conn(0x7fea380a1520 legacy=0x7fea380a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3659c640 1 --1- 172.21.15.63:0/3170660497 >> v1:172.21.15.6:6790/0 conn(0x7fea380a1520 0x7fea380a0240 :-1 s=OPENED pgs=580 cs=1 l=1).handle_message read tag failed 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3659c640 1 --1- 172.21.15.63:0/3170660497 >> v1:172.21.15.6:6790/0 conn(0x7fea380a1520 0x7fea380a0240 :-1 s=OPENED pgs=580 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6790/0 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2827704762 0 0) 0x7fea24003530 con 0x7fea38138d70 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 >> v1:172.21.15.6:6790/0 conn(0x7fea380a1520 legacy=0x7fea380a0240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 >> v1:172.21.15.63:6789/0 conn(0x7fea3809b540 legacy=0x7fea3809fb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:38:29.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 monclient: found mon.noname-a 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3809d110 con 0x7fea38138d70 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: _finish_auth 0 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 5 monclient: authenticate success, global_id 5616 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea300019a0 con 0x7fea38138d70 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:29.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: dump: 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:29.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea30001c10 con 0x7fea38138d70 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 4 set_mon_vals no callback set 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 1 -- 172.21.15.63:0/3170660497 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea30002020 con 0x7fea38138d70 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea35d9b640 10 monclient: dump: 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.646 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient: shutdown 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- 172.21.15.63:0/3170660497 >> v1:172.21.15.6:6789/0 conn(0x7fea38138d70 legacy=0x7fea3813b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- 172.21.15.63:0/3170660497 shutdown_connections 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- 172.21.15.63:0/3170660497 >> 172.21.15.63:0/3170660497 conn(0x7fea380a5290 msgr2=0x7fea380a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- 172.21.15.63:0/3170660497 shutdown_connections 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 1 -- 172.21.15.63:0/3170660497 wait complete. 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient: build_initial_monmap 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.630+0000 7fea3d03d640 10 monclient: monmap: 2024-09-16T05:38:29.647 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 1 Processor -- start 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 1 -- start start 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: init 2024-09-16T05:38:29.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient(hunting): picked mon.noname-a con 0x7fea3809b540 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient(hunting): picked mon.noname-c con 0x7fea380a1520 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient(hunting): picked mon.noname-b con 0x7fea38138d70 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea38072210 con 0x7fea3809b540 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea38133360 con 0x7fea38138d70 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea38071f30 con 0x7fea380a1520 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient(hunting): authenticate will time out at 1166.412884s 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3759e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fea38138d70 legacy=0x7fea381d3af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea36d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fea3809b540 0x7fea3812fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36876/0 (socket says 172.21.15.63:36876) 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fea380a1520 0x7fea381d03c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60024/0 (socket says 172.21.15.63:60024) 2024-09-16T05:38:29.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea36d9d640 1 -- 172.21.15.63:0/3213596657 learned_addr learned my addr 172.21.15.63:0/3213596657 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860467404 0 0) 0x7fea38072210 con 0x7fea3809b540 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): my global_id is 5619 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea18003530 con 0x7fea3809b540 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3259858568 0 0) 0x7fea38071f30 con 0x7fea380a1520 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): my global_id is 5603 2024-09-16T05:38:29.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea38072210 con 0x7fea380a1520 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2745612368 0 0) 0x7fea18003530 con 0x7fea3809b540 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea38071f30 con 0x7fea3809b540 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea28004160 con 0x7fea3809b540 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.651 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1928914498 0 0) 0x7fea38072210 con 0x7fea380a1520 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea18003530 con 0x7fea380a1520 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea200018c0 con 0x7fea380a1520 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient(hunting): dump: 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.652 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1782702319 0 0) 0x7fea38071f30 con 0x7fea3809b540 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 >> v1:172.21.15.6:6790/0 conn(0x7fea380a1520 legacy=0x7fea381d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 >> v1:172.21.15.63:6789/0 conn(0x7fea38138d70 legacy=0x7fea381d3af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 monclient: found mon.a 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea38070ab0 con 0x7fea3809b540 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 5 monclient: authenticate success, global_id 5619 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: _renew_subs 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fea38130190 con 0x7fea3809b540 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: _renew_subs 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fea381cb270 con 0x7fea3809b540 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea28004410 con 0x7fea3809b540 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:29.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea280047e0 con 0x7fea3809b540 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3559a640 4 set_mon_vals no callback set 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 10 monclient: dump: 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:29.655 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:29.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:29.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:29.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1c005090 con 0x7fea3809b540 2024-09-16T05:38:29.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.634+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fea28004a00 con 0x7fea3809b540 2024-09-16T05:38:29.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.638+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(135..135 src has 1..135) ==== 7061+0+0 (unknown 2614354796 0 0) 0x7fea28044fa0 con 0x7fea3809b540 2024-09-16T05:38:29.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.638+0000 7fea3659c640 10 monclient: get_auth_request con 0x7fea18036a20 auth_method 0 2024-09-16T05:38:29.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.650+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fea28015480 con 0x7fea3809b540 2024-09-16T05:38:29.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.650+0000 7fea2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:29.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.650+0000 7fea2f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:29.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.818+0000 7fea3d03d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T05:38:29.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.818+0000 7fea3d03d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T05:38:29.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.818+0000 7fea3d03d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:29.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:29.818+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) -- 0x7fea1c005380 con 0x7fea3809b540 2024-09-16T05:38:30.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:30.630+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v136) ==== 89+0+0 (unknown 3210802562 0 0) 0x7fea28014d80 con 0x7fea3809b540 2024-09-16T05:38:30.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:30.630+0000 7fea2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T05:38:30.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:30.630+0000 7fea2f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:30.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:30.746+0000 7fea3d03d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T05:38:30.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:30.746+0000 7fea3d03d640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T05:38:30.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:30.746+0000 7fea3d03d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:30.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:30.746+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) -- 0x7fea1c005770 con 0x7fea3809b540 2024-09-16T05:38:31.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.010+0000 7fea2f7fe640 1 -- 172.21.15.63:0/3213596657 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v137) ==== 89+0+0 (unknown 4245866859 0 0) 0x7fea280190d0 con 0x7fea3809b540 2024-09-16T05:38:31.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.010+0000 7fea2f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T05:38:31.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.010+0000 7fea2f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:31.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.014+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 >> v1:172.21.15.63:6812/3643207211 conn(0x7fea18036a20 legacy=0x7fea18038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.014+0000 7fea3d03d640 10 monclient: shutdown 2024-09-16T05:38:31.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.014+0000 7fea3d03d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:31.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.014+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 >> v1:172.21.15.6:6789/0 conn(0x7fea3809b540 legacy=0x7fea3812fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.014+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 shutdown_connections 2024-09-16T05:38:31.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.014+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 >> 172.21.15.63:0/3213596657 conn(0x7fea380a5290 msgr2=0x7fea380a1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:31.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.014+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 shutdown_connections 2024-09-16T05:38:31.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.014+0000 7fea3d03d640 1 -- 172.21.15.63:0/3213596657 wait complete. 2024-09-16T05:38:31.046 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set ls 2024-09-16T05:38:31.046 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -c barset 2024-09-16T05:38:31.046 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep -q 0 2024-09-16T05:38:31.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient: get_monmap_and_config 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient: build_initial_monmap 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient: monmap: 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:31.162644+0000 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:31.162644+0000 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:31.163 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:31.164 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 1 Processor -- start 2024-09-16T05:38:31.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 1 -- start start 2024-09-16T05:38:31.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient: init 2024-09-16T05:38:31.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:31.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient(hunting): picked mon.noname-a con 0x7f96040a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient(hunting): picked mon.noname-c con 0x7f96040a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.158+0000 7f960b89f640 10 monclient(hunting): picked mon.noname-b con 0x7f96040a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96040a4fd0 con 0x7f96040a3ca0 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f9608e3a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f96040a08a0 legacy=0x7f960413beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96040a5b40 con 0x7f96040a08a0 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96040a1a70 con 0x7f96040a4790 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 10 monclient(hunting): authenticate will time out at 1167.940135s 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960963b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f96040a3ca0 0x7f96040a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36878/0 (socket says 172.21.15.63:36878) 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960963b640 1 -- 172.21.15.63:0/2644756973 learned_addr learned my addr 172.21.15.63:0/2644756973 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:31.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432100739 0 0) 0x7f96040a1a70 con 0x7f96040a4790 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient(hunting): my global_id is 5606 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95f0003530 con 0x7f96040a4790 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2194677080 0 0) 0x7f96040a4fd0 con 0x7f96040a3ca0 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient(hunting): my global_id is 5622 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96040a1a70 con 0x7f96040a3ca0 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 35432650 0 0) 0x7f95f0003530 con 0x7f96040a4790 2024-09-16T05:38:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 >> v1:172.21.15.63:6789/0 conn(0x7f96040a08a0 legacy=0x7f960413beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 >> v1:172.21.15.6:6789/0 conn(0x7f96040a3ca0 legacy=0x7f96040a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 monclient: found mon.noname-b 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96040a53e0 con 0x7f96040a4790 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: _finish_auth 0 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f95fc0019a0 con 0x7f96040a4790 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:31.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: dump: 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 5 monclient: authenticate success, global_id 5606 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:31.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95fc001c50 con 0x7f96040a4790 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 1 -- 172.21.15.63:0/2644756973 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f95fc002020 con 0x7f96040a4790 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 4 set_mon_vals no callback set 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f95fbfff640 10 monclient: dump: 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.171 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.172 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.172 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.172 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 10 monclient: shutdown 2024-09-16T05:38:31.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:31.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- 172.21.15.63:0/2644756973 >> v1:172.21.15.6:6790/0 conn(0x7f96040a4790 legacy=0x7f960413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- 172.21.15.63:0/2644756973 shutdown_connections 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- 172.21.15.63:0/2644756973 >> 172.21.15.63:0/2644756973 conn(0x7f96040a67c0 msgr2=0x7f9604075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- 172.21.15.63:0/2644756973 shutdown_connections 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- 172.21.15.63:0/2644756973 wait complete. 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 10 monclient: build_initial_monmap 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 10 monclient: monmap: 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.177 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 Processor -- start 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 1 -- start start 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.162+0000 7f960b89f640 10 monclient: init 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient(hunting): picked mon.noname-b con 0x7f96040a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient(hunting): picked mon.noname-a con 0x7f96040a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient(hunting): picked mon.noname-c con 0x7f96040a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960963b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f96040a08a0 legacy=0x7f96041cb8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:31.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9604126f30 con 0x7f96040a3ca0 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96040a8750 con 0x7f96040a08a0 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96040a8220 con 0x7f96040a4790 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient(hunting): authenticate will time out at 1167.944376s 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f9608e3a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f96040a3ca0 0x7f96041d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36890/0 (socket says 172.21.15.63:36890) 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f9608e3a640 1 -- 172.21.15.63:0/906213495 learned_addr learned my addr 172.21.15.63:0/906213495 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516249022 0 0) 0x7f9604126f30 con 0x7f96040a3ca0 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:31.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): my global_id is 5625 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95e8003560 con 0x7f96040a3ca0 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2397542717 0 0) 0x7f96040a8220 con 0x7f96040a4790 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): my global_id is 5609 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9604126f30 con 0x7f96040a4790 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1323636525 0 0) 0x7f9604126f30 con 0x7f96040a4790 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96040a8220 con 0x7f96040a4790 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f95fc001fa0 con 0x7f96040a4790 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:31.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1300551254 0 0) 0x7f95e8003560 con 0x7f96040a3ca0 2024-09-16T05:38:31.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9604126f30 con 0x7f96040a3ca0 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f95f4004160 con 0x7f96040a3ca0 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.182 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1441152235 0 0) 0x7f96040a8220 con 0x7f96040a4790 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 >> v1:172.21.15.63:6789/0 conn(0x7f96040a08a0 legacy=0x7f96041cb8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 >> v1:172.21.15.6:6789/0 conn(0x7f96040a3ca0 legacy=0x7f96041d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 monclient: found mon.c 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96040a0500 con 0x7f96040a4790 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:31.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 5 monclient: authenticate success, global_id 5609 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient: _renew_subs 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9604126970 con 0x7f96040a4790 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient: _renew_subs 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9604123fb0 con 0x7f96040a4790 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95fc001c50 con 0x7f96040a4790 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f95fc003ed0 con 0x7f96040a4790 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95faffd640 4 set_mon_vals no callback set 2024-09-16T05:38:31.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 10 monclient: dump: 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:31.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:31.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e4005090 con 0x7f96040a4790 2024-09-16T05:38:31.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f95fc00ec70 con 0x7f96040a4790 2024-09-16T05:38:31.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(137..137 src has 1..137) ==== 6449+0+0 (unknown 288592062 0 0) 0x7f95fc043c70 con 0x7f96040a4790 2024-09-16T05:38:31.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.166+0000 7f960963b640 10 monclient: get_auth_request con 0x7f95e8036a00 auth_method 0 2024-09-16T05:38:31.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.182+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f95fc018140 con 0x7f96040a4790 2024-09-16T05:38:31.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.182+0000 7f95f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:31.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.182+0000 7f95f9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:31.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.342+0000 7f960b89f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:31.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.342+0000 7f960b89f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:31.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.342+0000 7f960b89f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:31.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.342+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f95e4005380 con 0x7f96040a4790 2024-09-16T05:38:31.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.346+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v137) ==== 71+0+0 (unknown 3410384225 0 0) 0x7f95fc01ca00 con 0x7f96040a4790 2024-09-16T05:38:31.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.346+0000 7f95f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:31.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.346+0000 7f95f9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:31.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.450+0000 7f960b89f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:31.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.450+0000 7f960b89f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:31.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.450+0000 7f960b89f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:31.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.450+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f95e4005ac0 con 0x7f96040a4790 2024-09-16T05:38:31.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.450+0000 7f95f9ffb640 1 -- 172.21.15.63:0/906213495 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v137) ==== 71+0+0 (unknown 3410384225 0 0) 0x7f95fc03f360 con 0x7f96040a4790 2024-09-16T05:38:31.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.450+0000 7f95f9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:31.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.450+0000 7f95f9ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:31.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.454+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 >> v1:172.21.15.63:6812/3643207211 conn(0x7f95e8036a00 legacy=0x7f95e8038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.454+0000 7f960b89f640 10 monclient: shutdown 2024-09-16T05:38:31.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.454+0000 7f960b89f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:31.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.454+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 >> v1:172.21.15.6:6790/0 conn(0x7f96040a4790 legacy=0x7f96041d3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.454+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 shutdown_connections 2024-09-16T05:38:31.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.454+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 >> 172.21.15.63:0/906213495 conn(0x7f96040a67c0 msgr2=0x7f96041bcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:31.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.454+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 shutdown_connections 2024-09-16T05:38:31.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.454+0000 7f960b89f640 1 -- 172.21.15.63:0/906213495 wait complete. 2024-09-16T05:38:31.473 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.590+0000 7fe104094640 10 monclient: get_monmap_and_config 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.590+0000 7fe104094640 10 monclient: build_initial_monmap 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.590+0000 7fe104094640 10 monclient: monmap: 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:31.595608+0000 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:31.595608+0000 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:31.596 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:31.597 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:31.597 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.590+0000 7fe104094640 1 Processor -- start 2024-09-16T05:38:31.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.590+0000 7fe104094640 1 -- start start 2024-09-16T05:38:31.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.590+0000 7fe104094640 10 monclient: init 2024-09-16T05:38:31.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:31.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient(hunting): picked mon.noname-c con 0x7fe0fc0a12b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient(hunting): picked mon.noname-b con 0x7fe0fc072fa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient(hunting): picked mon.noname-a con 0x7fe0fc1395a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0fc0a05b0 con 0x7fe0fc1395a0 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe101e30640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe0fc0a12b0 legacy=0x7fe0fc09f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0fc09c510 con 0x7fe0fc0a12b0 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0fc09ce10 con 0x7fe0fc072fa0 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient(hunting): authenticate will time out at 1168.372740s 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe102631640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe0fc1395a0 0x7fe0fc13b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36902/0 (socket says 172.21.15.63:36902) 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe10162f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe0fc072fa0 0x7fe0fc09fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60056/0 (socket says 172.21.15.63:60056) 2024-09-16T05:38:31.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe102631640 1 -- 172.21.15.63:0/3338527776 learned_addr learned my addr 172.21.15.63:0/3338527776 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 774845160 0 0) 0x7fe0fc09ce10 con 0x7fe0fc072fa0 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient(hunting): _negotiate 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient(hunting): my global_id is 5612 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0e8003530 con 0x7fe0fc072fa0 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2757161163 0 0) 0x7fe0fc0a05b0 con 0x7fe0fc1395a0 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient(hunting): _negotiate 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient(hunting): my global_id is 5628 2024-09-16T05:38:31.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0fc09ce10 con 0x7fe0fc1395a0 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1208794260 0 0) 0x7fe0e8003530 con 0x7fe0fc072fa0 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 >> v1:172.21.15.63:6789/0 conn(0x7fe0fc0a12b0 legacy=0x7fe0fc09f790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 >> v1:172.21.15.6:6789/0 conn(0x7fe0fc1395a0 legacy=0x7fe0fc13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: _finish_hunting 0 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 monclient: found mon.noname-b 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0fc09d0f0 con 0x7fe0fc072fa0 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: _finish_auth 0 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: _check_auth_tickets 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0ec001620 con 0x7fe0fc072fa0 2024-09-16T05:38:31.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: dump: 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 5 monclient: authenticate success, global_id 5612 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0ec0018d0 con 0x7fe0fc072fa0 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 1 -- 172.21.15.63:0/3338527776 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0ec001ca0 con 0x7fe0fc072fa0 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe100e2e640 10 monclient: dump: 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.605 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 4 set_mon_vals no callback set 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient: shutdown 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 1 -- 172.21.15.63:0/3338527776 >> v1:172.21.15.6:6790/0 conn(0x7fe0fc072fa0 legacy=0x7fe0fc09fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 1 -- 172.21.15.63:0/3338527776 shutdown_connections 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 1 -- 172.21.15.63:0/3338527776 >> 172.21.15.63:0/3338527776 conn(0x7fe0fc0a1c90 msgr2=0x7fe0fc0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:31.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 1 -- 172.21.15.63:0/3338527776 shutdown_connections 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 1 -- 172.21.15.63:0/3338527776 wait complete. 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient: build_initial_monmap 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.594+0000 7fe104094640 10 monclient: monmap: 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:31.607 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 1 Processor -- start 2024-09-16T05:38:31.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 1 -- start start 2024-09-16T05:38:31.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: init 2024-09-16T05:38:31.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:31.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:31.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient(hunting): picked mon.noname-a con 0x7fe0fc072fa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:31.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient(hunting): picked mon.noname-b con 0x7fe0fc0a12b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:31.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient(hunting): picked mon.noname-c con 0x7fe0fc1395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:31.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0fc11f340 con 0x7fe0fc072fa0 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0fc0a3d30 con 0x7fe0fc0a12b0 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0fc0a3870 con 0x7fe0fc1395a0 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient(hunting): authenticate will time out at 1168.376677s 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe10162f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe0fc0a12b0 legacy=0x7fe0fc1cbc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe101e30640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe0fc072fa0 0x7fe0fc134c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36908/0 (socket says 172.21.15.63:36908) 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe101e30640 1 -- 172.21.15.63:0/1234227044 learned_addr learned my addr 172.21.15.63:0/1234227044 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3381605182 0 0) 0x7fe0fc11f340 con 0x7fe0fc072fa0 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): my global_id is 5631 2024-09-16T05:38:31.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0e0003560 con 0x7fe0fc072fa0 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3882602120 0 0) 0x7fe0fc0a3870 con 0x7fe0fc1395a0 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): my global_id is 5615 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0fc11f340 con 0x7fe0fc1395a0 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1658517717 0 0) 0x7fe0e0003560 con 0x7fe0fc072fa0 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0fc0a3870 con 0x7fe0fc072fa0 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0f8004160 con 0x7fe0fc072fa0 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:31.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): dump: 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:31.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 612779591 0 0) 0x7fe0fc11f340 con 0x7fe0fc1395a0 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0e0003560 con 0x7fe0fc1395a0 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0f4001bc0 con 0x7fe0fc1395a0 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient(hunting): dump: 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.615 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3290979241 0 0) 0x7fe0fc0a3870 con 0x7fe0fc072fa0 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 >> v1:172.21.15.6:6790/0 conn(0x7fe0fc1395a0 legacy=0x7fe0fc1cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 >> v1:172.21.15.63:6789/0 conn(0x7fe0fc0a12b0 legacy=0x7fe0fc1cbc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 monclient: found mon.a 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0fc09c0f0 con 0x7fe0fc072fa0 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:31.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 5 monclient: authenticate success, global_id 5631 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: _renew_subs 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0fc11f060 con 0x7fe0fc072fa0 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: _renew_subs 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0fc1353a0 con 0x7fe0fc072fa0 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0f80043c0 con 0x7fe0fc072fa0 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0f8004710 con 0x7fe0fc072fa0 2024-09-16T05:38:31.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f3fff640 4 set_mon_vals no callback set 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 10 monclient: dump: 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:31.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:31.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:31.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e4005090 con 0x7fe0fc072fa0 2024-09-16T05:38:31.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe0f8004930 con 0x7fe0fc072fa0 2024-09-16T05:38:31.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.598+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(137..137 src has 1..137) ==== 6449+0+0 (unknown 288592062 0 0) 0x7fe0f8044c70 con 0x7fe0fc072fa0 2024-09-16T05:38:31.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.602+0000 7fe10162f640 10 monclient: get_auth_request con 0x7fe0e0036aa0 auth_method 0 2024-09-16T05:38:31.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.614+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe0f80153b0 con 0x7fe0fc072fa0 2024-09-16T05:38:31.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.614+0000 7fe0f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:31.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.614+0000 7fe0f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:31.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.782+0000 7fe104094640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:31.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.782+0000 7fe104094640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:31.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.782+0000 7fe104094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:31.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:31.782+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7fe0e4005380 con 0x7fe0fc072fa0 2024-09-16T05:38:32.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.642+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v138) ==== 82+0+0 (unknown 2061534354 0 0) 0x7fe0f8014cb0 con 0x7fe0fc072fa0 2024-09-16T05:38:32.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.642+0000 7fe0f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:32.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.642+0000 7fe0f27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:32.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.750+0000 7fe104094640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:32.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.750+0000 7fe104094640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:32.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.750+0000 7fe104094640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:32.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.750+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7fe0e4005770 con 0x7fe0fc072fa0 2024-09-16T05:38:32.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.750+0000 7fe0f27fc640 1 -- 172.21.15.63:0/1234227044 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v138) ==== 115+0+0 (unknown 2508283433 0 0) 0x7fe0f8019000 con 0x7fe0fc072fa0 2024-09-16T05:38:32.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.750+0000 7fe0f27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:32.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.750+0000 7fe0f27fc640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T05:38:32.755 INFO:tasks.workunit.client.0.smithi063.stderr:compat weight-set already created 2024-09-16T05:38:32.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.754+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe0e0036aa0 legacy=0x7fe0e0038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:32.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.754+0000 7fe104094640 10 monclient: shutdown 2024-09-16T05:38:32.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.754+0000 7fe104094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:32.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.754+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 >> v1:172.21.15.6:6789/0 conn(0x7fe0fc072fa0 legacy=0x7fe0fc134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:32.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.754+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 shutdown_connections 2024-09-16T05:38:32.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.754+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 >> 172.21.15.63:0/1234227044 conn(0x7fe0fc0a1c90 msgr2=0x7fe0fc0a2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:32.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.754+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 shutdown_connections 2024-09-16T05:38:32.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.754+0000 7fe104094640 1 -- 172.21.15.63:0/1234227044 wait complete. 2024-09-16T05:38:32.772 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set ls 2024-09-16T05:38:32.772 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '(compat)' 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient: get_monmap_and_config 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient: build_initial_monmap 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient: monmap: 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:32.893824+0000 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:32.893824+0000 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:32.894 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:32.895 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:32.895 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:32.895 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:32.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 1 Processor -- start 2024-09-16T05:38:32.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 1 -- start start 2024-09-16T05:38:32.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient: init 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec0a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec0a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec0a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ec0a4fd0 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ec0a5b40 con 0x7f37ec0a08a0 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ec0a1a70 con 0x7f37ec0a4790 2024-09-16T05:38:32.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:32.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f1c6e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f37ec0a08a0 legacy=0x7f37ec13beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:32.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f46d3640 10 monclient(hunting): authenticate will time out at 1169.671577s 2024-09-16T05:38:32.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f246f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f37ec0a3ca0 0x7f37ec0a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36912/0 (socket says 172.21.15.63:36912) 2024-09-16T05:38:32.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.890+0000 7f37f246f640 1 -- 172.21.15.63:0/2321010843 learned_addr learned my addr 172.21.15.63:0/2321010843 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481586525 0 0) 0x7f37ec0a1a70 con 0x7f37ec0a4790 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient(hunting): _negotiate 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient(hunting): my global_id is 5618 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37d4003530 con 0x7f37ec0a4790 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3107316760 0 0) 0x7f37ec0a4fd0 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient(hunting): _negotiate 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient(hunting): my global_id is 5634 2024-09-16T05:38:32.901 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37ec0a1a70 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 61521799 0 0) 0x7f37d4003530 con 0x7f37ec0a4790 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 >> v1:172.21.15.63:6789/0 conn(0x7f37ec0a08a0 legacy=0x7f37ec13beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 >> v1:172.21.15.6:6789/0 conn(0x7f37ec0a3ca0 legacy=0x7f37ec0a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: _finish_hunting 0 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 monclient: found mon.noname-b 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec0a53e0 con 0x7f37ec0a4790 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: _finish_auth 0 2024-09-16T05:38:32.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: _check_auth_tickets 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f37e4001620 con 0x7f37ec0a4790 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: dump: 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:32.903 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 5 monclient: authenticate success, global_id 5618 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37e40018d0 con 0x7f37ec0a4790 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 1 -- 172.21.15.63:0/2321010843 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f37e4001ca0 con 0x7f37ec0a4790 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f146d640 10 monclient: dump: 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:32.904 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 4 set_mon_vals no callback set 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient: shutdown 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- 172.21.15.63:0/2321010843 >> v1:172.21.15.6:6790/0 conn(0x7f37ec0a4790 legacy=0x7f37ec13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:32.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- 172.21.15.63:0/2321010843 shutdown_connections 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- 172.21.15.63:0/2321010843 >> 172.21.15.63:0/2321010843 conn(0x7f37ec0a67c0 msgr2=0x7f37ec075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- 172.21.15.63:0/2321010843 shutdown_connections 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- 172.21.15.63:0/2321010843 wait complete. 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient: build_initial_monmap 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient: monmap: 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:32.906 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 Processor -- start 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- start start 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient: init 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec0a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec0a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec0a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ec127c80 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ec126fc0 con 0x7f37ec0a4790 2024-09-16T05:38:32.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37ec125450 con 0x7f37ec0a08a0 2024-09-16T05:38:32.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f2c70640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f37ec0a4790 legacy=0x7f37ec1d4010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:32.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:32.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.894+0000 7f37f46d3640 10 monclient(hunting): authenticate will time out at 1169.675964s 2024-09-16T05:38:32.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f1c6e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f37ec0a3ca0 0x7f37ec1d08e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36926/0 (socket says 172.21.15.63:36926) 2024-09-16T05:38:32.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f1c6e640 1 -- 172.21.15.63:0/3337068111 learned_addr learned my addr 172.21.15.63:0/3337068111 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1180747542 0 0) 0x7f37ec125450 con 0x7f37ec0a08a0 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): my global_id is 5621 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37d8003560 con 0x7f37ec0a08a0 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 880552435 0 0) 0x7f37e0003aa0 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:32.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): my global_id is 5637 2024-09-16T05:38:32.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37ec125450 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1267216927 0 0) 0x7f37d8003560 con 0x7f37ec0a08a0 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37e0003aa0 con 0x7f37ec0a08a0 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f37dc001c40 con 0x7f37ec0a08a0 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): dump: 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.913 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2195212109 0 0) 0x7f37ec125450 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37d8003560 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f37e0004330 con 0x7f37ec0a3ca0 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:32.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient(hunting): dump: 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 735569401 0 0) 0x7f37e0003aa0 con 0x7f37ec0a08a0 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 >> v1:172.21.15.63:6789/0 conn(0x7f37ec0a4790 legacy=0x7f37ec1d4010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 >> v1:172.21.15.6:6789/0 conn(0x7f37ec0a3ca0 legacy=0x7f37ec1d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:32.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 monclient: found mon.c 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec1272a0 con 0x7f37ec0a08a0 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 5 monclient: authenticate success, global_id 5621 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 10 monclient: _renew_subs 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:32.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f37ec0a8750 con 0x7f37ec0a08a0 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 10 monclient: _renew_subs 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f37ec126ce0 con 0x7f37ec0a08a0 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37dc004b70 con 0x7f37ec0a08a0 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f37dc004f40 con 0x7f37ec0a08a0 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:32.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 10 monclient: dump: 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f0c6c640 4 set_mon_vals no callback set 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:32.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:32.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:32.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37cc005090 con 0x7f37ec0a08a0 2024-09-16T05:38:32.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f37dc0050e0 con 0x7f37ec0a08a0 2024-09-16T05:38:32.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(138..138 src has 1..138) ==== 6781+0+0 (unknown 1163854782 0 0) 0x7f37dc044fc0 con 0x7f37ec0a08a0 2024-09-16T05:38:32.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.898+0000 7f37f1c6e640 10 monclient: get_auth_request con 0x7f37d8036a80 auth_method 0 2024-09-16T05:38:32.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.910+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f37dc015620 con 0x7f37ec0a08a0 2024-09-16T05:38:32.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.914+0000 7f37eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:32.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:32.914+0000 7f37eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:33.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.082+0000 7f37f46d3640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:33.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.082+0000 7f37f46d3640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:33.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.082+0000 7f37f46d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.082+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f37cc005380 con 0x7f37ec0a08a0 2024-09-16T05:38:33.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.082+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v138) ==== 71+0+9 (unknown 3856367546 0 1822970295) 0x7f37dc014f20 con 0x7f37ec0a08a0 2024-09-16T05:38:33.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.082+0000 7f37eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:33.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.082+0000 7f37eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:33.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:33.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:33.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f37cc005ac0 con 0x7f37ec0a08a0 2024-09-16T05:38:33.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37eaffd640 1 -- 172.21.15.63:0/3337068111 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v138) ==== 71+0+9 (unknown 3856367546 0 1822970295) 0x7f37dc01dad0 con 0x7f37ec0a08a0 2024-09-16T05:38:33.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37eaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T05:38:33.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37eaffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:33.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 >> v1:172.21.15.63:6812/3643207211 conn(0x7f37d8036a80 legacy=0x7f37d8038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:33.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 10 monclient: shutdown 2024-09-16T05:38:33.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:33.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 >> v1:172.21.15.6:6790/0 conn(0x7f37ec0a08a0 legacy=0x7f37ec1cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:33.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 shutdown_connections 2024-09-16T05:38:33.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 >> 172.21.15.63:0/3337068111 conn(0x7f37ec0a67c0 msgr2=0x7f37ec0a2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:33.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.186+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 shutdown_connections 2024-09-16T05:38:33.193 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.190+0000 7f37f46d3640 1 -- 172.21.15.63:0/3337068111 wait complete. 2024-09-16T05:38:33.206 INFO:tasks.workunit.client.0.smithi063.stdout:(compat) 2024-09-16T05:38:33.207 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient: get_monmap_and_config 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient: build_initial_monmap 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient: monmap: 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:33.325953+0000 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:33.325953+0000 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:33.326 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:33.327 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:33.327 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:33.327 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:33.327 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:33.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 1 Processor -- start 2024-09-16T05:38:33.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 1 -- start start 2024-09-16T05:38:33.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient: init 2024-09-16T05:38:33.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:33.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:33.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient(hunting): picked mon.noname-b con 0x7f73680a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:33.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient(hunting): picked mon.noname-a con 0x7f73680a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:33.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient(hunting): picked mon.noname-c con 0x7f73680a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73680a4fd0 con 0x7f73680a08a0 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73680a5b40 con 0x7f73680a4790 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73680a1a70 con 0x7f73680a3ca0 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f736e0c4640 10 monclient(hunting): authenticate will time out at 1170.102925s 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f7367fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f73680a4790 legacy=0x7f736813f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f7366ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73680a08a0 0x7f736813beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36936/0 (socket says 172.21.15.63:36936) 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73677fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f73680a3ca0 0x7f73680a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60074/0 (socket says 172.21.15.63:60074) 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f7366ffd640 1 -- 172.21.15.63:0/2379710025 learned_addr learned my addr 172.21.15.63:0/2379710025 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2161140724 0 0) 0x7f73680a1a70 con 0x7f73680a3ca0 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:33.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 10 monclient(hunting): my global_id is 5624 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7350003530 con 0x7f73680a3ca0 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279409860 0 0) 0x7f73680a4fd0 con 0x7f73680a08a0 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 10 monclient(hunting): my global_id is 5640 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.322+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73680a1a70 con 0x7f73680a08a0 2024-09-16T05:38:33.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1123112891 0 0) 0x7f7350003530 con 0x7f73680a3ca0 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 >> v1:172.21.15.63:6789/0 conn(0x7f73680a4790 legacy=0x7f736813f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 >> v1:172.21.15.6:6789/0 conn(0x7f73680a08a0 legacy=0x7f736813beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 1 monclient: found mon.noname-b 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73680a53e0 con 0x7f73680a3ca0 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7358001620 con 0x7f73680a3ca0 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:33.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: dump: 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 5 monclient: authenticate success, global_id 5624 2024-09-16T05:38:33.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73580018d0 con 0x7f73680a3ca0 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 1 -- 172.21.15.63:0/2379710025 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7358001ca0 con 0x7f73680a3ca0 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73667fc640 10 monclient: dump: 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 4 set_mon_vals no callback set 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient: shutdown 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- 172.21.15.63:0/2379710025 >> v1:172.21.15.6:6790/0 conn(0x7f73680a3ca0 legacy=0x7f73680a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- 172.21.15.63:0/2379710025 shutdown_connections 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- 172.21.15.63:0/2379710025 >> 172.21.15.63:0/2379710025 conn(0x7f73680a67c0 msgr2=0x7f7368075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- 172.21.15.63:0/2379710025 shutdown_connections 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- 172.21.15.63:0/2379710025 wait complete. 2024-09-16T05:38:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient: build_initial_monmap 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient: monmap: 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:33.337 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:33.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 Processor -- start 2024-09-16T05:38:33.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- start start 2024-09-16T05:38:33.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient: init 2024-09-16T05:38:33.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:33.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:33.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient(hunting): picked mon.noname-a con 0x7f73680a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient(hunting): picked mon.noname-c con 0x7f73680a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient(hunting): picked mon.noname-b con 0x7f73680a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7368126650 con 0x7f73680a08a0 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73680a8930 con 0x7f73680a4790 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73680a8400 con 0x7f73680a3ca0 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f736e0c4640 10 monclient(hunting): authenticate will time out at 1170.106647s 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f7367fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f73680a4790 legacy=0x7f73681d3c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73677fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73680a08a0 0x7f73681367a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:36944/0 (socket says 172.21.15.63:36944) 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f73677fe640 1 -- 172.21.15.63:0/1320320965 learned_addr learned my addr 172.21.15.63:0/1320320965 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2843890265 0 0) 0x7f7368126650 con 0x7f73680a08a0 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:33.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): my global_id is 5643 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7354003530 con 0x7f73680a08a0 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 898818546 0 0) 0x7f73680a8400 con 0x7f73680a3ca0 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): my global_id is 5627 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7368126650 con 0x7f73680a3ca0 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2789916788 0 0) 0x7f7368126650 con 0x7f73680a3ca0 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73680a8400 con 0x7f73680a3ca0 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f735c0017e0 con 0x7f73680a3ca0 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:33.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): dump: 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.326+0000 7f734ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:33.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 424260002 0 0) 0x7f7354003530 con 0x7f73680a08a0 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7368126650 con 0x7f73680a08a0 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7358001c20 con 0x7f73680a08a0 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient(hunting): dump: 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:33.345 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 957311085 0 0) 0x7f73680a8400 con 0x7f73680a3ca0 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 >> v1:172.21.15.63:6789/0 conn(0x7f73680a4790 legacy=0x7f73681d3c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 >> v1:172.21.15.6:6789/0 conn(0x7f73680a08a0 legacy=0x7f73681367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 monclient: found mon.c 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7368126bd0 con 0x7f73680a3ca0 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient: _finish_auth 0 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:33.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 5 monclient: authenticate success, global_id 5627 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 10 monclient: _renew_subs 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7368126eb0 con 0x7f73680a3ca0 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 10 monclient: _renew_subs 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7368136eb0 con 0x7f73680a3ca0 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f735c0047f0 con 0x7f73680a3ca0 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f735c004bc0 con 0x7f73680a3ca0 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f7365ffb640 4 set_mon_vals no callback set 2024-09-16T05:38:33.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 10 monclient: dump: 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:33.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7348005090 con 0x7f73680a3ca0 2024-09-16T05:38:33.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f735c004de0 con 0x7f73680a3ca0 2024-09-16T05:38:33.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(138..138 src has 1..138) ==== 6781+0+0 (unknown 1163854782 0 0) 0x7f735c044cb0 con 0x7f73680a3ca0 2024-09-16T05:38:33.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.330+0000 7f73677fe640 10 monclient: get_auth_request con 0x7f73540369b0 auth_method 0 2024-09-16T05:38:33.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.342+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f735c018f00 con 0x7f73680a3ca0 2024-09-16T05:38:33.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.342+0000 7f734ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:33.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.342+0000 7f734ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:33.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.510+0000 7f736e0c4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:38:33.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.510+0000 7f736e0c4640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:38:33.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.510+0000 7f736e0c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.510+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7f7348005380 con 0x7f73680a3ca0 2024-09-16T05:38:33.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.658+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v139) ==== 78+0+0 (unknown 192515294 0 0) 0x7f735c01d7c0 con 0x7f73680a3ca0 2024-09-16T05:38:33.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.658+0000 7f734ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:38:33.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.658+0000 7f734ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:33.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.758+0000 7f736e0c4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:38:33.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.758+0000 7f736e0c4640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:38:33.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.758+0000 7f736e0c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:33.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:33.758+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7f7348005770 con 0x7f73680a3ca0 2024-09-16T05:38:34.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f734ffff640 1 -- 172.21.15.63:0/1320320965 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v140) ==== 78+0+0 (unknown 3419606228 0 0) 0x7f735c040340 con 0x7f73680a3ca0 2024-09-16T05:38:34.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f734ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:38:34.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f734ffff640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:38:34.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 >> v1:172.21.15.63:6812/3643207211 conn(0x7f73540369b0 legacy=0x7f7354038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:34.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f736e0c4640 10 monclient: shutdown 2024-09-16T05:38:34.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f736e0c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:34.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 >> v1:172.21.15.6:6790/0 conn(0x7f73680a3ca0 legacy=0x7f73681d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:34.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 shutdown_connections 2024-09-16T05:38:34.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 >> 172.21.15.63:0/1320320965 conn(0x7f73680a67c0 msgr2=0x7f7368137460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:34.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 shutdown_connections 2024-09-16T05:38:34.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.662+0000 7f736e0c4640 1 -- 172.21.15.63:0/1320320965 wait complete. 2024-09-16T05:38:34.690 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create cool 2 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.810+0000 7f2289f26640 10 monclient: get_monmap_and_config 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.810+0000 7f2289f26640 10 monclient: build_initial_monmap 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.810+0000 7f2289f26640 10 monclient: monmap: 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:34.815694+0000 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:34.815694+0000 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:34.816 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:34.817 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:34.817 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:34.817 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:34.817 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:34.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.810+0000 7f2289f26640 1 Processor -- start 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.810+0000 7f2289f26640 1 -- start start 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.810+0000 7f2289f26640 10 monclient: init 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient(hunting): picked mon.noname-b con 0x7f228409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient(hunting): picked mon.noname-c con 0x7f22840a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient(hunting): picked mon.noname-a con 0x7f228409b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2284074200 con 0x7f228409b780 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f228409f900 con 0x7f22840a0280 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f228409e870 con 0x7f228409ac90 2024-09-16T05:38:34.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient(hunting): authenticate will time out at 1171.592626s 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2282ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f22840a0280 legacy=0x7f22841398b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2283fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f228409b780 0x7f228413cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35732/0 (socket says 172.21.15.63:35732) 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22837fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f228409ac90 0x7f228409b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:50956/0 (socket says 172.21.15.63:50956) 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2283fff640 1 -- 172.21.15.63:0/3511505277 learned_addr learned my addr 172.21.15.63:0/3511505277 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260861055 0 0) 0x7f228409e870 con 0x7f228409ac90 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient(hunting): my global_id is 5630 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f226c003530 con 0x7f228409ac90 2024-09-16T05:38:34.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1852501257 0 0) 0x7f2284074200 con 0x7f228409b780 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient(hunting): my global_id is 5646 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f228409e870 con 0x7f228409b780 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3619760070 0 0) 0x7f228409e870 con 0x7f228409b780 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 >> v1:172.21.15.6:6790/0 conn(0x7f228409ac90 legacy=0x7f228409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 >> v1:172.21.15.63:6789/0 conn(0x7f22840a0280 legacy=0x7f22841398b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 monclient: found mon.noname-a 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:34.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2284141730 con 0x7f228409b780 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f227c001830 con 0x7f228409b780 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: dump: 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:34.823 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 5 monclient: authenticate success, global_id 5646 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f227c001a90 con 0x7f228409b780 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 1 -- 172.21.15.63:0/3511505277 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f227c001e60 con 0x7f228409b780 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f22827fc640 10 monclient: dump: 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:34.824 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 4 set_mon_vals no callback set 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient: shutdown 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:34.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- 172.21.15.63:0/3511505277 >> v1:172.21.15.6:6789/0 conn(0x7f228409b780 legacy=0x7f228413cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- 172.21.15.63:0/3511505277 shutdown_connections 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- 172.21.15.63:0/3511505277 >> 172.21.15.63:0/3511505277 conn(0x7f22840a5290 msgr2=0x7f22840a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- 172.21.15.63:0/3511505277 shutdown_connections 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- 172.21.15.63:0/3511505277 wait complete. 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient: build_initial_monmap 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient: monmap: 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:34.826 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:34.827 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:34.827 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:34.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 Processor -- start 2024-09-16T05:38:34.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 1 -- start start 2024-09-16T05:38:34.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.814+0000 7f2289f26640 10 monclient: init 2024-09-16T05:38:34.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:34.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient(hunting): picked mon.noname-c con 0x7f228409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient(hunting): picked mon.noname-a con 0x7f228409b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient(hunting): picked mon.noname-b con 0x7f22840a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f228411d5c0 con 0x7f228409b780 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f228411c170 con 0x7f22840a0280 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f228411c3f0 con 0x7f228409ac90 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient(hunting): authenticate will time out at 1171.596381s 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2283fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f22840a0280 legacy=0x7f22841d3fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2282ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f228409b780 0x7f22841d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35742/0 (socket says 172.21.15.63:35742) 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2282ffd640 1 -- 172.21.15.63:0/3519016799 learned_addr learned my addr 172.21.15.63:0/3519016799 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3748690110 0 0) 0x7f228411d5c0 con 0x7f228409b780 2024-09-16T05:38:34.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): my global_id is 5649 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2270003530 con 0x7f228409b780 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4113164929 0 0) 0x7f228411c3f0 con 0x7f228409ac90 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): my global_id is 5633 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f228411d5c0 con 0x7f228409ac90 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1571078558 0 0) 0x7f2270003530 con 0x7f228409b780 2024-09-16T05:38:34.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f228411c3f0 con 0x7f228409b780 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2278004090 con 0x7f228409b780 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): dump: 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 922408221 0 0) 0x7f228411d5c0 con 0x7f228409ac90 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2270003530 con 0x7f228409ac90 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2274001900 con 0x7f228409ac90 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient(hunting): dump: 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.834 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2120118856 0 0) 0x7f228411c3f0 con 0x7f228409b780 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 >> v1:172.21.15.6:6790/0 conn(0x7f228409ac90 legacy=0x7f22841cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 >> v1:172.21.15.63:6789/0 conn(0x7f22840a0280 legacy=0x7f22841d3fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 monclient: found mon.a 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:34.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228411d8a0 con 0x7f228409b780 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient: _finish_auth 0 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 5 monclient: authenticate success, global_id 5649 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient: _renew_subs 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f228406df90 con 0x7f228409b780 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient: _renew_subs 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f22841406e0 con 0x7f228409b780 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2278004380 con 0x7f228409b780 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:34.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2278004750 con 0x7f228409b780 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2281ffb640 4 set_mon_vals no callback set 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 10 monclient: dump: 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:34.837 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2264005090 con 0x7f228409b780 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2278004970 con 0x7f228409b780 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(140..140 src has 1..140) ==== 6449+0+0 (unknown 4060812276 0 0) 0x7f2278044c80 con 0x7f228409b780 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.818+0000 7f22837fe640 10 monclient: get_auth_request con 0x7f22700369d0 auth_method 0 2024-09-16T05:38:34.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.830+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f22780153c0 con 0x7f228409b780 2024-09-16T05:38:34.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.834+0000 7f226bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:34.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.834+0000 7f226bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:35.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.998+0000 7f2289f26640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T05:38:35.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.998+0000 7f2289f26640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T05:38:35.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.998+0000 7f2289f26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:35.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:34.998+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) -- 0x7f2264005b90 con 0x7f228409b780 2024-09-16T05:38:35.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.670+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' created v141) ==== 111+0+0 (unknown 2103772010 0 0) 0x7f2278014cc0 con 0x7f228409b780 2024-09-16T05:38:35.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.670+0000 7f226bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T05:38:35.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.670+0000 7f226bfff640 10 monclient: _finish_command 2 = system:0 pool 'cool' created 2024-09-16T05:38:35.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.770+0000 7f2289f26640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T05:38:35.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.770+0000 7f2289f26640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T05:38:35.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.770+0000 7f2289f26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:35.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.770+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) -- 0x7f2264005610 con 0x7f228409b780 2024-09-16T05:38:35.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f226bfff640 1 -- 172.21.15.63:0/3519016799 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' already exists v141) ==== 118+0+0 (unknown 3585635910 0 0) 0x7f2278019010 con 0x7f228409b780 2024-09-16T05:38:35.776 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f226bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T05:38:35.777 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f226bfff640 10 monclient: _finish_command 3 = system:0 pool 'cool' already exists 2024-09-16T05:38:35.777 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'cool' already exists 2024-09-16T05:38:35.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 >> v1:172.21.15.63:6812/3643207211 conn(0x7f22700369d0 legacy=0x7f2270038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:35.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f2289f26640 10 monclient: shutdown 2024-09-16T05:38:35.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f2289f26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:35.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 >> v1:172.21.15.6:6789/0 conn(0x7f228409b780 legacy=0x7f22841d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:35.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 shutdown_connections 2024-09-16T05:38:35.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 >> 172.21.15.63:0/3519016799 conn(0x7f22840a5290 msgr2=0x7f228406f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:35.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 shutdown_connections 2024-09-16T05:38:35.779 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.774+0000 7f2289f26640 1 -- 172.21.15.63:0/3519016799 wait complete. 2024-09-16T05:38:35.794 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool create cold 2 2024-09-16T05:38:35.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.914+0000 7fe244ad4640 10 monclient: get_monmap_and_config 2024-09-16T05:38:35.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.914+0000 7fe244ad4640 10 monclient: build_initial_monmap 2024-09-16T05:38:35.919 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.914+0000 7fe244ad4640 10 monclient: monmap: 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:35.919349+0000 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:35.919349+0000 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:35.920 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:35.921 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.914+0000 7fe244ad4640 1 Processor -- start 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- start start 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient: init 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient(hunting): picked mon.noname-b con 0x7fe24009b540 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient(hunting): picked mon.noname-c con 0x7fe2400a1520 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient(hunting): picked mon.noname-a con 0x7fe240138d70 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe24009f890 con 0x7fe240138d70 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe240072e80 con 0x7fe2400a1520 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe24009ce30 con 0x7fe24009b540 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient(hunting): authenticate will time out at 1172.696829s 2024-09-16T05:38:35.922 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23dd9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe2400a1520 legacy=0x7fe2400a0240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:35.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe240138d70 0x7fe24013b160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35750/0 (socket says 172.21.15.63:35750) 2024-09-16T05:38:35.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23ed9d640 1 -- 172.21.15.63:0/389597158 learned_addr learned my addr 172.21.15.63:0/389597158 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 107708092 0 0) 0x7fe24009f890 con 0x7fe240138d70 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient(hunting): my global_id is 5652 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe228003530 con 0x7fe240138d70 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539546097 0 0) 0x7fe24009ce30 con 0x7fe24009b540 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:35.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient(hunting): my global_id is 5636 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe24009f890 con 0x7fe24009b540 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266027597 0 0) 0x7fe228003530 con 0x7fe240138d70 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 >> v1:172.21.15.6:6790/0 conn(0x7fe24009b540 legacy=0x7fe24009fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 >> v1:172.21.15.63:6789/0 conn(0x7fe2400a1520 legacy=0x7fe2400a0240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 monclient: found mon.noname-a 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe24009d110 con 0x7fe240138d70 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: _finish_auth 0 2024-09-16T05:38:35.928 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2380019a0 con 0x7fe240138d70 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: dump: 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:35.929 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 5 monclient: authenticate success, global_id 5652 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe238001c00 con 0x7fe240138d70 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 1 -- 172.21.15.63:0/389597158 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe238001f50 con 0x7fe240138d70 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 4 set_mon_vals no callback set 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:35.930 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe23d59a640 10 monclient: dump: 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 10 monclient: shutdown 2024-09-16T05:38:35.931 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- 172.21.15.63:0/389597158 >> v1:172.21.15.6:6789/0 conn(0x7fe240138d70 legacy=0x7fe24013b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- 172.21.15.63:0/389597158 shutdown_connections 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- 172.21.15.63:0/389597158 >> 172.21.15.63:0/389597158 conn(0x7fe2400a5290 msgr2=0x7fe2400a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- 172.21.15.63:0/389597158 shutdown_connections 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.918+0000 7fe244ad4640 1 -- 172.21.15.63:0/389597158 wait complete. 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: build_initial_monmap 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: monmap: 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:35.932 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 1 Processor -- start 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 1 -- start start 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: init 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient(hunting): picked mon.noname-c con 0x7fe24009b540 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient(hunting): picked mon.noname-b con 0x7fe2400a1520 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient(hunting): picked mon.noname-a con 0x7fe240138d70 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:35.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2400707c0 con 0x7fe240138d70 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe240070e80 con 0x7fe2400a1520 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe240070320 con 0x7fe24009b540 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe23dd9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe2400a1520 legacy=0x7fe2401cc130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient(hunting): authenticate will time out at 1172.701604s 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe23e59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe24009b540 0x7fe24012f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:50972/0 (socket says 172.21.15.63:50972) 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe23e59c640 1 -- 172.21.15.63:0/2230215826 learned_addr learned my addr 172.21.15.63:0/2230215826 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe23ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe240138d70 0x7fe2401cf860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35764/0 (socket says 172.21.15.63:35764) 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566136957 0 0) 0x7fe240070320 con 0x7fe24009b540 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:35.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:35.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): my global_id is 5639 2024-09-16T05:38:35.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe220003530 con 0x7fe24009b540 2024-09-16T05:38:35.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936414871 0 0) 0x7fe2400707c0 con 0x7fe240138d70 2024-09-16T05:38:35.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:35.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:35.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:35.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): my global_id is 5655 2024-09-16T05:38:35.935 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe240070320 con 0x7fe240138d70 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2283001568 0 0) 0x7fe220003530 con 0x7fe24009b540 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2400707c0 con 0x7fe24009b540 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2300018c0 con 0x7fe24009b540 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): dump: 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:35.938 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 121339351 0 0) 0x7fe240070320 con 0x7fe240138d70 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe220003530 con 0x7fe240138d70 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2380018a0 con 0x7fe240138d70 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient(hunting): dump: 2024-09-16T05:38:35.939 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4212065724 0 0) 0x7fe2400707c0 con 0x7fe24009b540 2024-09-16T05:38:35.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 >> v1:172.21.15.63:6789/0 conn(0x7fe2400a1520 legacy=0x7fe2401cc130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 >> v1:172.21.15.6:6789/0 conn(0x7fe240138d70 legacy=0x7fe2401cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 monclient: found mon.c 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe240071d70 con 0x7fe24009b540 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient: _finish_auth 0 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 5 monclient: authenticate success, global_id 5639 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: _renew_subs 2024-09-16T05:38:35.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe240072050 con 0x7fe24009b540 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: _renew_subs 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe24012fd70 con 0x7fe24009b540 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe230003620 con 0x7fe24009b540 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe230003970 con 0x7fe24009b540 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe23cd99640 4 set_mon_vals no callback set 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.922+0000 7fe22effd640 10 monclient: dump: 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:35.942 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.926+0000 7fe244ad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.926+0000 7fe244ad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:35.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.926+0000 7fe244ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:35.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.926+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe224005090 con 0x7fe24009b540 2024-09-16T05:38:35.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.926+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe230003b90 con 0x7fe24009b540 2024-09-16T05:38:35.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.926+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(141..141 src has 1..141) ==== 6792+0+0 (unknown 1434019744 0 0) 0x7fe230044e40 con 0x7fe24009b540 2024-09-16T05:38:35.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.926+0000 7fe23dd9b640 10 monclient: get_auth_request con 0x7fe2200369b0 auth_method 0 2024-09-16T05:38:35.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.938+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe230019080 con 0x7fe24009b540 2024-09-16T05:38:35.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.938+0000 7fe22effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:35.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:35.938+0000 7fe22effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:36.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.106+0000 7fe244ad4640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T05:38:36.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.106+0000 7fe244ad4640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T05:38:36.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.106+0000 7fe244ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:36.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.106+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) -- 0x7fe224005b90 con 0x7fe24009b540 2024-09-16T05:38:36.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.678+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' created v142) ==== 111+0+0 (unknown 2109088656 0 0) 0x7fe23001d940 con 0x7fe24009b540 2024-09-16T05:38:36.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.678+0000 7fe22effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T05:38:36.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.678+0000 7fe22effd640 10 monclient: _finish_command 2 = system:0 pool 'cold' created 2024-09-16T05:38:36.788 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.786+0000 7fe244ad4640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T05:38:36.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.786+0000 7fe244ad4640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T05:38:36.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.786+0000 7fe244ad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:36.789 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.786+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) -- 0x7fe2240010a0 con 0x7fe24009b540 2024-09-16T05:38:36.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.786+0000 7fe22effd640 1 -- 172.21.15.63:0/2230215826 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' already exists v142) ==== 118+0+0 (unknown 2080619070 0 0) 0x7fe230041fa0 con 0x7fe24009b540 2024-09-16T05:38:36.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.786+0000 7fe22effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T05:38:36.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.786+0000 7fe22effd640 10 monclient: _finish_command 3 = system:0 pool 'cold' already exists 2024-09-16T05:38:36.791 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'cold' already exists 2024-09-16T05:38:36.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.790+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe2200369b0 legacy=0x7fe220038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:36.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.790+0000 7fe244ad4640 10 monclient: shutdown 2024-09-16T05:38:36.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.790+0000 7fe244ad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:36.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.790+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 >> v1:172.21.15.6:6790/0 conn(0x7fe24009b540 legacy=0x7fe24012f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:36.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.790+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 shutdown_connections 2024-09-16T05:38:36.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.790+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 >> 172.21.15.63:0/2230215826 conn(0x7fe2400a5290 msgr2=0x7fe240130b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:36.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.790+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 shutdown_connections 2024-09-16T05:38:36.793 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.790+0000 7fe244ad4640 1 -- 172.21.15.63:0/2230215826 wait complete. 2024-09-16T05:38:36.808 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool set cold size 2 2024-09-16T05:38:36.933 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient: get_monmap_and_config 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient: build_initial_monmap 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient: monmap: 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:36.933219+0000 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:36.933219+0000 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:36.934 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:36.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 1 Processor -- start 2024-09-16T05:38:36.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 1 -- start start 2024-09-16T05:38:36.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient: init 2024-09-16T05:38:36.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:36.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:36.937 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient(hunting): picked mon.noname-a con 0x7feb380a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient(hunting): picked mon.noname-b con 0x7feb380a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient(hunting): picked mon.noname-c con 0x7feb380a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb380a53b0 con 0x7feb380a3b10 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb380a0290 con 0x7feb380a4600 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb380a0dc0 con 0x7feb380a07a0 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 10 monclient(hunting): authenticate will time out at 1173.710480s 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb37fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7feb380a4600 legacy=0x7feb3813f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb36ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7feb380a07a0 0x7feb3813be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:50976/0 (socket says 172.21.15.63:50976) 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb36ffd640 1 -- 172.21.15.63:0/1875544708 learned_addr learned my addr 172.21.15.63:0/1875544708 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3993369732 0 0) 0x7feb380a0dc0 con 0x7feb380a07a0 2024-09-16T05:38:36.938 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient(hunting): my global_id is 5642 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb20003530 con 0x7feb380a07a0 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204515096 0 0) 0x7feb380a53b0 con 0x7feb380a3b10 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient(hunting): my global_id is 5658 2024-09-16T05:38:36.939 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb380a0dc0 con 0x7feb380a3b10 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2842727033 0 0) 0x7feb20003530 con 0x7feb380a07a0 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 >> v1:172.21.15.63:6789/0 conn(0x7feb380a4600 legacy=0x7feb3813f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 >> v1:172.21.15.6:6789/0 conn(0x7feb380a3b10 legacy=0x7feb380a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 monclient: found mon.noname-b 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb380a10a0 con 0x7feb380a07a0 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:36.940 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 5 monclient: authenticate success, global_id 5642 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feb2c001550 con 0x7feb380a07a0 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.930+0000 7feb3daad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 10 monclient: dump: 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:36.941 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb2c0017b0 con 0x7feb380a07a0 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 1 -- 172.21.15.63:0/1875544708 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feb2c001b00 con 0x7feb380a07a0 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 4 set_mon_vals no callback set 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb367fc640 10 monclient: dump: 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:36.942 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient: shutdown 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- 172.21.15.63:0/1875544708 >> v1:172.21.15.6:6790/0 conn(0x7feb380a07a0 legacy=0x7feb3813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- 172.21.15.63:0/1875544708 shutdown_connections 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- 172.21.15.63:0/1875544708 >> 172.21.15.63:0/1875544708 conn(0x7feb38075470 msgr2=0x7feb38075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:36.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- 172.21.15.63:0/1875544708 shutdown_connections 2024-09-16T05:38:36.944 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- 172.21.15.63:0/1875544708 wait complete. 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient: build_initial_monmap 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient: monmap: 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:36.948 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 Processor -- start 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- start start 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient: init 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient(hunting): picked mon.noname-c con 0x7feb380a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient(hunting): picked mon.noname-b con 0x7feb380a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient(hunting): picked mon.noname-a con 0x7feb380a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb38123890 con 0x7feb380a4600 2024-09-16T05:38:36.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb380a8970 con 0x7feb380a3b10 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb380a0500 con 0x7feb380a07a0 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb3daad640 10 monclient(hunting): authenticate will time out at 1173.714367s 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb36ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7feb380a3b10 legacy=0x7feb381d0620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb377fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7feb380a07a0 0x7feb38135ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:50992/0 (socket says 172.21.15.63:50992) 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb377fe640 1 -- 172.21.15.63:0/242050499 learned_addr learned my addr 172.21.15.63:0/242050499 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2720733183 0 0) 0x7feb380a0500 con 0x7feb380a07a0 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): my global_id is 5645 2024-09-16T05:38:36.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb24003530 con 0x7feb380a07a0 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4089317265 0 0) 0x7feb38123890 con 0x7feb380a4600 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): my global_id is 5661 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb380a0500 con 0x7feb380a4600 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1967350682 0 0) 0x7feb24003530 con 0x7feb380a07a0 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb38123890 con 0x7feb380a07a0 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feb28001980 con 0x7feb380a07a0 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:36.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): dump: 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4074462850 0 0) 0x7feb380a0500 con 0x7feb380a4600 2024-09-16T05:38:36.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb24003530 con 0x7feb380a4600 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feb30004340 con 0x7feb380a4600 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient(hunting): dump: 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:36.953 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 913284939 0 0) 0x7feb38123890 con 0x7feb380a07a0 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 >> v1:172.21.15.63:6789/0 conn(0x7feb380a3b10 legacy=0x7feb381d0620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 >> v1:172.21.15.6:6789/0 conn(0x7feb380a4600 legacy=0x7feb381d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 monclient: found mon.c 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb381232d0 con 0x7feb380a07a0 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient: _finish_auth 0 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:36.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.934+0000 7feb1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 5 monclient: authenticate success, global_id 5645 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 10 monclient: _renew_subs 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb381235b0 con 0x7feb380a07a0 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 10 monclient: _renew_subs 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feb381cb500 con 0x7feb380a07a0 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb28001c60 con 0x7feb380a07a0 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb1ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feb28003a70 con 0x7feb380a07a0 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:36.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb35ffb640 4 set_mon_vals no callback set 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb1ffff640 10 monclient: dump: 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:36.956 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:36.957 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:36.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:36.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:36.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:36.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb18005090 con 0x7feb380a07a0 2024-09-16T05:38:36.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7feb28003c10 con 0x7feb380a07a0 2024-09-16T05:38:36.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(142..142 src has 1..142) ==== 7135+0+0 (unknown 3774411432 0 0) 0x7feb28044e70 con 0x7feb380a07a0 2024-09-16T05:38:36.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.938+0000 7feb36ffd640 10 monclient: get_auth_request con 0x7feb24036a20 auth_method 0 2024-09-16T05:38:36.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.950+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7feb28018f60 con 0x7feb380a07a0 2024-09-16T05:38:36.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.950+0000 7feb1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:36.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:36.950+0000 7feb1ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:36.968 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:38:36.968 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:38:36.969 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:38:36.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:38:36.971 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:38:36.972 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:38:36.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:38:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.097+0000 7fbcd6d57640 10 monclient: get_monmap_and_config 2024-09-16T05:38:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.097+0000 7fbcd6d57640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.097+0000 7fbcd6d57640 10 monclient: monmap: 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:37.101866+0000 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:37.101866+0000 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 1 Processor -- start 2024-09-16T05:38:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 1 -- start start 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 10 monclient: init 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd00a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd00a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd00a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd006e520 con 0x7fbcd00a1a00 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd00a0780 con 0x7fbcd00a37a0 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd00a4c70 con 0x7fbcd00a4290 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd6d57640 10 monclient(hunting): authenticate will time out at 1178.909119s 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd52f4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbcd00a1a00 0x7fbcd013f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51228/0 (socket says 172.21.15.6:51228) 2024-09-16T05:38:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.101+0000 7fbcd52f4640 1 -- 172.21.15.6:0/844150414 learned_addr learned my addr 172.21.15.6:0/844150414 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578839310 0 0) 0x7fbcd00a4c70 con 0x7fbcd00a4290 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient(hunting): my global_id is 5648 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcc0003530 con 0x7fbcd00a4290 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 904832012 0 0) 0x7fbcc0003530 con 0x7fbcd00a4290 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 >> v1:172.21.15.63:6789/0 conn(0x7fbcd00a37a0 legacy=0x7fbcd00a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 >> v1:172.21.15.6:6789/0 conn(0x7fbcd00a1a00 legacy=0x7fbcd013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 monclient: found mon.noname-b 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd00a65f0 con 0x7fbcd00a4290 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbcbc001620 con 0x7fbcd00a4290 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: dump: 2024-09-16T05:38:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbcd6d57640 5 monclient: authenticate success, global_id 5648 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbcd6d57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbcbc001880 con 0x7fbcd00a4290 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 1 -- 172.21.15.6:0/844150414 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbcbc001c50 con 0x7fbcd00a4290 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbcd6d57640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbcd6d57640 4 set_mon_vals no callback set 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.121+0000 7fbccf7fe640 10 monclient: dump: 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient: shutdown 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- 172.21.15.6:0/844150414 >> v1:172.21.15.6:6790/0 conn(0x7fbcd00a4290 legacy=0x7fbcd013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- 172.21.15.6:0/844150414 shutdown_connections 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- 172.21.15.6:0/844150414 >> 172.21.15.6:0/844150414 conn(0x7fbcd00a58a0 msgr2=0x7fbcd00a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- 172.21.15.6:0/844150414 shutdown_connections 2024-09-16T05:38:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- 172.21.15.6:0/844150414 wait complete. 2024-09-16T05:38:37.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.122+0000 7feb3daad640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T05:38:37.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.122+0000 7feb3daad640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T05:38:37.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.122+0000 7feb3daad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.122+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) -- 0x7feb18005b90 con 0x7feb380a07a0 2024-09-16T05:38:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient: monmap: 2024-09-16T05:38:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:37.137 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 Processor -- start 2024-09-16T05:38:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- start start 2024-09-16T05:38:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient: init 2024-09-16T05:38:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd00a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd00a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd00a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd013dc20 con 0x7fbcd00a4290 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd013fa50 con 0x7fbcd00a37a0 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd013dea0 con 0x7fbcd00a1a00 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd6d57640 10 monclient(hunting): authenticate will time out at 1178.935066s 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd52f4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbcd00a4290 0x7fbcd01d4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51244/0 (socket says 172.21.15.6:51244) 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbcd52f4640 1 -- 172.21.15.6:0/86608114 learned_addr learned my addr 172.21.15.6:0/86608114 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944644087 0 0) 0x7fbcd013dc20 con 0x7fbcd00a4290 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 10 monclient(hunting): my global_id is 5664 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb4003560 con 0x7fbcd00a4290 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1039886191 0 0) 0x7fbcb4003560 con 0x7fbcd00a4290 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcd013dc20 con 0x7fbcd00a4290 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbcc8001bc0 con 0x7fbcd00a4290 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 10 monclient(hunting): dump: 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.125+0000 7fbccd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 941595952 0 0) 0x7fbcd013dc20 con 0x7fbcd00a4290 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 >> v1:172.21.15.6:6790/0 conn(0x7fbcd00a1a00 legacy=0x7fbcd0135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 >> v1:172.21.15.63:6789/0 conn(0x7fbcd00a37a0 legacy=0x7fbcd01d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 monclient: found mon.a 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd013f630 con 0x7fbcd00a4290 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 5 monclient: authenticate success, global_id 5664 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 10 monclient: _renew_subs 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd00a82d0 con 0x7fbcd00a4290 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 10 monclient: _renew_subs 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbcd013f350 con 0x7fbcd00a4290 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbcc8004b70 con 0x7fbcd00a4290 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbcc8004f40 con 0x7fbcd00a4290 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcceffd640 4 set_mon_vals no callback set 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 10 monclient: dump: 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbcc80050e0 con 0x7fbcd00a4290 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcb0005090 con 0x7fbcd00a4290 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(142..142 src has 1..142) ==== 7135+0+0 (unknown 3774411432 0 0) 0x7fbcc8045270 con 0x7fbcd00a4290 2024-09-16T05:38:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.129+0000 7fbcd4af3640 10 monclient: get_auth_request con 0x7fbcb4036b10 auth_method 0 2024-09-16T05:38:37.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.133+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbcc8015770 con 0x7fbcd00a4290 2024-09-16T05:38:37.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.133+0000 7fbccd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.133+0000 7fbccd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.361+0000 7fbcd6d57640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:38:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.361+0000 7fbcd6d57640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.361+0000 7fbcd6d57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.361+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbcb0005380 con 0x7fbcd00a4290 2024-09-16T05:38:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.361+0000 7fbccd7fa640 1 -- 172.21.15.6:0/86608114 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1619844805) 0x7fbcc8015070 con 0x7fbcd00a4290 2024-09-16T05:38:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.361+0000 7fbccd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.361+0000 7fbccd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:37.364 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:38:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.365+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbcb4036b10 legacy=0x7fbcb4038fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.365+0000 7fbcd6d57640 10 monclient: shutdown 2024-09-16T05:38:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.365+0000 7fbcd6d57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.365+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 >> v1:172.21.15.6:6789/0 conn(0x7fbcd00a4290 legacy=0x7fbcd01d4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.365+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 shutdown_connections 2024-09-16T05:38:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.365+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 >> 172.21.15.6:0/86608114 conn(0x7fbcd00a58a0 msgr2=0x7fbcd009d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.365+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 shutdown_connections 2024-09-16T05:38:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.365+0000 7fbcd6d57640 1 -- 172.21.15.6:0/86608114 wait complete. 2024-09-16T05:38:37.383 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:38:37.383 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:38:37.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:38:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient: get_monmap_and_config 2024-09-16T05:38:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient: monmap: 2024-09-16T05:38:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:37.507432+0000 2024-09-16T05:38:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:37.507432+0000 2024-09-16T05:38:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:37.508 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 1 Processor -- start 2024-09-16T05:38:37.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 1 -- start start 2024-09-16T05:38:37.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient: init 2024-09-16T05:38:37.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient(hunting): picked mon.noname-a con 0x7f870c09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient(hunting): picked mon.noname-b con 0x7f870c09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient(hunting): picked mon.noname-c con 0x7f870c0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870c0a1370 con 0x7f870c09aa30 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870c09ead0 con 0x7f870c0a0da0 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870c09edb0 con 0x7f870c09e230 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient(hunting): authenticate will time out at 1179.314033s 2024-09-16T05:38:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8712101640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f870c09aa30 0x7f870c09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51260/0 (socket says 172.21.15.6:51260) 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8712101640 1 -- 172.21.15.6:0/1978812552 learned_addr learned my addr 172.21.15.6:0/1978812552 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2627027399 0 0) 0x7f870c09edb0 con 0x7f870c09e230 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient(hunting): my global_id is 5651 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f4003530 con 0x7f870c09e230 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1648153978 0 0) 0x7f870c0a1370 con 0x7f870c09aa30 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient(hunting): my global_id is 5667 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f870c09edb0 con 0x7f870c09aa30 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4074099068 0 0) 0x7f870c09edb0 con 0x7f870c09aa30 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 >> v1:172.21.15.6:6790/0 conn(0x7f870c09e230 legacy=0x7f870c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 >> v1:172.21.15.63:6789/0 conn(0x7f870c0a0da0 legacy=0x7f870c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 monclient: found mon.noname-a 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c09dd50 con 0x7f870c09aa30 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 5 monclient: authenticate success, global_id 5667 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86fc0019a0 con 0x7f870c09aa30 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: dump: 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86fc001b90 con 0x7f870c09aa30 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 1 -- 172.21.15.6:0/1978812552 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86fc001f60 con 0x7f870c09aa30 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f87110ff640 10 monclient: dump: 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.517 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.517 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 4 set_mon_vals no callback set 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 10 monclient: shutdown 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.505+0000 7f8714365640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- 172.21.15.6:0/1978812552 >> v1:172.21.15.6:6789/0 conn(0x7f870c09aa30 legacy=0x7f870c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- 172.21.15.6:0/1978812552 shutdown_connections 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- 172.21.15.6:0/1978812552 >> 172.21.15.6:0/1978812552 conn(0x7f870c0a1650 msgr2=0x7f870c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- 172.21.15.6:0/1978812552 shutdown_connections 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- 172.21.15.6:0/1978812552 wait complete. 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: monmap: 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:37.523 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 Processor -- start 2024-09-16T05:38:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- start start 2024-09-16T05:38:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: init 2024-09-16T05:38:37.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient(hunting): picked mon.noname-a con 0x7f870c09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient(hunting): picked mon.noname-c con 0x7f870c09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient(hunting): picked mon.noname-b con 0x7f870c0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870c11ea50 con 0x7f870c09aa30 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870c11ed30 con 0x7f870c0a0da0 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f870c09bb40 con 0x7f870c09e230 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient(hunting): authenticate will time out at 1179.317614s 2024-09-16T05:38:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8712101640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f870c09aa30 0x7f870c1cb9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51264/0 (socket says 172.21.15.6:51264) 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8711900640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f870c09e230 0x7f870c1d09a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41580/0 (socket says 172.21.15.6:41580) 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8712101640 1 -- 172.21.15.6:0/2875612541 learned_addr learned my addr 172.21.15.6:0/2875612541 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3562838391 0 0) 0x7f870c11ea50 con 0x7f870c09aa30 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient(hunting): my global_id is 5670 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f8003530 con 0x7f870c09aa30 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 635138967 0 0) 0x7f86f8003530 con 0x7f870c09aa30 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f870c11ea50 con 0x7f870c09aa30 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86fc001fa0 con 0x7f870c09aa30 2024-09-16T05:38:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient(hunting): dump: 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4088463559 0 0) 0x7f870c11ea50 con 0x7f870c09aa30 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 >> v1:172.21.15.6:6790/0 conn(0x7f870c09e230 legacy=0x7f870c1d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 >> v1:172.21.15.63:6789/0 conn(0x7f870c0a0da0 legacy=0x7f870c1d40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 monclient: found mon.a 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c11e770 con 0x7f870c09aa30 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 5 monclient: authenticate success, global_id 5670 2024-09-16T05:38:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: _renew_subs 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f870c11d2b0 con 0x7f870c09aa30 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: _renew_subs 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f870c11c380 con 0x7f870c09aa30 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86fc001b90 con 0x7f870c09aa30 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f86fc004eb0 con 0x7f870c09aa30 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 10 monclient: dump: 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f87108fe640 4 set_mon_vals no callback set 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f86fc005050 con 0x7f870c09aa30 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.509+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86ec0065d0 con 0x7f86ec002870 2024-09-16T05:38:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(142..142 src has 1..142) ==== 7135+0+0 (unknown 3774411432 0 0) 0x7f86fc045290 con 0x7f870c09aa30 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2788515119 0 0) 0x7f86ec0065d0 con 0x7f86ec002870 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient(hunting): my global_id is 5673 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f870c11ea50 con 0x7f86ec002870 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f8712902640 10 monclient: get_auth_request con 0x7f86f8036ab0 auth_method 0 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 538503953 0 0) 0x7f870c11ea50 con 0x7f86ec002870 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86ec0065d0 con 0x7f86ec002870 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8700001190 con 0x7f86ec002870 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient: dump: 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2135895126 0 0) 0x7f86ec0065d0 con 0x7f86ec002870 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86ec00a160 con 0x7f86ec002870 2024-09-16T05:38:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f87000013f0 con 0x7f86ec002870 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 >> v1:172.21.15.6:6789/0 conn(0x7f86ec002870 legacy=0x7f86ec004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.513+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 >> v1:172.21.15.6:6789/0 conn(0x7f86ec002870 legacy=0x7f86ec004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f8714365640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86ec009e50 con 0x7f86ec009110 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2575375814 0 0) 0x7f86ec009e50 con 0x7f86ec009110 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 10 monclient(hunting): my global_id is 5676 2024-09-16T05:38:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86ec0065d0 con 0x7f86ec009110 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1711091656 0 0) 0x7f86ec0065d0 con 0x7f86ec009110 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86ec009e50 con 0x7f86ec009110 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8700003600 con 0x7f86ec009110 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 10 monclient: dump: 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3527509309 0 0) 0x7f86ec009e50 con 0x7f86ec009110 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.517+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86ec00c3e0 con 0x7f86ec009110 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 904442075) 0x7f8700005790 con 0x7f86ec009110 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f870affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f870affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 >> v1:172.21.15.6:6789/0 conn(0x7f86ec009110 legacy=0x7f86ec0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f870affd640 1 -- 172.21.15.6:0/2875612541 >> v1:172.21.15.6:6789/0 conn(0x7f86ec009110 legacy=0x7f86ec0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:37.538 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 20, 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 246577, 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:38:37.539 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.540 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.541 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:38:37.542 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:38:37.543 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:38:37.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.546 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:37.547 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.548 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:38:37.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.550 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:38:37.551 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 >> v1:172.21.15.63:6812/3643207211 conn(0x7f86f8036ab0 legacy=0x7f86f8038f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8714365640 10 monclient: shutdown 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8714365640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 >> v1:172.21.15.6:6789/0 conn(0x7f870c09aa30 legacy=0x7f870c1cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8712902640 1 -- 172.21.15.6:0/2875612541 reap_dead start 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8712902640 1 -- 172.21.15.6:0/2875612541 reap_dead start 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 shutdown_connections 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 >> 172.21.15.6:0/2875612541 conn(0x7f870c0a1650 msgr2=0x7f870c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 shutdown_connections 2024-09-16T05:38:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.521+0000 7f8714365640 1 -- 172.21.15.6:0/2875612541 wait complete. 2024-09-16T05:38:37.553 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:38:37.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:38:37.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:38:37.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.661+0000 7fb1ab474640 10 monclient: get_monmap_and_config 2024-09-16T05:38:37.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.661+0000 7fb1ab474640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.661+0000 7fb1ab474640 10 monclient: monmap: 2024-09-16T05:38:37.666 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.666 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.666 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:37.666692+0000 2024-09-16T05:38:37.666 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:37.666692+0000 2024-09-16T05:38:37.667 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.667 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.667 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.667 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:37.667 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:37.667 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 Processor -- start 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- start start 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient: init 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a406dd90 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a40a2c00 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a413cfe0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a40a4c20 con 0x7fb1a413cfe0 2024-09-16T05:38:37.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a40a49a0 con 0x7fb1a406dd90 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a40a0720 con 0x7fb1a40a2c00 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient(hunting): authenticate will time out at 1179.473157s 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1a8a0f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb1a40a2c00 0x7fb1a413b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41584/0 (socket says 172.21.15.6:41584) 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1a8a0f640 1 -- 172.21.15.6:0/996732933 learned_addr learned my addr 172.21.15.6:0/996732933 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942061400 0 0) 0x7fb1a40a0720 con 0x7fb1a40a2c00 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient(hunting): my global_id is 5654 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb190003530 con 0x7fb1a40a2c00 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031975282 0 0) 0x7fb1a40a4c20 con 0x7fb1a413cfe0 2024-09-16T05:38:37.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient(hunting): my global_id is 5679 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1a40a0720 con 0x7fb1a413cfe0 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1365595896 0 0) 0x7fb190003530 con 0x7fb1a40a2c00 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 >> v1:172.21.15.63:6789/0 conn(0x7fb1a406dd90 legacy=0x7fb1a40a24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 >> v1:172.21.15.6:6789/0 conn(0x7fb1a413cfe0 legacy=0x7fb1a413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 monclient: found mon.noname-b 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a4136a80 con 0x7fb1a40a2c00 2024-09-16T05:38:37.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1a0001620 con 0x7fb1a40a2c00 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 5 monclient: authenticate success, global_id 5654 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: dump: 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.677 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1a0001880 con 0x7fb1a40a2c00 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 1 -- 172.21.15.6:0/996732933 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1a0001c90 con 0x7fb1a40a2c00 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb197fff640 10 monclient: dump: 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.678 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:37.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 4 set_mon_vals no callback set 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient: shutdown 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- 172.21.15.6:0/996732933 >> v1:172.21.15.6:6790/0 conn(0x7fb1a40a2c00 legacy=0x7fb1a413b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- 172.21.15.6:0/996732933 shutdown_connections 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- 172.21.15.6:0/996732933 >> 172.21.15.6:0/996732933 conn(0x7fb1a40a5800 msgr2=0x7fb1a40a5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- 172.21.15.6:0/996732933 shutdown_connections 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- 172.21.15.6:0/996732933 wait complete. 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient: monmap: 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:37.680 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 Processor -- start 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 1 -- start start 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.665+0000 7fb1ab474640 10 monclient: init 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a406dd90 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a40a2c00 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a413cfe0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a40a6b00 con 0x7fb1a413cfe0 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a4138450 con 0x7fb1a406dd90 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a4139200 con 0x7fb1a40a2c00 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient(hunting): authenticate will time out at 1179.476446s 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1a8a0f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb1a40a2c00 0x7fb1a4137d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41586/0 (socket says 172.21.15.6:41586) 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1a8a0f640 1 -- 172.21.15.6:0/1048621785 learned_addr learned my addr 172.21.15.6:0/1048621785 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3260111189 0 0) 0x7fb1a4139200 con 0x7fb1a40a2c00 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): my global_id is 5657 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb188003530 con 0x7fb1a40a2c00 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1929124774 0 0) 0x7fb19c001560 con 0x7fb1a413cfe0 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): my global_id is 5682 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1a4139200 con 0x7fb1a413cfe0 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2445743375 0 0) 0x7fb1a4139200 con 0x7fb1a413cfe0 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb19c001560 con 0x7fb1a413cfe0 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2223760400 0 0) 0x7fb188003530 con 0x7fb1a40a2c00 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1a4139200 con 0x7fb1a40a2c00 2024-09-16T05:38:37.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb19c001de0 con 0x7fb1a413cfe0 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.684 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1a0001bd0 con 0x7fb1a40a2c00 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.685 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2397840540 0 0) 0x7fb1a4138450 con 0x7fb1a406dd90 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): my global_id is 104101 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb188003530 con 0x7fb1a406dd90 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2628155977 0 0) 0x7fb1a4139200 con 0x7fb1a40a2c00 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 >> v1:172.21.15.63:6789/0 conn(0x7fb1a406dd90 legacy=0x7fb1a4137630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 >> v1:172.21.15.6:6789/0 conn(0x7fb1a413cfe0 legacy=0x7fb1a41d3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 monclient: found mon.c 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a41398c0 con 0x7fb1a40a2c00 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 5 monclient: authenticate success, global_id 5657 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient: _renew_subs 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a41d49a0 con 0x7fb1a40a2c00 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient: _renew_subs 2024-09-16T05:38:37.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1a41d4d40 con 0x7fb1a40a2c00 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1a00011d0 con 0x7fb1a40a2c00 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1a0004da0 con 0x7fb1a40a2c00 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: dump: 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.688 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1977fe640 4 set_mon_vals no callback set 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb1840065d0 con 0x7fb184002870 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb1a000fb40 con 0x7fb1a40a2c00 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(142..142 src has 1..142) ==== 7135+0+0 (unknown 3774411432 0 0) 0x7fb1a0044f10 con 0x7fb1a40a2c00 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399971489 0 0) 0x7fb1840065d0 con 0x7fb184002870 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient(hunting): my global_id is 5660 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1a4139200 con 0x7fb184002870 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3564524172 0 0) 0x7fb1a4139200 con 0x7fb184002870 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1840065d0 con 0x7fb184002870 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb198001700 con 0x7fb184002870 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: dump: 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.690 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb1a9a11640 10 monclient: get_auth_request con 0x7fb188036b60 auth_method 0 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1028657623 0 0) 0x7fb1840065d0 con 0x7fb184002870 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb18400a160 con 0x7fb184002870 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb198001af0 con 0x7fb184002870 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 >> v1:172.21.15.6:6790/0 conn(0x7fb184002870 legacy=0x7fb184004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.669+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 >> v1:172.21.15.6:6790/0 conn(0x7fb184002870 legacy=0x7fb184004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:37.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:38:37.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb184009e50 con 0x7fb184009110 2024-09-16T05:38:37.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 385486470 0 0) 0x7fb184009e50 con 0x7fb184009110 2024-09-16T05:38:37.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient(hunting): my global_id is 5663 2024-09-16T05:38:37.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1840065d0 con 0x7fb184009110 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3001585038 0 0) 0x7fb1840065d0 con 0x7fb184009110 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb184009e50 con 0x7fb184009110 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1980037a0 con 0x7fb184009110 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient: dump: 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.694 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1693215039 0 0) 0x7fb184009e50 con 0x7fb184009110 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb18400c3e0 con 0x7fb184009110 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 743946263) 0x7fb198005aa0 con 0x7fb184009110 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 >> v1:172.21.15.6:6790/0 conn(0x7fb184009110 legacy=0x7fb1840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb195ffb640 1 -- 172.21.15.6:0/1048621785 >> v1:172.21.15.6:6790/0 conn(0x7fb184009110 legacy=0x7fb1840095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:37.695 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 20, 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 246733, 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.696 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.697 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.698 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:37.699 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.700 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:37.701 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.702 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.703 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.704 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.706 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:38:37.707 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb188036b60 legacy=0x7fb188039020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 10 monclient: shutdown 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 >> v1:172.21.15.6:6790/0 conn(0x7fb1a40a2c00 legacy=0x7fb1a4137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1a9a11640 1 -- 172.21.15.6:0/1048621785 reap_dead start 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1a9a11640 1 -- 172.21.15.6:0/1048621785 reap_dead start 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 shutdown_connections 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 >> 172.21.15.6:0/1048621785 conn(0x7fb1a40a5800 msgr2=0x7fb1a40a8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 shutdown_connections 2024-09-16T05:38:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.677+0000 7fb1ab474640 1 -- 172.21.15.6:0/1048621785 wait complete. 2024-09-16T05:38:37.710 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:38:37.710 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:38:37.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:38:37.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.690+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v143) ==== 124+0+0 (unknown 1907979827 0 0) 0x7feb2801d820 con 0x7feb380a07a0 2024-09-16T05:38:37.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.690+0000 7feb1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T05:38:37.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.690+0000 7feb1ffff640 10 monclient: _finish_command 2 = system:0 set pool 12 size to 2 2024-09-16T05:38:37.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.790+0000 7feb3daad640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T05:38:37.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.790+0000 7feb3daad640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T05:38:37.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.790+0000 7feb3daad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.796 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:37.790+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) -- 0x7feb18005610 con 0x7feb380a07a0 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient: get_monmap_and_config 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient: monmap: 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:37.823615+0000 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:37.823615+0000 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.823 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.824 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.824 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:37.824 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:37.824 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 Processor -- start 2024-09-16T05:38:37.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- start start 2024-09-16T05:38:37.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient: init 2024-09-16T05:38:37.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:37.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient(hunting): picked mon.noname-b con 0x7f12340a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient(hunting): picked mon.noname-c con 0x7f12340a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient(hunting): picked mon.noname-a con 0x7f12340a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12340a33a0 con 0x7f12340a2980 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12340a07a0 con 0x7f12340a2570 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12340a2d90 con 0x7f12340a2190 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient(hunting): authenticate will time out at 1179.630170s 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1239a4b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f12340a2190 0x7f12340a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41596/0 (socket says 172.21.15.6:41596) 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1239a4b640 1 -- 172.21.15.6:0/41396985 learned_addr learned my addr 172.21.15.6:0/41396985 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2015207208 0 0) 0x7f12340a2d90 con 0x7f12340a2190 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient(hunting): my global_id is 5666 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1220003530 con 0x7f12340a2190 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1971783030 0 0) 0x7f12340a33a0 con 0x7f12340a2980 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient(hunting): my global_id is 5685 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12340a2d90 con 0x7f12340a2980 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3799048077 0 0) 0x7f1220003530 con 0x7f12340a2190 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 >> v1:172.21.15.63:6789/0 conn(0x7f12340a2570 legacy=0x7f123413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 >> v1:172.21.15.6:6789/0 conn(0x7f12340a2980 legacy=0x7f123413f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 monclient: found mon.noname-b 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12340a66e0 con 0x7f12340a2190 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1230001620 con 0x7f12340a2190 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: dump: 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.829 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 5 monclient: authenticate success, global_id 5666 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:37.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1230001940 con 0x7f12340a2190 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 1 -- 172.21.15.6:0/41396985 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1230001d10 con 0x7f12340a2190 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 4 set_mon_vals no callback set 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f1238a49640 10 monclient: dump: 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.831 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 10 monclient: shutdown 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- 172.21.15.6:0/41396985 >> v1:172.21.15.6:6790/0 conn(0x7f12340a2190 legacy=0x7f12340a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- 172.21.15.6:0/41396985 shutdown_connections 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- 172.21.15.6:0/41396985 >> 172.21.15.6:0/41396985 conn(0x7f12340a9850 msgr2=0x7f12340a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- 172.21.15.6:0/41396985 shutdown_connections 2024-09-16T05:38:37.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.821+0000 7f123bcaf640 1 -- 172.21.15.6:0/41396985 wait complete. 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: monmap: 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:37.837 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 1 Processor -- start 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 1 -- start start 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: init 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient(hunting): picked mon.noname-b con 0x7f12340a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient(hunting): picked mon.noname-c con 0x7f12340a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient(hunting): picked mon.noname-a con 0x7f12340a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12341243c0 con 0x7f12340a2980 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12341246a0 con 0x7f12340a2190 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12341231f0 con 0x7f12340a2570 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient(hunting): authenticate will time out at 1179.633218s 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123924a640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f12340a2570 0x7f12341d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41606/0 (socket says 172.21.15.6:41606) 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123924a640 1 -- 172.21.15.6:0/3969323511 learned_addr learned my addr 172.21.15.6:0/3969323511 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123a24c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12340a2980 0x7f12341d44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51312/0 (socket says 172.21.15.6:51312) 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2044180513 0 0) 0x7f12341231f0 con 0x7f12340a2570 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): my global_id is 5669 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1218003530 con 0x7f12340a2570 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3564741021 0 0) 0x7f12341243c0 con 0x7f12340a2980 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): my global_id is 5688 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12341231f0 con 0x7f12340a2980 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2734769253 0 0) 0x7f1218003530 con 0x7f12340a2570 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12341243c0 con 0x7f12340a2570 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1224001920 con 0x7f12340a2570 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.840 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.841 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1511593698 0 0) 0x7f12341231f0 con 0x7f12340a2980 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1218003530 con 0x7f12340a2980 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f122c001bc0 con 0x7f12340a2980 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.842 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1281110787 0 0) 0x7f12341246a0 con 0x7f12340a2190 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): my global_id is 104107 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12341231f0 con 0x7f12340a2190 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2745069452 0 0) 0x7f12341243c0 con 0x7f12340a2570 2024-09-16T05:38:37.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 >> v1:172.21.15.63:6789/0 conn(0x7f12340a2190 legacy=0x7f12341cbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 >> v1:172.21.15.6:6789/0 conn(0x7f12340a2980 legacy=0x7f12341d44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 monclient: found mon.c 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123412d6d0 con 0x7f12340a2570 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 5 monclient: authenticate success, global_id 5669 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: _renew_subs 2024-09-16T05:38:37.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1234124980 con 0x7f12340a2570 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: _renew_subs 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1234121a50 con 0x7f12340a2570 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1224001cf0 con 0x7f12340a2570 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1224003af0 con 0x7f12340a2570 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient: dump: 2024-09-16T05:38:37.845 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122bfff640 4 set_mon_vals no callback set 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1224003c90 con 0x7f12340a2570 2024-09-16T05:38:37.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f121c0065d0 con 0x7f121c002870 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(143..143 src has 1..143) ==== 7135+0+0 (unknown 1594094581 0 0) 0x7f1224044fe0 con 0x7f12340a2570 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f123a24c640 10 monclient: get_auth_request con 0x7f1218036b80 auth_method 0 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187455197 0 0) 0x7f121c0065d0 con 0x7f121c002870 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 10 monclient(hunting): my global_id is 104110 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.825+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12341243c0 con 0x7f121c002870 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3426150105 0 0) 0x7f12341243c0 con 0x7f121c002870 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f121c0065d0 con 0x7f121c002870 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12300015e0 con 0x7f121c002870 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 10 monclient: dump: 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 517617835 0 0) 0x7f121c0065d0 con 0x7f121c002870 2024-09-16T05:38:37.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f121c00a160 con 0x7f121c002870 2024-09-16T05:38:37.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f12300019b0 con 0x7f121c002870 2024-09-16T05:38:37.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:37.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:37.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 >> v1:172.21.15.63:6789/0 conn(0x7f121c002870 legacy=0x7f121c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.829+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 >> v1:172.21.15.63:6789/0 conn(0x7f121c002870 legacy=0x7f121c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.833+0000 7f123bcaf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:38:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.833+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f121c009e50 con 0x7f121c009110 2024-09-16T05:38:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757345863 0 0) 0x7f121c009e50 con 0x7f121c009110 2024-09-16T05:38:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient(hunting): my global_id is 104113 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f121c0065d0 con 0x7f121c009110 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1818771506 0 0) 0x7f121c0065d0 con 0x7f121c009110 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f121c009e50 con 0x7f121c009110 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1230004b70 con 0x7f121c009110 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient: dump: 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 813559120 0 0) 0x7f121c009e50 con 0x7f121c009110 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f121c00c3e0 con 0x7f121c009110 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3959 (unknown 0 0 189746552) 0x7f1230004b00 con 0x7f121c009110 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:38:37.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 >> v1:172.21.15.63:6789/0 conn(0x7f121c009110 legacy=0x7f121c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f122a7fc640 1 -- 172.21.15.6:0/3969323511 >> v1:172.21.15.63:6789/0 conn(0x7f121c009110 legacy=0x7f121c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 20, 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 799, 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:38:37.853 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.854 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.855 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:38:37.856 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:37.857 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:38:37.862 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:37.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:38:37.864 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:38:37.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1218036b80 legacy=0x7f1218039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123bcaf640 10 monclient: shutdown 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123bcaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 >> v1:172.21.15.6:6790/0 conn(0x7f12340a2570 legacy=0x7f12341d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123a24c640 1 -- 172.21.15.6:0/3969323511 reap_dead start 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123a24c640 1 -- 172.21.15.6:0/3969323511 reap_dead start 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 shutdown_connections 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 >> 172.21.15.6:0/3969323511 conn(0x7f12340a9850 msgr2=0x7f123406ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 shutdown_connections 2024-09-16T05:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.837+0000 7f123bcaf640 1 -- 172.21.15.6:0/3969323511 wait complete. 2024-09-16T05:38:37.866 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:38:37.867 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:38:37.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:38:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.977+0000 7f6a1dd0e640 10 monclient: get_monmap_and_config 2024-09-16T05:38:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.977+0000 7f6a1dd0e640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.977+0000 7f6a1dd0e640 10 monclient: monmap: 2024-09-16T05:38:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:37.981417+0000 2024-09-16T05:38:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:37.981417+0000 2024-09-16T05:38:37.982 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.982 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.982 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.982 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:37.982 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:37.982 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.977+0000 7f6a1dd0e640 1 Processor -- start 2024-09-16T05:38:37.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.977+0000 7f6a1dd0e640 1 -- start start 2024-09-16T05:38:37.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.977+0000 7f6a1dd0e640 10 monclient: init 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): picked mon.noname-a con 0x7f6a180a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): picked mon.noname-b con 0x7f6a180a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): picked mon.noname-c con 0x7f6a180a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a1806e520 con 0x7f6a180a37a0 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a180a0780 con 0x7f6a180a1a00 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a180a4c70 con 0x7f6a180a4290 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): authenticate will time out at 1179.788723s 2024-09-16T05:38:37.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a16ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6a180a4290 0x7f6a1813bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41608/0 (socket says 172.21.15.6:41608) 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a16ffd640 1 -- 172.21.15.6:0/4207941526 learned_addr learned my addr 172.21.15.6:0/4207941526 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237714685 0 0) 0x7f6a1806e520 con 0x7f6a180a37a0 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient(hunting): my global_id is 5691 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a00003530 con 0x7f6a180a37a0 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945214586 0 0) 0x7f6a180a4c70 con 0x7f6a180a4290 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient(hunting): my global_id is 5672 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a1806e520 con 0x7f6a180a4290 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1877393840 0 0) 0x7f6a00003530 con 0x7f6a180a37a0 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 >> v1:172.21.15.6:6790/0 conn(0x7f6a180a4290 legacy=0x7f6a1813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 >> v1:172.21.15.63:6789/0 conn(0x7f6a180a1a00 legacy=0x7f6a1813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 monclient: found mon.noname-a 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a180a65f0 con 0x7f6a180a37a0 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a080019a0 con 0x7f6a180a37a0 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: dump: 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.989 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 5 monclient: authenticate success, global_id 5691 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a08001c00 con 0x7f6a180a37a0 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 1 -- 172.21.15.6:0/4207941526 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a08001fd0 con 0x7f6a180a37a0 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 4 set_mon_vals no callback set 2024-09-16T05:38:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a167fc640 10 monclient: dump: 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: shutdown 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/4207941526 >> v1:172.21.15.6:6789/0 conn(0x7f6a180a37a0 legacy=0x7f6a180a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/4207941526 shutdown_connections 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/4207941526 >> 172.21.15.6:0/4207941526 conn(0x7f6a180a58a0 msgr2=0x7f6a180a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/4207941526 shutdown_connections 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/4207941526 wait complete. 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: build_initial_monmap 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: monmap: 2024-09-16T05:38:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 Processor -- start 2024-09-16T05:38:37.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- start start 2024-09-16T05:38:37.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: init 2024-09-16T05:38:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): picked mon.noname-b con 0x7f6a180a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): picked mon.noname-a con 0x7f6a180a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): picked mon.noname-c con 0x7f6a180a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a1813ef20 con 0x7f6a180a37a0 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a1813c090 con 0x7f6a180a1a00 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a180a8010 con 0x7f6a180a4290 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a1dd0e640 10 monclient(hunting): authenticate will time out at 1179.791852s 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a16ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a180a37a0 0x7f6a1813b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51326/0 (socket says 172.21.15.6:51326) 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.981+0000 7f6a16ffd640 1 -- 172.21.15.6:0/3239840941 learned_addr learned my addr 172.21.15.6:0/3239840941 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783163149 0 0) 0x7f6a180a8010 con 0x7f6a180a4290 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): my global_id is 5675 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a04003530 con 0x7f6a180a4290 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161677492 0 0) 0x7f6a1813ef20 con 0x7f6a180a37a0 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): my global_id is 5694 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a180a8010 con 0x7f6a180a37a0 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3230170495 0 0) 0x7f6a04003530 con 0x7f6a180a4290 2024-09-16T05:38:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a1813ef20 con 0x7f6a180a4290 2024-09-16T05:38:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2876573815 0 0) 0x7f6a180a8010 con 0x7f6a180a37a0 2024-09-16T05:38:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a04003530 con 0x7f6a180a37a0 2024-09-16T05:38:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a10001800 con 0x7f6a180a4290 2024-09-16T05:38:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): dump: 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2155678525 0 0) 0x7f6a1813c090 con 0x7f6a180a1a00 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): my global_id is 104119 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a180a8010 con 0x7f6a180a1a00 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a0c0019c0 con 0x7f6a180a37a0 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:38.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient(hunting): dump: 2024-09-16T05:38:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2047914164 0 0) 0x7f6a1813ef20 con 0x7f6a180a4290 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 >> v1:172.21.15.63:6789/0 conn(0x7f6a180a1a00 legacy=0x7f6a1813b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 >> v1:172.21.15.6:6789/0 conn(0x7f6a180a37a0 legacy=0x7f6a1813b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 monclient: found mon.c 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a1813dc90 con 0x7f6a180a4290 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient: _finish_auth 0 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 5 monclient: authenticate success, global_id 5675 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 10 monclient: _renew_subs 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a1813ec40 con 0x7f6a180a4290 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 10 monclient: _renew_subs 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a1813f620 con 0x7f6a180a4290 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a10003600 con 0x7f6a180a4290 2024-09-16T05:38:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a10003950 con 0x7f6a180a4290 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 10 monclient: dump: 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a157fa640 4 set_mon_vals no callback set 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69f8005090 con 0x7f6a180a4290 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6a10003b70 con 0x7f6a180a4290 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(143..143 src has 1..143) ==== 7135+0+0 (unknown 1594094581 0 0) 0x7f6a10043da0 con 0x7f6a180a4290 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.985+0000 7f6a177fe640 10 monclient: get_auth_request con 0x7f6a04036bd0 auth_method 0 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.989+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6a10014240 con 0x7f6a180a4290 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.989+0000 7f69fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:37.989+0000 7f69fffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.217+0000 7f6a1dd0e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:38:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.217+0000 7f6a1dd0e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:38:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.217+0000 7f6a1dd0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.217+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f69f8005b90 con 0x7f6a180a4290 2024-09-16T05:38:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f69fffff640 1 -- 172.21.15.6:0/3239840941 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f6a10013b40 con 0x7f6a180a4290 2024-09-16T05:38:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f69fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:38:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f69fffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:38.226 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.226 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=1892242244,config=3541143443,health=2440873570,kv=2361287224,logm=2634192984}) 2024-09-16T05:38:38.226 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=11,logm=61} crc {auth=3743977287,config=3541143443,health=2440873570,kv=1476060497,logm=1940339168}) 2024-09-16T05:38:38.226 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.226 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=1892242244,config=3541143443,health=2440873570,kv=2361287224,logm=2634192984}) 2024-09-16T05:38:38.226 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.221+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=11,kv=11,logm=60} crc {auth=3255798330,config=3541143443,health=786169053,kv=2361287224,logm=1490180799}) 2024-09-16T05:38:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6a04036bd0 legacy=0x7f6a04039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f6a1dd0e640 10 monclient: shutdown 2024-09-16T05:38:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f6a1dd0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 >> v1:172.21.15.6:6790/0 conn(0x7f6a180a4290 legacy=0x7f6a181d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 shutdown_connections 2024-09-16T05:38:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 >> 172.21.15.6:0/3239840941 conn(0x7f6a180a58a0 msgr2=0x7f6a18140060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 shutdown_connections 2024-09-16T05:38:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f6a1dd0e640 1 -- 172.21.15.6:0/3239840941 wait complete. 2024-09-16T05:38:38.229 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.229 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421371271}) 2024-09-16T05:38:38.229 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1516537949}) 2024-09-16T05:38:38.229 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.229 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421371271}) 2024-09-16T05:38:38.229 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.225+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988494225}) 2024-09-16T05:38:38.231 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.229+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.231 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.229+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438552452}) 2024-09-16T05:38:38.232 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.229+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2090763584}) 2024-09-16T05:38:38.232 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.229+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.232 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.229+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438552452}) 2024-09-16T05:38:38.232 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.229+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2047269012}) 2024-09-16T05:38:38.234 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.229+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.235 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593281758}) 2024-09-16T05:38:38.235 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3479443635}) 2024-09-16T05:38:38.235 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.235 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593281758}) 2024-09-16T05:38:38.235 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3828466408}) 2024-09-16T05:38:38.237 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.238 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466329827}) 2024-09-16T05:38:38.238 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4175464364}) 2024-09-16T05:38:38.238 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.238 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466329827}) 2024-09-16T05:38:38.238 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.233+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=978367974}) 2024-09-16T05:38:38.240 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.237+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.241 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.237+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693058612}) 2024-09-16T05:38:38.241 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.237+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3150980722}) 2024-09-16T05:38:38.241 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.237+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.241 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.237+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693058612}) 2024-09-16T05:38:38.241 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.237+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018147513}) 2024-09-16T05:38:38.243 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.241+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.244 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.241+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3656229942}) 2024-09-16T05:38:38.244 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.241+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679217244}) 2024-09-16T05:38:38.244 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.241+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.244 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.241+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3656229942}) 2024-09-16T05:38:38.244 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.241+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2549430087}) 2024-09-16T05:38:38.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:38:38.246 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.241+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.246 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192244864}) 2024-09-16T05:38:38.247 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2024961462}) 2024-09-16T05:38:38.247 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.247 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192244864}) 2024-09-16T05:38:38.247 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3651514195}) 2024-09-16T05:38:38.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008769097}) 2024-09-16T05:38:38.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4154378737}) 2024-09-16T05:38:38.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008769097}) 2024-09-16T05:38:38.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.245+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748773083}) 2024-09-16T05:38:38.253 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.249+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.253 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.249+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1545527997}) 2024-09-16T05:38:38.253 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.249+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4136024795}) 2024-09-16T05:38:38.253 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.249+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.253 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.249+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1545527997}) 2024-09-16T05:38:38.253 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.249+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3867953956}) 2024-09-16T05:38:38.256 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.253+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.256 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.253+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200337172}) 2024-09-16T05:38:38.256 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.253+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1176113772}) 2024-09-16T05:38:38.256 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.253+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.256 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.253+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200337172}) 2024-09-16T05:38:38.256 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.253+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666922979}) 2024-09-16T05:38:38.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597491856}) 2024-09-16T05:38:38.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3443808445}) 2024-09-16T05:38:38.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597491856}) 2024-09-16T05:38:38.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511557769}) 2024-09-16T05:38:38.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347588555}) 2024-09-16T05:38:38.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=547058593}) 2024-09-16T05:38:38.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347588555}) 2024-09-16T05:38:38.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.257+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390863788}) 2024-09-16T05:38:38.265 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.261+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.265 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.261+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911938743}) 2024-09-16T05:38:38.265 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.261+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2051104120}) 2024-09-16T05:38:38.265 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.261+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.265 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.261+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911938743}) 2024-09-16T05:38:38.265 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.261+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4266544398}) 2024-09-16T05:38:38.267 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.267 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753118558}) 2024-09-16T05:38:38.267 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3003645341}) 2024-09-16T05:38:38.268 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.268 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753118558}) 2024-09-16T05:38:38.268 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1162287624}) 2024-09-16T05:38:38.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597654535}) 2024-09-16T05:38:38.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=774764910}) 2024-09-16T05:38:38.270 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.270 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597654535}) 2024-09-16T05:38:38.270 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.265+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538322998}) 2024-09-16T05:38:38.272 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.272 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20111159}) 2024-09-16T05:38:38.272 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2278092342}) 2024-09-16T05:38:38.272 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.272 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20111159}) 2024-09-16T05:38:38.272 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1406308442}) 2024-09-16T05:38:38.274 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.274 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=51,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=12,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=712761440,mdsmap=3140514009,mgr=827571645,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=883719832,mon_config_key=2822818051,monmap=3625042558,osd_metadata=4164660051,osd_pg_creating=540871547,osd_snap=723727393,osdmap=2163629777}) 2024-09-16T05:38:38.274 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=3} crc {logm=2459884030,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2278098619,mon_config_key=2880090005,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=540871547,osd_snap=723727393,osdmap=3127721017}) 2024-09-16T05:38:38.275 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.275 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=51,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=12,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=6} crc {logm=712761440,mdsmap=3140514009,mgr=827571645,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=883719832,mon_config_key=2822818051,monmap=3625042558,osd_metadata=4164660051,osd_pg_creating=540871547,osd_snap=723727393,osdmap=2163629777}) 2024-09-16T05:38:38.275 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.269+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=12,monmap=3,osd_metadata=6,osd_snap=2,osdmap=9} crc {logm=3979146416,mdsmap=1832746017,mgr=1552851386,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2278098619,mon_config_key=2848018662,monmap=3625042558,osd_metadata=1319166417,osd_snap=723727393,osdmap=1149805030}) 2024-09-16T05:38:38.277 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.273+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.277 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.273+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1733084655}) 2024-09-16T05:38:38.277 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.273+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2820361384}) 2024-09-16T05:38:38.277 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.273+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.277 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.273+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1733084655}) 2024-09-16T05:38:38.277 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.273+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2594514273}) 2024-09-16T05:38:38.280 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.277+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.280 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.277+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1988783805}) 2024-09-16T05:38:38.280 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.277+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2213898256}) 2024-09-16T05:38:38.280 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.277+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.281 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.277+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1988783805}) 2024-09-16T05:38:38.281 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.277+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1281132915}) 2024-09-16T05:38:38.287 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.285+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:38:38.287 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.285+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=18} crc {osdmap=3821859889}) 2024-09-16T05:38:38.287 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:38:38.285+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=15} crc {osdmap=2966602395}) 2024-09-16T05:38:38.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.698+0000 7feb1ffff640 1 -- 172.21.15.63:0/242050499 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v144) ==== 124+0+0 (unknown 3460844211 0 0) 0x7feb280403a0 con 0x7feb380a07a0 2024-09-16T05:38:38.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.698+0000 7feb1ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T05:38:38.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.698+0000 7feb1ffff640 10 monclient: _finish_command 3 = system:0 set pool 12 size to 2 2024-09-16T05:38:38.704 INFO:tasks.workunit.client.0.smithi063.stderr:set pool 12 size to 2 2024-09-16T05:38:38.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.702+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 >> v1:172.21.15.63:6812/3643207211 conn(0x7feb24036a20 legacy=0x7feb24038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.702+0000 7feb3daad640 10 monclient: shutdown 2024-09-16T05:38:38.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.702+0000 7feb3daad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:38.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.702+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 >> v1:172.21.15.6:6790/0 conn(0x7feb380a07a0 legacy=0x7feb38135ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.702+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 shutdown_connections 2024-09-16T05:38:38.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.702+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 >> 172.21.15.63:0/242050499 conn(0x7feb38075470 msgr2=0x7feb38136e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:38.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.702+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 shutdown_connections 2024-09-16T05:38:38.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.702+0000 7feb3daad640 1 -- 172.21.15.63:0/242050499 wait complete. 2024-09-16T05:38:38.728 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.846+0000 7f63a54de640 10 monclient: get_monmap_and_config 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.846+0000 7f63a54de640 10 monclient: build_initial_monmap 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.846+0000 7f63a54de640 10 monclient: monmap: 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:38.850899+0000 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:38.850899+0000 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:38.851 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:38.852 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:38.852 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:38.852 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:38.852 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:38.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.846+0000 7f63a54de640 1 Processor -- start 2024-09-16T05:38:38.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.846+0000 7f63a54de640 1 -- start start 2024-09-16T05:38:38.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.846+0000 7f63a54de640 10 monclient: init 2024-09-16T05:38:38.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:38.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:38.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): picked mon.noname-b con 0x7f63a00a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:38.853 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): picked mon.noname-c con 0x7f63a00a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:38.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): picked mon.noname-a con 0x7f63a00a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:38.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a4fd0 con 0x7f63a00a4790 2024-09-16T05:38:38.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a5b40 con 0x7f63a00a08a0 2024-09-16T05:38:38.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a1a70 con 0x7f63a00a3ca0 2024-09-16T05:38:38.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:38.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): authenticate will time out at 1175.628591s 2024-09-16T05:38:38.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639e7fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f63a00a08a0 0x7f63a013beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43768/0 (socket says 172.21.15.63:43768) 2024-09-16T05:38:38.854 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639e7fc640 1 -- 172.21.15.63:0/3261618246 learned_addr learned my addr 172.21.15.63:0/3261618246 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2055439269 0 0) 0x7f63a00a5b40 con 0x7f63a00a08a0 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient(hunting): my global_id is 104122 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6388003530 con 0x7f63a00a08a0 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 827991001 0 0) 0x7f6388003530 con 0x7f63a00a08a0 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 >> v1:172.21.15.6:6790/0 conn(0x7f63a00a3ca0 legacy=0x7f63a00a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 >> v1:172.21.15.6:6789/0 conn(0x7f63a00a4790 legacy=0x7f63a013f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 monclient: found mon.noname-c 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a00a53e0 con 0x7f63a00a08a0 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6394001620 con 0x7f63a00a08a0 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: dump: 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:38.860 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 5 monclient: authenticate success, global_id 104122 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6394001890 con 0x7f63a00a08a0 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:38.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 1 -- 172.21.15.63:0/3261618246 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6394001c60 con 0x7f63a00a08a0 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 4 set_mon_vals no callback set 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639dffb640 10 monclient: dump: 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:38.862 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: shutdown 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- 172.21.15.63:0/3261618246 >> v1:172.21.15.63:6789/0 conn(0x7f63a00a08a0 legacy=0x7f63a013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- 172.21.15.63:0/3261618246 shutdown_connections 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- 172.21.15.63:0/3261618246 >> 172.21.15.63:0/3261618246 conn(0x7f63a00a67c0 msgr2=0x7f63a0075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- 172.21.15.63:0/3261618246 shutdown_connections 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- 172.21.15.63:0/3261618246 wait complete. 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: build_initial_monmap 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: monmap: 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:38.863 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 Processor -- start 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- start start 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: init 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:38.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): picked mon.noname-c con 0x7f63a00a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): picked mon.noname-a con 0x7f63a00a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): picked mon.noname-b con 0x7f63a00a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a0127390 con 0x7f63a00a3ca0 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a00a8890 con 0x7f63a00a4790 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a0127670 con 0x7f63a00a08a0 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f63a54de640 10 monclient(hunting): authenticate will time out at 1175.631752s 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639f7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f63a00a4790 0x7f63a01d3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43782/0 (socket says 172.21.15.63:43782) 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.850+0000 7f639f7fe640 1 -- 172.21.15.63:0/2865539053 learned_addr learned my addr 172.21.15.63:0/2865539053 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3924188346 0 0) 0x7f63a00a8890 con 0x7f63a00a4790 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:38.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): my global_id is 104125 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f638c003530 con 0x7f63a00a4790 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 379091001 0 0) 0x7f638c003530 con 0x7f63a00a4790 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63a00a8890 con 0x7f63a00a4790 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6398001c80 con 0x7f63a00a4790 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): dump: 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:38.866 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 677644955 0 0) 0x7f63a0127670 con 0x7f63a00a08a0 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): my global_id is 5681 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f638c003530 con 0x7f63a00a08a0 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2195936086 0 0) 0x7f63a0127390 con 0x7f63a00a3ca0 2024-09-16T05:38:38.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient(hunting): my global_id is 5700 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63a0127670 con 0x7f63a00a3ca0 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2971990393 0 0) 0x7f63a00a8890 con 0x7f63a00a4790 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 >> v1:172.21.15.6:6790/0 conn(0x7f63a00a08a0 legacy=0x7f63a0136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 >> v1:172.21.15.6:6789/0 conn(0x7f63a00a3ca0 legacy=0x7f63a01d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 monclient: found mon.b 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a0137650 con 0x7f63a00a4790 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:38.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 5 monclient: authenticate success, global_id 104125 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 10 monclient: _renew_subs 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63a01cb1d0 con 0x7f63a00a4790 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 10 monclient: _renew_subs 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63a01d5120 con 0x7f63a00a4790 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6398002000 con 0x7f63a00a4790 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6398003e70 con 0x7f63a00a4790 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f639d7fa640 4 set_mon_vals no callback set 2024-09-16T05:38:38.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 10 monclient: dump: 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:38.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:38.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6380005090 con 0x7f63a00a4790 2024-09-16T05:38:38.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6398004090 con 0x7f63a00a4790 2024-09-16T05:38:38.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(144..144 src has 1..144) ==== 7135+0+0 (unknown 1572139235 0 0) 0x7f6398045420 con 0x7f63a00a4790 2024-09-16T05:38:38.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.854+0000 7f639effd640 10 monclient: get_auth_request con 0x7f638c036d30 auth_method 0 2024-09-16T05:38:38.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.878+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6398015920 con 0x7f63a00a4790 2024-09-16T05:38:38.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.878+0000 7f63877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:38.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:38.878+0000 7f63877fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:39.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.050+0000 7f63a54de640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:39.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.050+0000 7f63a54de640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:39.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.050+0000 7f63a54de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:39.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.050+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7f6380005380 con 0x7f63a00a4790 2024-09-16T05:38:39.246 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:38:39.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:38:39.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient: get_monmap_and_config 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient: build_initial_monmap 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient: monmap: 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:39.380177+0000 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:39.380177+0000 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:39.380 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:39.381 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 1 Processor -- start 2024-09-16T05:38:39.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 1 -- start start 2024-09-16T05:38:39.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient: init 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient(hunting): picked mon.noname-c con 0x7f567409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient(hunting): picked mon.noname-a con 0x7f567409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient(hunting): picked mon.noname-b con 0x7f56740a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56740a1370 con 0x7f567409e230 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f567409ead0 con 0x7f567409aa30 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f567409edb0 con 0x7f56740a0da0 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 10 monclient(hunting): authenticate will time out at 1181.186802s 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567a235640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f56740a0da0 0x7f567413d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46156/0 (socket says 172.21.15.6:46156) 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567a235640 1 -- 172.21.15.6:0/129545497 learned_addr learned my addr 172.21.15.6:0/129545497 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:39.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2438534696 0 0) 0x7f567409edb0 con 0x7f56740a0da0 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient(hunting): my global_id is 5684 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5660003530 con 0x7f56740a0da0 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3263012644 0 0) 0x7f56740a1370 con 0x7f567409e230 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient(hunting): my global_id is 5703 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f567409edb0 con 0x7f567409e230 2024-09-16T05:38:39.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1391499633 0 0) 0x7f5660003530 con 0x7f56740a0da0 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 >> v1:172.21.15.63:6789/0 conn(0x7f567409aa30 legacy=0x7f567409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 >> v1:172.21.15.6:6789/0 conn(0x7f567409e230 legacy=0x7f56741399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: _finish_hunting 0 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 monclient: found mon.noname-b 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567409dd50 con 0x7f56740a0da0 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: _finish_auth 0 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: _check_auth_tickets 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f566c001620 con 0x7f56740a0da0 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 5 monclient: authenticate success, global_id 5684 2024-09-16T05:38:39.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: dump: 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f567bc98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f566c001850 con 0x7f56740a0da0 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 1 -- 172.21.15.6:0/129545497 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f566c001c60 con 0x7f56740a0da0 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.377+0000 7f5678a32640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f5678a32640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f5678a32640 10 monclient: dump: 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 4 set_mon_vals no callback set 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: shutdown 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- 172.21.15.6:0/129545497 >> v1:172.21.15.6:6790/0 conn(0x7f56740a0da0 legacy=0x7f567413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- 172.21.15.6:0/129545497 shutdown_connections 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- 172.21.15.6:0/129545497 >> 172.21.15.6:0/129545497 conn(0x7f56740a1650 msgr2=0x7f56740a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- 172.21.15.6:0/129545497 shutdown_connections 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- 172.21.15.6:0/129545497 wait complete. 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: build_initial_monmap 2024-09-16T05:38:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: monmap: 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:39.390 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 Processor -- start 2024-09-16T05:38:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- start start 2024-09-16T05:38:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: init 2024-09-16T05:38:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient(hunting): picked mon.noname-b con 0x7f567409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient(hunting): picked mon.noname-c con 0x7f567409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient(hunting): picked mon.noname-a con 0x7f56740a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56740a0210 con 0x7f56740a0da0 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56740740c0 con 0x7f567409aa30 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f567413b840 con 0x7f567409e230 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient(hunting): authenticate will time out at 1181.190349s 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f5679233640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f567409e230 0x7f567409c4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46158/0 (socket says 172.21.15.6:46158) 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f5679233640 1 -- 172.21.15.6:0/2197110329 learned_addr learned my addr 172.21.15.6:0/2197110329 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599496660 0 0) 0x7f567413b840 con 0x7f567409e230 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): my global_id is 5687 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5658003530 con 0x7f567409e230 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618172604 0 0) 0x7f56740a0210 con 0x7f56740a0da0 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): my global_id is 5706 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f567413b840 con 0x7f56740a0da0 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3700482760 0 0) 0x7f5658003530 con 0x7f567409e230 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56740a0210 con 0x7f567409e230 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 548989157 0 0) 0x7f56740740c0 con 0x7f567409aa30 2024-09-16T05:38:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): my global_id is 104131 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5658003530 con 0x7f567409aa30 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5664001a20 con 0x7f567409e230 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3561046711 0 0) 0x7f567413b840 con 0x7f56740a0da0 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56740740c0 con 0x7f56740a0da0 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f566c001a00 con 0x7f56740a0da0 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient(hunting): dump: 2024-09-16T05:38:39.397 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3670488524 0 0) 0x7f56740a0210 con 0x7f567409e230 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 >> v1:172.21.15.63:6789/0 conn(0x7f567409aa30 legacy=0x7f567409bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 >> v1:172.21.15.6:6789/0 conn(0x7f56740a0da0 legacy=0x7f56741d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 monclient: found mon.c 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567413bb20 con 0x7f567409e230 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 5 monclient: authenticate success, global_id 5687 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: _renew_subs 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f567413a520 con 0x7f567409e230 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: _renew_subs 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f567409cdc0 con 0x7f567409e230 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56640038b0 con 0x7f567409e230 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5664003c00 con 0x7f567409e230 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566bfff640 4 set_mon_vals no callback set 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.381+0000 7f566a7fc640 10 monclient: dump: 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.385+0000 7f567bc98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.385+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5664003da0 con 0x7f567409e230 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.385+0000 7f567bc98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.385+0000 7f567bc98640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.385+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f565c005090 con 0x7f567409e230 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.385+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(144..144 src has 1..144) ==== 7135+0+0 (unknown 1572139235 0 0) 0x7f5664044f20 con 0x7f567409e230 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.385+0000 7f5679a34640 10 monclient: get_auth_request con 0x7f5658036e50 auth_method 0 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.389+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5664015360 con 0x7f567409e230 2024-09-16T05:38:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.389+0000 7f566a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:39.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.389+0000 7f566a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:39.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.617+0000 7f567bc98640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:38:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.617+0000 7f567bc98640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.617+0000 7f567bc98640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.617+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f565c005380 con 0x7f567409e230 2024-09-16T05:38:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.617+0000 7f566a7fc640 1 -- 172.21.15.6:0/2197110329 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1744450195) 0x7f5664019010 con 0x7f567409e230 2024-09-16T05:38:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.617+0000 7f566a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:39.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.617+0000 7f566a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:39.621 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:38:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.621+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5658036e50 legacy=0x7f5658039310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.621+0000 7f567bc98640 10 monclient: shutdown 2024-09-16T05:38:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.621+0000 7f567bc98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.621+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 >> v1:172.21.15.6:6790/0 conn(0x7f567409e230 legacy=0x7f567409c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.621+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 shutdown_connections 2024-09-16T05:38:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.621+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 >> 172.21.15.6:0/2197110329 conn(0x7f56740a1650 msgr2=0x7f56740a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.621+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 shutdown_connections 2024-09-16T05:38:39.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.621+0000 7f567bc98640 1 -- 172.21.15.6:0/2197110329 wait complete. 2024-09-16T05:38:39.640 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:38:39.640 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:38:39.640 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:38:39.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:38:39.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.718+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v145) ==== 82+0+0 (unknown 3140719339 0 0) 0x7f6398015220 con 0x7f63a00a4790 2024-09-16T05:38:39.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.718+0000 7f63877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:39.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.718+0000 7f63877fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient: get_monmap_and_config 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient: build_initial_monmap 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient: monmap: 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:39.763543+0000 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:39.763543+0000 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:39.763 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.764 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:39.764 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:39.764 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:39.764 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 Processor -- start 2024-09-16T05:38:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- start start 2024-09-16T05:38:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient: init 2024-09-16T05:38:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e40a0df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:39.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e4135330 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e4138e40 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e409d2f0 con 0x7fc3e4135330 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e413d960 con 0x7fc3e40a0df0 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e413eb40 con 0x7fc3e4138e40 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient(hunting): authenticate will time out at 1181.570083s 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3e886a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc3e4135330 0x7fc3e4137720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58862/0 (socket says 172.21.15.6:58862) 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3e886a640 1 -- 172.21.15.6:0/413089244 learned_addr learned my addr 172.21.15.6:0/413089244 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030255 0 0) 0x7fc3e409d2f0 con 0x7fc3e4135330 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): my global_id is 5709 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d0003450 con 0x7fc3e4135330 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2421825326 0 0) 0x7fc3e413eb40 con 0x7fc3e4138e40 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): my global_id is 5690 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3e409d2f0 con 0x7fc3e4138e40 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 567385137 0 0) 0x7fc3e413d960 con 0x7fc3e40a0df0 2024-09-16T05:38:39.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient(hunting): my global_id is 104134 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3e413eb40 con 0x7fc3e40a0df0 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3264152194 0 0) 0x7fc3d0003450 con 0x7fc3e4135330 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 >> v1:172.21.15.6:6790/0 conn(0x7fc3e4138e40 legacy=0x7fc3e413b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 >> v1:172.21.15.63:6789/0 conn(0x7fc3e40a0df0 legacy=0x7fc3e409de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 monclient: found mon.noname-a 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e413fd20 con 0x7fc3e4135330 2024-09-16T05:38:39.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: _finish_auth 0 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 5 monclient: authenticate success, global_id 5709 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3e0001620 con 0x7fc3e4135330 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: dump: 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.775 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3e0001880 con 0x7fc3e4135330 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 4 set_mon_vals no callback set 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 1 -- 172.21.15.6:0/413089244 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3e0001c10 con 0x7fc3e4135330 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:39.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3d7fff640 10 monclient: dump: 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.777 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 10 monclient: shutdown 2024-09-16T05:38:39.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/413089244 >> v1:172.21.15.6:6789/0 conn(0x7fc3e4135330 legacy=0x7fc3e4137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/413089244 shutdown_connections 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/413089244 >> 172.21.15.6:0/413089244 conn(0x7fc3e40a1650 msgr2=0x7fc3e40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/413089244 shutdown_connections 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.761+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/413089244 wait complete. 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: build_initial_monmap 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: monmap: 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:39.781 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 1 Processor -- start 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 1 -- start start 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: init 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:39.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e40a0df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e4135330 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e4138e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e413e470 con 0x7fc3e4138e40 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e413ba80 con 0x7fc3e40a0df0 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e413c040 con 0x7fc3e4135330 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient(hunting): authenticate will time out at 1181.573263s 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3e986c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc3e4138e40 0x7fc3e41cf380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58868/0 (socket says 172.21.15.6:58868) 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3e986c640 1 -- 172.21.15.6:0/3579265368 learned_addr learned my addr 172.21.15.6:0/3579265368 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213889936 0 0) 0x7fc3dc0015f0 con 0x7fc3e4138e40 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): my global_id is 5712 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c8003530 con 0x7fc3e4138e40 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764856625 0 0) 0x7fc3e413c040 con 0x7fc3e4135330 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): my global_id is 5693 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3dc0015f0 con 0x7fc3e4135330 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573669489 0 0) 0x7fc3e413ba80 con 0x7fc3e40a0df0 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): my global_id is 104137 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3e413c040 con 0x7fc3e40a0df0 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2346589163 0 0) 0x7fc3c8003530 con 0x7fc3e4138e40 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3e413ba80 con 0x7fc3e4138e40 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3dc001860 con 0x7fc3e4138e40 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 559776945 0 0) 0x7fc3dc0015f0 con 0x7fc3e4135330 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3c8003530 con 0x7fc3e4135330 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3e0001380 con 0x7fc3e4135330 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1545281782 0 0) 0x7fc3e413c040 con 0x7fc3e40a0df0 2024-09-16T05:38:39.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3dc0015f0 con 0x7fc3e40a0df0 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 694345659 0 0) 0x7fc3e413ba80 con 0x7fc3e4138e40 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 >> v1:172.21.15.6:6790/0 conn(0x7fc3e4135330 legacy=0x7fc3e41cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 >> v1:172.21.15.63:6789/0 conn(0x7fc3e40a0df0 legacy=0x7fc3e4134b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 monclient: found mon.a 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e413b7a0 con 0x7fc3e4138e40 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 5 monclient: authenticate success, global_id 5712 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: _renew_subs 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e413bd60 con 0x7fc3e4138e40 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: _renew_subs 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3e41d0aa0 con 0x7fc3e4138e40 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3dc003cb0 con 0x7fc3e4138e40 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3dc004000 con 0x7fc3e4138e40 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d77fe640 4 set_mon_vals no callback set 2024-09-16T05:38:39.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient: dump: 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc3dc004000 con 0x7fc3e4138e40 2024-09-16T05:38:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3c40065d0 con 0x7fc3c4002870 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7fc3dc045400 con 0x7fc3e4138e40 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156283985 0 0) 0x7fc3c40065d0 con 0x7fc3c4002870 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 10 monclient(hunting): my global_id is 5715 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3e413ba80 con 0x7fc3c4002870 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.765+0000 7fc3e886a640 10 monclient: get_auth_request con 0x7fc3c8036c30 auth_method 0 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4101671778 0 0) 0x7fc3e413ba80 con 0x7fc3c4002870 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3c40065d0 con 0x7fc3c4002870 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3d80037a0 con 0x7fc3c4002870 2024-09-16T05:38:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 10 monclient: dump: 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4130829409 0 0) 0x7fc3c40065d0 con 0x7fc3c4002870 2024-09-16T05:38:39.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3c400a160 con 0x7fc3c4002870 2024-09-16T05:38:39.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc3d8005a00 con 0x7fc3c4002870 2024-09-16T05:38:39.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:39.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:39.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 >> v1:172.21.15.6:6789/0 conn(0x7fc3c4002870 legacy=0x7fc3c4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.769+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 >> v1:172.21.15.6:6789/0 conn(0x7fc3c4002870 legacy=0x7fc3c4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:39.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3eb2cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:38:39.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3c4009e30 con 0x7fc3c40090f0 2024-09-16T05:38:39.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3876898499 0 0) 0x7fc3c4009e30 con 0x7fc3c40090f0 2024-09-16T05:38:39.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient(hunting): my global_id is 5718 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c40065d0 con 0x7fc3c40090f0 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4111893139 0 0) 0x7fc3c40065d0 con 0x7fc3c40090f0 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3c4009e30 con 0x7fc3c40090f0 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3d8005090 con 0x7fc3c40090f0 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient: dump: 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 362714498 0 0) 0x7fc3c4009e30 con 0x7fc3c40090f0 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc3c400c3a0 con 0x7fc3c40090f0 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 3885968028) 0x7fc3d8004ef0 con 0x7fc3c40090f0 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:38:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 >> v1:172.21.15.6:6789/0 conn(0x7fc3c40090f0 legacy=0x7fc3c4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.773+0000 7fc3d5ffb640 1 -- 172.21.15.6:0/3579265368 >> v1:172.21.15.6:6789/0 conn(0x7fc3c40090f0 legacy=0x7fc3c4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 20, 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 248829, 2024-09-16T05:38:39.797 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:39.798 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:39.799 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:38:39.800 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:39.801 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.802 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.803 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:39.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:38:39.806 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:39.807 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:39.808 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:38:39.809 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc3c8036c30 legacy=0x7fc3c80390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3eb2cf640 10 monclient: shutdown 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3eb2cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 >> v1:172.21.15.6:6789/0 conn(0x7fc3e4138e40 legacy=0x7fc3e41cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3e986c640 1 -- 172.21.15.6:0/3579265368 reap_dead start 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3e986c640 1 -- 172.21.15.6:0/3579265368 reap_dead start 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 shutdown_connections 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 >> 172.21.15.6:0/3579265368 conn(0x7fc3e40a1650 msgr2=0x7fc3e40a38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 shutdown_connections 2024-09-16T05:38:39.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.777+0000 7fc3eb2cf640 1 -- 172.21.15.6:0/3579265368 wait complete. 2024-09-16T05:38:39.812 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:38:39.812 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:38:39.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:38:39.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.830+0000 7f63a54de640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:39.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.830+0000 7f63a54de640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:39.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.830+0000 7f63a54de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:39.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.830+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7f6380002220 con 0x7f63a00a4790 2024-09-16T05:38:39.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.830+0000 7f63877fe640 1 -- 172.21.15.63:0/2865539053 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v145) ==== 115+0+0 (unknown 292995017 0 0) 0x7f639801ddd0 con 0x7f63a00a4790 2024-09-16T05:38:39.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.830+0000 7f63877fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:38:39.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.830+0000 7f63877fe640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T05:38:39.836 INFO:tasks.workunit.client.0.smithi063.stderr:compat weight-set already created 2024-09-16T05:38:39.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.834+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 >> v1:172.21.15.63:6812/3643207211 conn(0x7f638c036d30 legacy=0x7f638c0391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.834+0000 7f63a54de640 10 monclient: shutdown 2024-09-16T05:38:39.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.834+0000 7f63a54de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:39.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.834+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 >> v1:172.21.15.63:6789/0 conn(0x7f63a00a4790 legacy=0x7f63a01d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.834+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 shutdown_connections 2024-09-16T05:38:39.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.834+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 >> 172.21.15.63:0/2865539053 conn(0x7f63a00a67c0 msgr2=0x7f63a00701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:39.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.834+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 shutdown_connections 2024-09-16T05:38:39.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.834+0000 7f63a54de640 1 -- 172.21.15.63:0/2865539053 wait complete. 2024-09-16T05:38:39.854 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set create cool flat 2024-09-16T05:38:39.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient: get_monmap_and_config 2024-09-16T05:38:39.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient: build_initial_monmap 2024-09-16T05:38:39.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient: monmap: 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:39.927917+0000 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:39.927917+0000 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:39.928 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 1 Processor -- start 2024-09-16T05:38:39.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 1 -- start start 2024-09-16T05:38:39.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient: init 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient(hunting): picked mon.noname-a con 0x7fb8b40a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient(hunting): picked mon.noname-c con 0x7fb8b40a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient(hunting): picked mon.noname-b con 0x7fb8b40a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b406e520 con 0x7fb8b40a37a0 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b40a0780 con 0x7fb8b40a4290 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b40a4c70 con 0x7fb8b40a1a00 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient(hunting): authenticate will time out at 1181.734501s 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb8b40a1a00 0x7fb8b413f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46192/0 (socket says 172.21.15.6:46192) 2024-09-16T05:38:39.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b3fff640 1 -- 172.21.15.6:0/1769223144 learned_addr learned my addr 172.21.15.6:0/1769223144 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb8b40a37a0 0x7fb8b40a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58890/0 (socket says 172.21.15.6:58890) 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 740841251 0 0) 0x7fb8b40a4c70 con 0x7fb8b40a1a00 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): my global_id is 5696 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb89c003530 con 0x7fb8b40a1a00 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2256145819 0 0) 0x7fb8b406e520 con 0x7fb8b40a37a0 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): my global_id is 5721 2024-09-16T05:38:39.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8b40a4c70 con 0x7fb8b40a37a0 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983420682 0 0) 0x7fb8b40a0780 con 0x7fb8b40a4290 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient(hunting): my global_id is 104140 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8b406e520 con 0x7fb8b40a4290 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1631974786 0 0) 0x7fb8b40a4c70 con 0x7fb8b40a37a0 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 >> v1:172.21.15.6:6790/0 conn(0x7fb8b40a1a00 legacy=0x7fb8b413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 >> v1:172.21.15.63:6789/0 conn(0x7fb8b40a4290 legacy=0x7fb8b413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 monclient: found mon.noname-a 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b40a65f0 con 0x7fb8b40a37a0 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb8a4001600 con 0x7fb8b40a37a0 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: dump: 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.935 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 5 monclient: authenticate success, global_id 5721 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8a4001860 con 0x7fb8b40a37a0 2024-09-16T05:38:39.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 1 -- 172.21.15.6:0/1769223144 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb8a4001cb0 con 0x7fb8b40a37a0 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8b27fc640 10 monclient: dump: 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.937 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.925+0000 7fb8ba11a640 4 set_mon_vals no callback set 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: shutdown 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- 172.21.15.6:0/1769223144 >> v1:172.21.15.6:6789/0 conn(0x7fb8b40a37a0 legacy=0x7fb8b40a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- 172.21.15.6:0/1769223144 shutdown_connections 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- 172.21.15.6:0/1769223144 >> 172.21.15.6:0/1769223144 conn(0x7fb8b40a58a0 msgr2=0x7fb8b40a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- 172.21.15.6:0/1769223144 shutdown_connections 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- 172.21.15.6:0/1769223144 wait complete. 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: build_initial_monmap 2024-09-16T05:38:39.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: monmap: 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 Processor -- start 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- start start 2024-09-16T05:38:39.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: init 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient(hunting): picked mon.noname-c con 0x7fb8b40a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient(hunting): picked mon.noname-a con 0x7fb8b40a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient(hunting): picked mon.noname-b con 0x7fb8b40a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b4135b10 con 0x7fb8b40a37a0 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b40a0100 con 0x7fb8b40a4290 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b4133c90 con 0x7fb8b40a1a00 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient(hunting): authenticate will time out at 1181.737859s 2024-09-16T05:38:39.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8b2ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb8b40a37a0 0x7fb8b41cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58896/0 (socket says 172.21.15.6:58896) 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8b2ffd640 1 -- 172.21.15.6:0/2675375986 learned_addr learned my addr 172.21.15.6:0/2675375986 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890966136 0 0) 0x7fb8b4135b10 con 0x7fb8b40a37a0 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): my global_id is 5724 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8a0003530 con 0x7fb8b40a37a0 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016093477 0 0) 0x7fb8b4133c90 con 0x7fb8b40a1a00 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): my global_id is 5699 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8b4135b10 con 0x7fb8b40a1a00 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 182004026 0 0) 0x7fb8a0003530 con 0x7fb8b40a37a0 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8b4133c90 con 0x7fb8b40a37a0 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb8a80019a0 con 0x7fb8b40a37a0 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): dump: 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1802250793 0 0) 0x7fb8b40a0100 con 0x7fb8b40a4290 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): my global_id is 104143 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8a0003530 con 0x7fb8b40a4290 2024-09-16T05:38:39.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1268187956 0 0) 0x7fb8b4135b10 con 0x7fb8b40a1a00 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8b40a0100 con 0x7fb8b40a1a00 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb8a4001bb0 con 0x7fb8b40a1a00 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient(hunting): dump: 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.949 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2811312882 0 0) 0x7fb8b4133c90 con 0x7fb8b40a37a0 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 >> v1:172.21.15.6:6790/0 conn(0x7fb8b40a1a00 legacy=0x7fb8b4135f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 >> v1:172.21.15.63:6789/0 conn(0x7fb8b40a4290 legacy=0x7fb8b41d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 monclient: found mon.a 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b4135830 con 0x7fb8b40a37a0 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient: _finish_auth 0 2024-09-16T05:38:39.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 5 monclient: authenticate success, global_id 5724 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: _renew_subs 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b41d4b70 con 0x7fb8b40a37a0 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: _renew_subs 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb8b41d4f10 con 0x7fb8b40a37a0 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8a80038d0 con 0x7fb8b40a37a0 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb8a8003ca0 con 0x7fb8b40a37a0 2024-09-16T05:38:39.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 10 monclient: dump: 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.952 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8b1ffb640 4 set_mon_vals no callback set 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb8940065d0 con 0x7fb894002870 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.929+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb8a8003ec0 con 0x7fb8b40a37a0 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7fb8a80452e0 con 0x7fb8b40a37a0 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3465035311 0 0) 0x7fb8940065d0 con 0x7fb894002870 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient(hunting): my global_id is 5702 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8b4133c90 con 0x7fb894002870 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb8b3fff640 10 monclient: get_auth_request con 0x7fb8a0036ba0 auth_method 0 2024-09-16T05:38:39.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1602195347 0 0) 0x7fb8b4133c90 con 0x7fb894002870 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8940065d0 con 0x7fb894002870 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb8a4004a30 con 0x7fb894002870 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient: dump: 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.954 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 644842433 0 0) 0x7fb8940065d0 con 0x7fb894002870 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb89400a160 con 0x7fb894002870 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb8a40049c0 con 0x7fb894002870 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 >> v1:172.21.15.6:6790/0 conn(0x7fb894002870 legacy=0x7fb894004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.933+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 >> v1:172.21.15.6:6790/0 conn(0x7fb894002870 legacy=0x7fb894004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb8ba11a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:38:39.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb894009e30 con 0x7fb8940090f0 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3040832578 0 0) 0x7fb894009e30 con 0x7fb8940090f0 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 10 monclient(hunting): my global_id is 5705 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8940065d0 con 0x7fb8940090f0 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3122771280 0 0) 0x7fb8940065d0 con 0x7fb8940090f0 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb894009e30 con 0x7fb8940090f0 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb8a4004fc0 con 0x7fb8940090f0 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 10 monclient: dump: 2024-09-16T05:38:39.956 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2682290855 0 0) 0x7fb894009e30 con 0x7fb8940090f0 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.937+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb89400c3a0 con 0x7fb8940090f0 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 920415731) 0x7fb8a4005220 con 0x7fb8940090f0 2024-09-16T05:38:39.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb89bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:38:39.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb89bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:39.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 >> v1:172.21.15.6:6790/0 conn(0x7fb8940090f0 legacy=0x7fb894009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb89bfff640 1 -- 172.21.15.6:0/2675375986 >> v1:172.21.15.6:6790/0 conn(0x7fb8940090f0 legacy=0x7fb894009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:39.958 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:38:39.958 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:39.958 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:39.958 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:38:39.958 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 20, 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 248997, 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:38:39.960 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:39.961 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:38:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:39.963 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:39.964 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:39.965 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.966 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:39.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.968 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:38:39.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:39.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:39.971 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:38:39.971 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb8a0036ba0 legacy=0x7fb8a0039060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8ba11a640 10 monclient: shutdown 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8ba11a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 >> v1:172.21.15.6:6789/0 conn(0x7fb8b40a37a0 legacy=0x7fb8b41cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8b3fff640 1 -- 172.21.15.6:0/2675375986 reap_dead start 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8b3fff640 1 -- 172.21.15.6:0/2675375986 reap_dead start 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 shutdown_connections 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 >> 172.21.15.6:0/2675375986 conn(0x7fb8b40a58a0 msgr2=0x7fb8b41308a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:39.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 shutdown_connections 2024-09-16T05:38:39.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:39.941+0000 7fb8ba11a640 1 -- 172.21.15.6:0/2675375986 wait complete. 2024-09-16T05:38:39.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:38:39.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:38:39.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:38:39.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient: get_monmap_and_config 2024-09-16T05:38:39.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient: build_initial_monmap 2024-09-16T05:38:39.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient: monmap: 2024-09-16T05:38:39.981 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:39.981 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.981 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:39.981057+0000 2024-09-16T05:38:39.981 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:39.981057+0000 2024-09-16T05:38:39.981 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:39.982 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:39.982 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:39.982 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:39.982 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:39.982 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:39.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 Processor -- start 2024-09-16T05:38:39.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 -- start start 2024-09-16T05:38:39.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient: init 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec0134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec00a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec006dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec0135420 con 0x7f7ec006dfa0 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebf7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7ec0134930 0x7f7ec0134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43796/0 (socket says 172.21.15.63:43796) 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebf7fe640 1 -- 172.21.15.63:0/4172845791 learned_addr learned my addr 172.21.15.63:0/4172845791 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 -- 172.21.15.63:0/4172845791 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec009e830 con 0x7f7ec0134930 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 -- 172.21.15.63:0/4172845791 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec00a0dc0 con 0x7f7ec00a0810 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:39.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient(hunting): authenticate will time out at 1176.758302s 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247110199 0 0) 0x7f7ec009e830 con 0x7f7ec0134930 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): my global_id is 104146 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ea8003530 con 0x7f7ec0134930 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2167145851 0 0) 0x7f7ec0135420 con 0x7f7ec006dfa0 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): my global_id is 5727 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ec009e830 con 0x7f7ec006dfa0 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 646285345 0 0) 0x7f7ec00a0dc0 con 0x7f7ec00a0810 2024-09-16T05:38:39.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient(hunting): my global_id is 5708 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ec0135420 con 0x7f7ec00a0810 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 915718535 0 0) 0x7f7ea8003530 con 0x7f7ec0134930 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 >> v1:172.21.15.6:6790/0 conn(0x7f7ec00a0810 legacy=0x7f7ec0139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 >> v1:172.21.15.6:6789/0 conn(0x7f7ec006dfa0 legacy=0x7f7ec013cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 monclient: found mon.noname-c 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:38:39.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec00a2500 con 0x7f7ec0134930 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7eb0001620 con 0x7f7ec0134930 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 5 monclient: authenticate success, global_id 104146 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: dump: 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.988 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7eb0001940 con 0x7f7ec0134930 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 1 -- 172.21.15.63:0/4172845791 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7eb0001d50 con 0x7f7ec0134930 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ebe7fc640 10 monclient: dump: 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:39.989 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 4 set_mon_vals no callback set 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 10 monclient: shutdown 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 -- 172.21.15.63:0/4172845791 >> v1:172.21.15.63:6789/0 conn(0x7f7ec0134930 legacy=0x7f7ec0134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 -- 172.21.15.63:0/4172845791 shutdown_connections 2024-09-16T05:38:39.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 -- 172.21.15.63:0/4172845791 >> 172.21.15.63:0/4172845791 conn(0x7f7ec00a1e90 msgr2=0x7f7ec0075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:39.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.978+0000 7f7ec5b92640 1 -- 172.21.15.63:0/4172845791 shutdown_connections 2024-09-16T05:38:39.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 -- 172.21.15.63:0/4172845791 wait complete. 2024-09-16T05:38:39.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: build_initial_monmap 2024-09-16T05:38:39.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: monmap: 2024-09-16T05:38:39.995 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:39.995 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 Processor -- start 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 -- start start 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: init 2024-09-16T05:38:39.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec006dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec00a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec013ccf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec01cc550 con 0x7f7ec006dfa0 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec011c1b0 con 0x7f7ec00a0810 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec00a4090 con 0x7f7ec013ccf0 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient(hunting): authenticate will time out at 1176.761749s 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ebeffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7ec00a0810 0x7f7ec01cbe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:43810/0 (socket says 172.21.15.63:43810) 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ebeffd640 1 -- 172.21.15.63:0/2291547042 learned_addr learned my addr 172.21.15.63:0/2291547042 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 562456290 0 0) 0x7f7ec011c1b0 con 0x7f7ec00a0810 2024-09-16T05:38:39.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): my global_id is 104149 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eac003530 con 0x7f7ec00a0810 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 712975704 0 0) 0x7f7ec01cc550 con 0x7f7ec006dfa0 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): my global_id is 5730 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ec011c1b0 con 0x7f7ec006dfa0 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228347888 0 0) 0x7f7ec00a4090 con 0x7f7ec013ccf0 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:39.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): my global_id is 5711 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ec01cc550 con 0x7f7ec013ccf0 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1133821660 0 0) 0x7f7eac003530 con 0x7f7ec00a0810 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ec00a4090 con 0x7f7ec00a0810 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7eb40019a0 con 0x7f7ec00a0810 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): dump: 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:39.999 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3072546412 0 0) 0x7f7ec01cc550 con 0x7f7ec013ccf0 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eac003530 con 0x7f7ec013ccf0 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7eb80015e0 con 0x7f7ec013ccf0 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): dump: 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:40.000 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1996243454 0 0) 0x7f7ec011c1b0 con 0x7f7ec006dfa0 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ec01cc550 con 0x7f7ec006dfa0 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7eb00015e0 con 0x7f7ec006dfa0 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:40.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient(hunting): dump: 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1579343582 0 0) 0x7f7ec00a4090 con 0x7f7ec00a0810 2024-09-16T05:38:40.002 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.6:6790/0 conn(0x7f7ec013ccf0 legacy=0x7f7ec01d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.6:6789/0 conn(0x7f7ec006dfa0 legacy=0x7f7ec01cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 monclient: found mon.b 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec009c0f0 con 0x7f7ec00a0810 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient: _finish_auth 0 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 5 monclient: authenticate success, global_id 104149 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: _renew_subs 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:40.003 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec011d380 con 0x7f7ec00a0810 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7eb40038d0 con 0x7f7ec00a0810 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: _renew_subs 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7ec011bf30 con 0x7f7ec00a0810 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7eb4003ca0 con 0x7f7ec00a0810 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ebdffb640 4 set_mon_vals no callback set 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ea7fff640 10 monclient: dump: 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.004 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.982+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea0005090 con 0x7f7ec00a0810 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.986+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7eb4003ec0 con 0x7f7ec00a0810 2024-09-16T05:38:40.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.986+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7f7eb4044ff0 con 0x7f7ec00a0810 2024-09-16T05:38:40.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.986+0000 7f7ebf7fe640 10 monclient: get_auth_request con 0x7f7eac036d40 auth_method 0 2024-09-16T05:38:40.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.990+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7eb4015360 con 0x7f7ec00a0810 2024-09-16T05:38:40.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.990+0000 7f7ea7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:40.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:39.990+0000 7f7ea7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.081+0000 7f1f916ab640 10 monclient: get_monmap_and_config 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.081+0000 7f1f916ab640 10 monclient: build_initial_monmap 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.081+0000 7f1f916ab640 10 monclient: monmap: 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:40.086457+0000 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:40.086457+0000 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:40.086 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:40.087 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:40.087 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 Processor -- start 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- start start 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient: init 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient(hunting): picked mon.noname-c con 0x7f1f8c09ed10 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8c130300 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8c139de0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c09c220 con 0x7f1f8c130300 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c0a1500 con 0x7f1f8c09ed10 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c0a17e0 con 0x7f1f8c139de0 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient(hunting): authenticate will time out at 1181.893463s 2024-09-16T05:38:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f8a7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1f8c130300 0x7f1f8c1386c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58902/0 (socket says 172.21.15.6:58902) 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f8a7fc640 1 -- 172.21.15.6:0/2553568138 learned_addr learned my addr 172.21.15.6:0/2553568138 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f8b7fe640 1 --1- 172.21.15.6:0/2553568138 >> v1:172.21.15.6:6790/0 conn(0x7f1f8c139de0 0x7f1f8c13c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46220/0 (socket says 172.21.15.6:46220) 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3826906231 0 0) 0x7f1f8c09c220 con 0x7f1f8c130300 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient(hunting): my global_id is 5733 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f74003530 con 0x7f1f8c130300 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102119900 0 0) 0x7f1f8c0a17e0 con 0x7f1f8c139de0 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient(hunting): my global_id is 5714 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f8c09c220 con 0x7f1f8c139de0 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3723264693 0 0) 0x7f1f74003530 con 0x7f1f8c130300 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 >> v1:172.21.15.6:6790/0 conn(0x7f1f8c139de0 legacy=0x7f1f8c13c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 >> v1:172.21.15.63:6789/0 conn(0x7f1f8c09ed10 legacy=0x7f1f8c09f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 monclient: found mon.noname-a 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c0754f0 con 0x7f1f8c130300 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:40.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1f80001620 con 0x7f1f8c130300 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 5 monclient: authenticate success, global_id 5733 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: dump: 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.093 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1f800018c0 con 0x7f1f8c130300 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 1 -- 172.21.15.6:0/2553568138 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1f80001cd0 con 0x7f1f8c130300 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 4 set_mon_vals no callback set 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f89ffb640 10 monclient: dump: 2024-09-16T05:38:40.094 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient: shutdown 2024-09-16T05:38:40.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- 172.21.15.6:0/2553568138 >> v1:172.21.15.6:6789/0 conn(0x7f1f8c130300 legacy=0x7f1f8c1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- 172.21.15.6:0/2553568138 shutdown_connections 2024-09-16T05:38:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- 172.21.15.6:0/2553568138 >> 172.21.15.6:0/2553568138 conn(0x7f1f8c0a1f00 msgr2=0x7f1f8c0a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- 172.21.15.6:0/2553568138 shutdown_connections 2024-09-16T05:38:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- 172.21.15.6:0/2553568138 wait complete. 2024-09-16T05:38:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient: build_initial_monmap 2024-09-16T05:38:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient: monmap: 2024-09-16T05:38:40.096 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:40.097 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 Processor -- start 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 1 -- start start 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.085+0000 7f1f916ab640 10 monclient: init 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8c09ed10 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient(hunting): picked mon.noname-c con 0x7f1f8c130300 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8c139de0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c09d210 con 0x7f1f8c139de0 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c1331f0 con 0x7f1f8c09ed10 2024-09-16T05:38:40.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c134980 con 0x7f1f8c130300 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient(hunting): authenticate will time out at 1181.896409s 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f8b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1f8c139de0 0x7f1f8c1cfad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58916/0 (socket says 172.21.15.6:58916) 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f8b7fe640 1 -- 172.21.15.6:0/211150306 learned_addr learned my addr 172.21.15.6:0/211150306 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4218878102 0 0) 0x7f1f84001560 con 0x7f1f8c139de0 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): my global_id is 5736 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f78003530 con 0x7f1f8c139de0 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1863985118 0 0) 0x7f1f8c134980 con 0x7f1f8c130300 2024-09-16T05:38:40.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): my global_id is 5717 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f84001560 con 0x7f1f8c130300 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2071514574 0 0) 0x7f1f78003530 con 0x7f1f8c139de0 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f8c134980 con 0x7f1f8c139de0 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1f84001ea0 con 0x7f1f8c139de0 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): dump: 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.105 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2972880034 0 0) 0x7f1f84001560 con 0x7f1f8c130300 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f78003530 con 0x7f1f8c130300 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1f80001380 con 0x7f1f8c130300 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:40.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): dump: 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2513667862 0 0) 0x7f1f8c1331f0 con 0x7f1f8c09ed10 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): my global_id is 104155 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f84001560 con 0x7f1f8c09ed10 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3630423025 0 0) 0x7f1f8c134980 con 0x7f1f8c139de0 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 >> v1:172.21.15.6:6790/0 conn(0x7f1f8c130300 legacy=0x7f1f8c1cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 >> v1:172.21.15.63:6789/0 conn(0x7f1f8c09ed10 legacy=0x7f1f8c12cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: _finish_hunting 0 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 monclient: found mon.a 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c1346a0 con 0x7f1f8c139de0 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: _finish_auth 0 2024-09-16T05:38:40.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: _check_auth_tickets 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 5 monclient: authenticate success, global_id 5736 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient: _renew_subs 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8c132f70 con 0x7f1f8c139de0 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient: _renew_subs 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1f8c131f20 con 0x7f1f8c139de0 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1f84004e20 con 0x7f1f8c139de0 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1f84005170 con 0x7f1f8c139de0 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f897fa640 4 set_mon_vals no callback set 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: dump: 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1f6c0065d0 con 0x7f1f6c002870 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1f8400ff10 con 0x7f1f8c139de0 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7f1f84045580 con 0x7f1f8c139de0 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845284493 0 0) 0x7f1f6c0065d0 con 0x7f1f6c002870 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient(hunting): my global_id is 104158 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f8c134980 con 0x7f1f6c002870 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f8a7fc640 10 monclient: get_auth_request con 0x7f1f78036ce0 auth_method 0 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1357845943 0 0) 0x7f1f8c134980 con 0x7f1f6c002870 2024-09-16T05:38:40.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f6c0065d0 con 0x7f1f6c002870 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1f7c001720 con 0x7f1f6c002870 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 10 monclient: dump: 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.112 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 506614578 0 0) 0x7f1f6c0065d0 con 0x7f1f6c002870 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.089+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f6c00a160 con 0x7f1f6c002870 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.093+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1f7c0019c0 con 0x7f1f6c002870 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.093+0000 7f1f737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.093+0000 7f1f737fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.093+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 >> v1:172.21.15.63:6789/0 conn(0x7f1f6c002870 legacy=0x7f1f6c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.093+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 >> v1:172.21.15.63:6789/0 conn(0x7f1f6c002870 legacy=0x7f1f6c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f916ab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1f6c009e10 con 0x7f1f6c0090f0 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 151251971 0 0) 0x7f1f6c009e10 con 0x7f1f6c0090f0 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 10 monclient(hunting): my global_id is 104161 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f6c0065d0 con 0x7f1f6c0090f0 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3038893192 0 0) 0x7f1f6c0065d0 con 0x7f1f6c0090f0 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f6c009e10 con 0x7f1f6c0090f0 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1f7c0037a0 con 0x7f1f6c0090f0 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 10 monclient: dump: 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.114 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1778451811 0 0) 0x7f1f6c009e10 con 0x7f1f6c0090f0 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.097+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1f6c00c390 con 0x7f1f6c0090f0 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3960 (unknown 0 0 1498502447) 0x7f1f7c004650 con 0x7f1f6c0090f0 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f737fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 >> v1:172.21.15.63:6789/0 conn(0x7f1f6c0090f0 legacy=0x7f1f6c009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f737fe640 1 -- 172.21.15.6:0/211150306 >> v1:172.21.15.63:6789/0 conn(0x7f1f6c0090f0 legacy=0x7f1f6c009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 20, 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 3063, 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:38:40.116 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:40.117 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:40.118 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:38:40.119 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:38:40.120 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:38:40.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:40.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:40.123 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:40.124 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:40.125 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:40.126 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:38:40.127 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1f78036ce0 legacy=0x7f1f780391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f916ab640 10 monclient: shutdown 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f916ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 >> v1:172.21.15.6:6789/0 conn(0x7f1f8c139de0 legacy=0x7f1f8c1cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f8b7fe640 1 -- 172.21.15.6:0/211150306 reap_dead start 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f8b7fe640 1 -- 172.21.15.6:0/211150306 reap_dead start 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 shutdown_connections 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 >> 172.21.15.6:0/211150306 conn(0x7f1f8c0a1f00 msgr2=0x7f1f8c12e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 shutdown_connections 2024-09-16T05:38:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.101+0000 7f1f916ab640 1 -- 172.21.15.6:0/211150306 wait complete. 2024-09-16T05:38:40.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:38:40.130 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:38:40.130 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:38:40.130 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:38:40.130 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:38:40.130 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:38:40.131 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebeffd640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.63:6789/0 conn(0x7f7ec00a0810 legacy=0x7f7ec01cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebeffd640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.63:6789/0 conn(0x7f7ec00a0810 legacy=0x7f7ec01cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebeffd640 1 --1- 172.21.15.63:0/2291547042 >> v1:172.21.15.63:6789/0 conn(0x7f7ec00a0810 0x7f7ec01cbe40 :-1 s=OPENED pgs=43 cs=1 l=1).handle_message read tag failed 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebeffd640 1 --1- 172.21.15.63:0/2291547042 >> v1:172.21.15.63:6789/0 conn(0x7f7ec00a0810 0x7f7ec01cbe40 :-1 s=OPENED pgs=43 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.63:6789/0 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.63:6789/0 conn(0x7f7ec00a0810 legacy=0x7f7ec01cbe40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): picked mon.c con 0x7f7ec013ccf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): picked mon.b con 0x7f7eac044fd0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:40.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): picked mon.a con 0x7f7ec006dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7ec00a4090 con 0x7f7ec006dfa0 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebffff640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.63:6789/0 conn(0x7f7eac044fd0 legacy=0x7f7eac047460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7ec011c1b0 con 0x7f7eac044fd0 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7eac04ec60 con 0x7f7ec013ccf0 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354546932 0 0) 0x7f7ec00a4090 con 0x7f7ec006dfa0 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f7eac04ee30 con 0x7f7ec006dfa0 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebf7fe640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.6:6789/0 conn(0x7f7ec006dfa0 legacy=0x7f7eac04b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebf7fe640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.6:6789/0 conn(0x7f7ec006dfa0 legacy=0x7f7eac04b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:38:40.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebf7fe640 1 --1- 172.21.15.63:0/2291547042 >> v1:172.21.15.6:6789/0 conn(0x7f7ec006dfa0 0x7f7eac04b0a0 :-1 s=OPENED pgs=630 cs=1 l=1).handle_message read tag failed 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebf7fe640 1 --1- 172.21.15.63:0/2291547042 >> v1:172.21.15.6:6789/0 conn(0x7f7ec006dfa0 0x7f7eac04b0a0 :-1 s=OPENED pgs=630 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 235826143 0 0) 0x7f7eac04ec60 con 0x7f7ec013ccf0 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f7ec00a4090 con 0x7f7ec013ccf0 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 2637133886 0 0) 0x7f7ec00a4090 con 0x7f7ec013ccf0 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.63:6789/0 conn(0x7f7eac044fd0 legacy=0x7f7eac047460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.6:6789/0 conn(0x7f7ec006dfa0 legacy=0x7f7eac04b0a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 monclient: found mon.c 2024-09-16T05:38:40.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7f7eac04fe60 con 0x7f7ec013ccf0 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: _finish_auth 0 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7eb4012e00 con 0x7f7ec013ccf0 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: dump: 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.139 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7eb40147c0 con 0x7f7ec013ccf0 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ea7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.130+0000 7f7ebdffb640 4 set_mon_vals no callback set 2024-09-16T05:38:40.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.134+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7eb4041ed0 con 0x7f7ec013ccf0 2024-09-16T05:38:40.148 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:38:40.148 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:38:40.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:38:40.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.162+0000 7f7ec5b92640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T05:38:40.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.162+0000 7f7ec5b92640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T05:38:40.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.162+0000 7f7ec5b92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:40.166 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:40.162+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) -- 0x7f7ea0005380 con 0x7f7ec013ccf0 2024-09-16T05:38:40.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient: get_monmap_and_config 2024-09-16T05:38:40.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient: build_initial_monmap 2024-09-16T05:38:40.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient: monmap: 2024-09-16T05:38:40.255 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:40.255 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:40.255620+0000 2024-09-16T05:38:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:40.255620+0000 2024-09-16T05:38:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:40.256 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 1 Processor -- start 2024-09-16T05:38:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 1 -- start start 2024-09-16T05:38:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient: init 2024-09-16T05:38:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:38:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient(hunting): picked mon.noname-a con 0x7f952c09de60 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient(hunting): picked mon.noname-c con 0x7f952c09e240 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:40.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient(hunting): picked mon.noname-b con 0x7f952c139460 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c09d2f0 con 0x7f952c09de60 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c13df80 con 0x7f952c09e240 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c13f160 con 0x7f952c139460 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient(hunting): authenticate will time out at 1182.062603s 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9532388640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f952c09de60 0x7f952c0a0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58926/0 (socket says 172.21.15.6:58926) 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9532388640 1 -- 172.21.15.6:0/333859832 learned_addr learned my addr 172.21.15.6:0/333859832 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9532b89640 1 --1- 172.21.15.6:0/333859832 >> v1:172.21.15.6:6790/0 conn(0x7f952c139460 0x7f952c13b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46234/0 (socket says 172.21.15.6:46234) 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531b87640 1 -- 172.21.15.6:0/333859832 >> v1:172.21.15.63:6789/0 conn(0x7f952c09e240 legacy=0x7f952c0a0c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3121647714 0 0) 0x7f952c09d2f0 con 0x7f952c09de60 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient(hunting): my global_id is 5739 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9514003530 con 0x7f952c09de60 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3518557583 0 0) 0x7f952c13f160 con 0x7f952c139460 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient(hunting): my global_id is 5720 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f952c09d2f0 con 0x7f952c139460 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2858528880 0 0) 0x7f9514003530 con 0x7f952c09de60 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 >> v1:172.21.15.6:6790/0 conn(0x7f952c139460 legacy=0x7f952c13b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 >> v1:172.21.15.63:6789/0 conn(0x7f952c09e240 legacy=0x7f952c0a0c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: _finish_hunting 0 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 monclient: found mon.noname-a 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c140340 con 0x7f952c09de60 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: _finish_auth 0 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: _check_auth_tickets 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f95240019a0 con 0x7f952c09de60 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: dump: 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.262 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 5 monclient: authenticate success, global_id 5739 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9524001bd0 con 0x7f952c09de60 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 1 -- 172.21.15.6:0/333859832 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9524001fe0 con 0x7f952c09de60 2024-09-16T05:38:40.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 4 set_mon_vals no callback set 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f9531386640 10 monclient: dump: 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.264 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 10 monclient: shutdown 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.253+0000 7f95345ec640 1 -- 172.21.15.6:0/333859832 >> v1:172.21.15.6:6789/0 conn(0x7f952c09de60 legacy=0x7f952c0a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- 172.21.15.6:0/333859832 shutdown_connections 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- 172.21.15.6:0/333859832 >> 172.21.15.6:0/333859832 conn(0x7f952c0a1650 msgr2=0x7f952c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- 172.21.15.6:0/333859832 shutdown_connections 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- 172.21.15.6:0/333859832 wait complete. 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: build_initial_monmap 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: monmap: 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:40.265 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 Processor -- start 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- start start 2024-09-16T05:38:40.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: init 2024-09-16T05:38:40.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient(hunting): picked mon.noname-b con 0x7f952c09de60 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient(hunting): picked mon.noname-c con 0x7f952c09e240 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient(hunting): picked mon.noname-a con 0x7f952c139460 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c11d800 con 0x7f952c139460 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c1358b0 con 0x7f952c09de60 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f952c0a0760 con 0x7f952c09e240 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient(hunting): authenticate will time out at 1182.065450s 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f9532388640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f952c09de60 legacy=0x7f952c134a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f9532b89640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f952c139460 0x7f952c1d36f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58932/0 (socket says 172.21.15.6:58932) 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f9532b89640 1 -- 172.21.15.6:0/2894518388 learned_addr learned my addr 172.21.15.6:0/2894518388 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f9531b87640 1 --1- 172.21.15.6:0/2894518388 >> v1:172.21.15.6:6790/0 conn(0x7f952c09e240 0x7f952c1351a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46248/0 (socket says 172.21.15.6:46248) 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 50242384 0 0) 0x7f952c11d800 con 0x7f952c139460 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): my global_id is 5742 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9518003530 con 0x7f952c139460 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 933811046 0 0) 0x7f952c0a0760 con 0x7f952c09e240 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): my global_id is 5723 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f952c11d800 con 0x7f952c09e240 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3163522804 0 0) 0x7f9518003530 con 0x7f952c139460 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f952c0a0760 con 0x7f952c139460 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f951c0012a0 con 0x7f952c139460 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): dump: 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3366802962 0 0) 0x7f952c11d800 con 0x7f952c09e240 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9518003530 con 0x7f952c09e240 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9520004080 con 0x7f952c09e240 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient(hunting): dump: 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3059036777 0 0) 0x7f952c0a0760 con 0x7f952c139460 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 >> v1:172.21.15.6:6790/0 conn(0x7f952c09e240 legacy=0x7f952c1351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 >> v1:172.21.15.63:6789/0 conn(0x7f952c09de60 legacy=0x7f952c134a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 monclient: found mon.a 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c0a47c0 con 0x7f952c139460 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient: _finish_auth 0 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 5 monclient: authenticate success, global_id 5742 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: _renew_subs 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f952c11d520 con 0x7f952c139460 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f951c0047f0 con 0x7f952c139460 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f951c004b40 con 0x7f952c139460 2024-09-16T05:38:40.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: _renew_subs 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f952c13dc50 con 0x7f952c139460 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f9530b85640 4 set_mon_vals no callback set 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 10 monclient: dump: 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f952c11dd20 con 0x7f952c139460 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f951c00f8e0 con 0x7f952c139460 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.257+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7f951c044fe0 con 0x7f952c139460 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.261+0000 7f9532388640 10 monclient: get_auth_request con 0x7f9518036ac0 auth_method 0 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.265+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f951c0152f0 con 0x7f952c139460 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.265+0000 7f952affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.265+0000 7f952affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.485+0000 7f95345ec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:38:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.485+0000 7f95345ec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.485+0000 7f95345ec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.485+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f952c06bc10 con 0x7f952c139460 2024-09-16T05:38:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.485+0000 7f952affd640 1 -- 172.21.15.6:0/2894518388 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1681999032) 0x7f951c018fa0 con 0x7f952c139460 2024-09-16T05:38:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.485+0000 7f952affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:40.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.485+0000 7f952affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:40.491 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:38:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.489+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9518036ac0 legacy=0x7f9518038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.489+0000 7f95345ec640 10 monclient: shutdown 2024-09-16T05:38:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.489+0000 7f95345ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.489+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 >> v1:172.21.15.6:6789/0 conn(0x7f952c139460 legacy=0x7f952c1d36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.489+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 shutdown_connections 2024-09-16T05:38:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.489+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 >> 172.21.15.6:0/2894518388 conn(0x7f952c0a1650 msgr2=0x7f952c0a3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:40.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.489+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 shutdown_connections 2024-09-16T05:38:40.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:40.489+0000 7f95345ec640 1 -- 172.21.15.6:0/2894518388 wait complete. 2024-09-16T05:38:42.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:42.982+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:42.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:42.982+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:42.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:42.982+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:42.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:42.982+0000 7f7ea6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:43.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:38:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient: get_monmap_and_config 2024-09-16T05:38:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient: build_initial_monmap 2024-09-16T05:38:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient: monmap: 2024-09-16T05:38:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:43.640566+0000 2024-09-16T05:38:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:43.640566+0000 2024-09-16T05:38:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:43.641 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 1 Processor -- start 2024-09-16T05:38:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 1 -- start start 2024-09-16T05:38:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient: init 2024-09-16T05:38:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient(hunting): picked mon.noname-b con 0x7f4d880a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient(hunting): picked mon.noname-c con 0x7f4d880a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient(hunting): picked mon.noname-a con 0x7f4d880a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d880a33a0 con 0x7f4d880a2980 2024-09-16T05:38:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d880a07a0 con 0x7f4d880a2570 2024-09-16T05:38:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d880a2d90 con 0x7f4d880a2190 2024-09-16T05:38:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8f66d640 10 monclient(hunting): authenticate will time out at 1185.447787s 2024-09-16T05:38:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8d409640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4d880a2190 0x7f4d880a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46260/0 (socket says 172.21.15.6:46260) 2024-09-16T05:38:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8d409640 1 -- 172.21.15.6:0/96280528 learned_addr learned my addr 172.21.15.6:0/96280528 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.637+0000 7f4d8cc08640 1 -- 172.21.15.6:0/96280528 >> v1:172.21.15.63:6789/0 conn(0x7f4d880a2570 legacy=0x7f4d8813b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 381345963 0 0) 0x7f4d880a33a0 con 0x7f4d880a2980 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient(hunting): my global_id is 5745 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d74003530 con 0x7f4d880a2980 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1492548433 0 0) 0x7f4d74003530 con 0x7f4d880a2980 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 >> v1:172.21.15.6:6790/0 conn(0x7f4d880a2190 legacy=0x7f4d880a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 >> v1:172.21.15.63:6789/0 conn(0x7f4d880a2570 legacy=0x7f4d8813b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 monclient: found mon.noname-a 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d880a66e0 con 0x7f4d880a2980 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: _finish_auth 0 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 5 monclient: authenticate success, global_id 5745 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d800018d0 con 0x7f4d880a2980 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: dump: 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d80001c30 con 0x7f4d880a2980 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 1 -- 172.21.15.6:0/96280528 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d80002000 con 0x7f4d880a2980 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 4 set_mon_vals no callback set 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7ffff640 10 monclient: dump: 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:43.649 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:43.649 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:43.649 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:43.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: shutdown 2024-09-16T05:38:43.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:43.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- 172.21.15.6:0/96280528 >> v1:172.21.15.6:6789/0 conn(0x7f4d880a2980 legacy=0x7f4d8813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:43.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- 172.21.15.6:0/96280528 shutdown_connections 2024-09-16T05:38:43.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- 172.21.15.6:0/96280528 >> 172.21.15.6:0/96280528 conn(0x7f4d880a9850 msgr2=0x7f4d880a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- 172.21.15.6:0/96280528 shutdown_connections 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- 172.21.15.6:0/96280528 wait complete. 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: build_initial_monmap 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: monmap: 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:43.653 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 Processor -- start 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- start start 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: init 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient(hunting): picked mon.noname-a con 0x7f4d880a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient(hunting): picked mon.noname-c con 0x7f4d880a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient(hunting): picked mon.noname-b con 0x7f4d880a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d88122d30 con 0x7f4d880a2190 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d88121600 con 0x7f4d880a2980 2024-09-16T05:38:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d880a7fb0 con 0x7f4d880a2570 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient(hunting): authenticate will time out at 1185.450837s 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8cc08640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4d880a2570 0x7f4d88120ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46270/0 (socket says 172.21.15.6:46270) 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8cc08640 1 -- 172.21.15.6:0/3489537659 learned_addr learned my addr 172.21.15.6:0/3489537659 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8dc0a640 1 -- 172.21.15.6:0/3489537659 >> v1:172.21.15.63:6789/0 conn(0x7f4d880a2980 legacy=0x7f4d881d3ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2210618045 0 0) 0x7f4d88122d30 con 0x7f4d880a2190 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient(hunting): my global_id is 5748 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d6c003530 con 0x7f4d880a2190 2024-09-16T05:38:43.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1924224110 0 0) 0x7f4d6c003530 con 0x7f4d880a2190 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d88122d30 con 0x7f4d880a2190 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d840017f0 con 0x7f4d880a2190 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient(hunting): dump: 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:43.656 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2313077631 0 0) 0x7f4d88122d30 con 0x7f4d880a2190 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 >> v1:172.21.15.6:6790/0 conn(0x7f4d880a2570 legacy=0x7f4d88120ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 >> v1:172.21.15.63:6789/0 conn(0x7f4d880a2980 legacy=0x7f4d881d3ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:43.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 monclient: found mon.a 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d880a0100 con 0x7f4d880a2190 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 5 monclient: authenticate success, global_id 5748 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: _renew_subs 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4d88123f00 con 0x7f4d880a2190 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: _renew_subs 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.641+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4d881248e0 con 0x7f4d880a2190 2024-09-16T05:38:43.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d84001d20 con 0x7f4d880a2190 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7effd640 4 set_mon_vals no callback set 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d84004990 con 0x7f4d880a2190 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7dffb640 10 monclient: dump: 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:43.659 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4d84004bb0 con 0x7f4d880a2190 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d8f66d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d8f66d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d8f66d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d68005090 con 0x7f4d880a2190 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7f4d84045060 con 0x7f4d880a2190 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.645+0000 7f4d8cc08640 10 monclient: get_auth_request con 0x7f4d6c036a40 auth_method 0 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.649+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4d8401d8c0 con 0x7f4d880a2190 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.649+0000 7f4d7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:43.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.649+0000 7f4d7dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:43.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.877+0000 7f4d8f66d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:38:43.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.877+0000 7f4d8f66d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:43.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.877+0000 7f4d8f66d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:43.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.877+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4d68005380 con 0x7f4d880a2190 2024-09-16T05:38:43.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.877+0000 7f4d7dffb640 1 -- 172.21.15.6:0/3489537659 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1763174975) 0x7f4d84019000 con 0x7f4d880a2190 2024-09-16T05:38:43.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.877+0000 7f4d7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:43.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.877+0000 7f4d7dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:43.882 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:38:43.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.881+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4d6c036a40 legacy=0x7f4d6c038f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:43.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.881+0000 7f4d8f66d640 10 monclient: shutdown 2024-09-16T05:38:43.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.881+0000 7f4d8f66d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:43.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.881+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 >> v1:172.21.15.6:6789/0 conn(0x7f4d880a2190 legacy=0x7f4d881207e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:43.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.881+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 shutdown_connections 2024-09-16T05:38:43.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.881+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 >> 172.21.15.6:0/3489537659 conn(0x7f4d880a9850 msgr2=0x7f4d8812f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:43.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.881+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 shutdown_connections 2024-09-16T05:38:43.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:43.881+0000 7f4d8f66d640 1 -- 172.21.15.6:0/3489537659 wait complete. 2024-09-16T05:38:43.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:43.982+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:43.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:43.982+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:43.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:43.982+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:44.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:44.982+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:44.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:44.982+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:44.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:44.982+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:45.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:45.982+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:45.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:45.982+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:45.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:45.982+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:46.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:38:46.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:46.986+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:46.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:46.986+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:46.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:46.986+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient: get_monmap_and_config 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient: build_initial_monmap 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient: monmap: 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:47.033274+0000 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:47.033274+0000 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:47.033 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:47.034 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:47.034 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 1 Processor -- start 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 1 -- start start 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient: init 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient(hunting): picked mon.noname-a con 0x7f56a40a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient(hunting): picked mon.noname-b con 0x7f56a40a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient(hunting): picked mon.noname-c con 0x7f56a40a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a40a33a0 con 0x7f56a40a2190 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a40a07a0 con 0x7f56a40a2980 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a40a2d90 con 0x7f56a40a2570 2024-09-16T05:38:47.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:47.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.029+0000 7f56ab881640 10 monclient(hunting): authenticate will time out at 1188.839873s 2024-09-16T05:38:47.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.033+0000 7f56a8e1c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f56a40a2570 0x7f56a413b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46278/0 (socket says 172.21.15.6:46278) 2024-09-16T05:38:47.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.033+0000 7f56a961d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f56a40a2190 0x7f56a40a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58962/0 (socket says 172.21.15.6:58962) 2024-09-16T05:38:47.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.033+0000 7f56a9e1e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f56a413f120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:47.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.033+0000 7f56a8e1c640 1 -- 172.21.15.6:0/3041821434 learned_addr learned my addr 172.21.15.6:0/3041821434 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:47.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.237+0000 7f56a9e1e640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f56a413f120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:47.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:47.641+0000 7f56a9e1e640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f56a413f120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:47.986+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:47.986+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:47.986+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:48.445+0000 7f56a9e1e640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f56a413f120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:48.986+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:48.986+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:48.986+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:49.982+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:49.982+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:49.982+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.029+0000 7f56a2ffd640 10 monclient(hunting): tick 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.029+0000 7f56a2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.029+0000 7f56a2ffd640 1 monclient(hunting): continuing hunt 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.029+0000 7f56a2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.029+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.6:6790/0 conn(0x7f56a40a2570 legacy=0x7f56a413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.029+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f56a413f120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.029+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.6:6789/0 conn(0x7f56a40a2190 legacy=0x7f56a40a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:50.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f56900036d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:50.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f5690007360 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:50.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f56a40a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:50.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a40a66e0 con 0x7f5690007360 2024-09-16T05:38:50.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a40a07a0 con 0x7f56a40a2980 2024-09-16T05:38:50.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a9e1e640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f569000d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:50.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56900108b0 con 0x7f56900036d0 2024-09-16T05:38:50.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.033+0000 7f56a2ffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.237+0000 7f56a9e1e640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f569000d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:50.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:50.641+0000 7f56a9e1e640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f569000d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:50.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:50.982+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:50.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:50.982+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:50.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:50.982+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:51.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:51.445+0000 7f56a9e1e640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f569000d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:51.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:51.986+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:51.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:51.986+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:51.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:51.986+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:52.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:52.986+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:52.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:52.986+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:52.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:52.986+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:52.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:52.986+0000 7f7ea6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:53.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 10 monclient(hunting): tick 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 1 monclient(hunting): continuing hunt 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.6:6790/0 conn(0x7f56900036d0 legacy=0x7f5690005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f56a40a2980 legacy=0x7f569000d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.6:6789/0 conn(0x7f5690007360 legacy=0x7f56900097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f5690010b80 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f5690012e40 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f5690016950 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:53.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5690011b00 con 0x7f5690016950 2024-09-16T05:38:53.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a40a07a0 con 0x7f5690010b80 2024-09-16T05:38:53.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 1 -- 172.21.15.6:0/3041821434 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f569000d830 con 0x7f5690012e40 2024-09-16T05:38:53.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a2ffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:53.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.033+0000 7f56a961d640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f5690010b80 legacy=0x7f5690010fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:53.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.237+0000 7f56a961d640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f5690010b80 legacy=0x7f5690010fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:53.641+0000 7f56a961d640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f5690010b80 legacy=0x7f5690010fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:53.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:53.986+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:53.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:53.986+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:53.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:53.986+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.445+0000 7f56a961d640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f5690010b80 legacy=0x7f5690010fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:54.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1781081256 0 0) 0x7f569000d830 con 0x7f5690012e40 2024-09-16T05:38:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient(hunting): my global_id is 5726 2024-09-16T05:38:54.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f568c003530 con 0x7f5690012e40 2024-09-16T05:38:54.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2634997411 0 0) 0x7f568c003530 con 0x7f5690012e40 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.63:6789/0 conn(0x7f5690010b80 legacy=0x7f5690010fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.6:6789/0 conn(0x7f5690016950 legacy=0x7f5690018d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 monclient: found mon.noname-b 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5690009920 con 0x7f5690012e40 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: _finish_auth 0 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5698001520 con 0x7f5690012e40 2024-09-16T05:38:54.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: dump: 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:54.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56ab881640 5 monclient: authenticate success, global_id 5726 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56ab881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5698001780 con 0x7f5690012e40 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56ab881640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 1 -- 172.21.15.6:0/3041821434 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5698001bd0 con 0x7f5690012e40 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56ab881640 4 set_mon_vals no callback set 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.749+0000 7f56a3fff640 10 monclient: dump: 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:54.757 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 10 monclient: shutdown 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 1 -- 172.21.15.6:0/3041821434 >> v1:172.21.15.6:6790/0 conn(0x7f5690012e40 legacy=0x7f5690015230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 1 -- 172.21.15.6:0/3041821434 shutdown_connections 2024-09-16T05:38:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 1 -- 172.21.15.6:0/3041821434 >> 172.21.15.6:0/3041821434 conn(0x7f56a40a9850 msgr2=0x7f56a40a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:54.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 1 -- 172.21.15.6:0/3041821434 shutdown_connections 2024-09-16T05:38:54.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 1 -- 172.21.15.6:0/3041821434 wait complete. 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 10 monclient: build_initial_monmap 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 10 monclient: monmap: 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:54.762 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 1 Processor -- start 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 1 -- start start 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.753+0000 7f56ab881640 10 monclient: init 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient(hunting): picked mon.noname-c con 0x7f5690010b80 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient(hunting): picked mon.noname-a con 0x7f5690012e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient(hunting): picked mon.noname-b con 0x7f5690016950 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a4127dd0 con 0x7f5690012e40 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a4126920 con 0x7f5690016950 2024-09-16T05:38:54.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56a41280b0 con 0x7f5690010b80 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient(hunting): authenticate will time out at 1196.564549s 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a961d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5690010b80 0x7f56a41cbb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:36046/0 (socket says 172.21.15.6:36046) 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a961d640 1 -- 172.21.15.6:0/1401201781 learned_addr learned my addr 172.21.15.6:0/1401201781 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a9e1e640 1 -- 172.21.15.6:0/1401201781 >> v1:172.21.15.63:6789/0 conn(0x7f5690016950 legacy=0x7f56a41d3f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104638747 0 0) 0x7f56a41280b0 con 0x7f5690010b80 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient(hunting): my global_id is 5729 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5690009b30 con 0x7f5690010b80 2024-09-16T05:38:54.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2938844606 0 0) 0x7f5690009b30 con 0x7f5690010b80 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56a41280b0 con 0x7f5690010b80 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5694001810 con 0x7f5690010b80 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient(hunting): dump: 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:54.765 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2962176057 0 0) 0x7f56a41280b0 con 0x7f5690010b80 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 >> v1:172.21.15.63:6789/0 conn(0x7f5690016950 legacy=0x7f56a41d3f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 >> v1:172.21.15.6:6789/0 conn(0x7f5690012e40 legacy=0x7f56a41d0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 monclient: found mon.c 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a40a8850 con 0x7f5690010b80 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:54.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 5 monclient: authenticate success, global_id 5729 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient: _renew_subs 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56a4127af0 con 0x7f5690010b80 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient: _renew_subs 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f56a41251b0 con 0x7f5690010b80 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56940047f0 con 0x7f5690010b80 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5694004b40 con 0x7f5690010b80 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:54.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 10 monclient: dump: 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a37fe640 4 set_mon_vals no callback set 2024-09-16T05:38:54.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:54.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:54.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:54.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5684005090 con 0x7f5690010b80 2024-09-16T05:38:54.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5694004d60 con 0x7f5690010b80 2024-09-16T05:38:54.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.757+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7f5694044f40 con 0x7f5690010b80 2024-09-16T05:38:54.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.761+0000 7f56a8e1c640 10 monclient: get_auth_request con 0x7f5690038a00 auth_method 0 2024-09-16T05:38:54.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.765+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5694015370 con 0x7f5690010b80 2024-09-16T05:38:54.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.765+0000 7f56a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:54.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:54.765+0000 7f56a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:54.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:54.982+0000 7f7ea6ffd640 10 monclient: tick 2024-09-16T05:38:54.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:54.982+0000 7f7ea6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:54.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:54.982+0000 7f7ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.001+0000 7f56ab881640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:38:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.001+0000 7f56ab881640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.001+0000 7f56ab881640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.001+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5684005380 con 0x7f5690010b80 2024-09-16T05:38:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.001+0000 7f56a1ffb640 1 -- 172.21.15.6:0/1401201781 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 3809501156) 0x7f5694019020 con 0x7f5690010b80 2024-09-16T05:38:55.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.001+0000 7f56a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:38:55.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.001+0000 7f56a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:55.005 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:38:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.005+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5690038a00 legacy=0x7f569003aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.005+0000 7f56ab881640 10 monclient: shutdown 2024-09-16T05:38:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.005+0000 7f56ab881640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.005+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 >> v1:172.21.15.6:6790/0 conn(0x7f5690010b80 legacy=0x7f56a41cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.005+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 shutdown_connections 2024-09-16T05:38:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.005+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 >> 172.21.15.6:0/1401201781 conn(0x7f56a40a9850 msgr2=0x7f56a409e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.005+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 shutdown_connections 2024-09-16T05:38:55.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.005+0000 7f56ab881640 1 -- 172.21.15.6:0/1401201781 wait complete. 2024-09-16T05:38:55.024 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:38:55.024 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:38:55.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:38:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient: get_monmap_and_config 2024-09-16T05:38:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient: build_initial_monmap 2024-09-16T05:38:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient: monmap: 2024-09-16T05:38:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.147 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:55.147602+0000 2024-09-16T05:38:55.148 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:55.147602+0000 2024-09-16T05:38:55.148 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:55.148 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.148 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:55.148 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:55.148 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:55.148 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 Processor -- start 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- start start 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient: init 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient(hunting): picked mon.noname-c con 0x7f70a809aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient(hunting): picked mon.noname-b con 0x7f70a809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient(hunting): picked mon.noname-a con 0x7f70a80a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a80a1370 con 0x7f70a80a0da0 2024-09-16T05:38:55.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a809ead0 con 0x7f70a809aa30 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a809edb0 con 0x7f70a809e230 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient(hunting): authenticate will time out at 1196.954204s 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70ad34f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f70a809aa30 legacy=0x7f70a809ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70adb50640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70a80a0da0 0x7f70a813d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58224/0 (socket says 172.21.15.6:58224) 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70acb4e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f70a809e230 0x7f70a81399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:36056/0 (socket says 172.21.15.6:36056) 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70adb50640 1 -- 172.21.15.6:0/1225660655 learned_addr learned my addr 172.21.15.6:0/1225660655 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1774301978 0 0) 0x7f70a809edb0 con 0x7f70a809e230 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient(hunting): my global_id is 5732 2024-09-16T05:38:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7094003530 con 0x7f70a809e230 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011081321 0 0) 0x7f70a80a1370 con 0x7f70a80a0da0 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient(hunting): my global_id is 5751 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70a809edb0 con 0x7f70a80a0da0 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3419047583 0 0) 0x7f7094003530 con 0x7f70a809e230 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 >> v1:172.21.15.63:6789/0 conn(0x7f70a809aa30 legacy=0x7f70a809ae10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 >> v1:172.21.15.6:6789/0 conn(0x7f70a80a0da0 legacy=0x7f70a813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 monclient: found mon.noname-b 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a809dd50 con 0x7f70a809e230 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: _finish_auth 0 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7098001620 con 0x7f70a809e230 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: dump: 2024-09-16T05:38:55.159 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 5 monclient: authenticate success, global_id 5732 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:55.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70980018d0 con 0x7f70a809e230 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 1 -- 172.21.15.6:0/1225660655 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7098001ce0 con 0x7f70a809e230 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f709ffff640 10 monclient: dump: 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.161 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 4 set_mon_vals no callback set 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 10 monclient: shutdown 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- 172.21.15.6:0/1225660655 >> v1:172.21.15.6:6790/0 conn(0x7f70a809e230 legacy=0x7f70a81399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- 172.21.15.6:0/1225660655 shutdown_connections 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- 172.21.15.6:0/1225660655 >> 172.21.15.6:0/1225660655 conn(0x7f70a80a1650 msgr2=0x7f70a80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- 172.21.15.6:0/1225660655 shutdown_connections 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.145+0000 7f70af5b3640 1 -- 172.21.15.6:0/1225660655 wait complete. 2024-09-16T05:38:55.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: build_initial_monmap 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: monmap: 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 1 Processor -- start 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 1 -- start start 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: init 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient(hunting): picked mon.noname-c con 0x7f70a809aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient(hunting): picked mon.noname-a con 0x7f70a809e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient(hunting): picked mon.noname-b con 0x7f70a80a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a8139fa0 con 0x7f70a809e230 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a809be70 con 0x7f70a80a0da0 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a80a0290 con 0x7f70a809aa30 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient(hunting): authenticate will time out at 1196.957365s 2024-09-16T05:38:55.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70acb4e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70a809e230 0x7f70a8134160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58240/0 (socket says 172.21.15.6:58240) 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70acb4e640 1 -- 172.21.15.6:0/3316951603 learned_addr learned my addr 172.21.15.6:0/3316951603 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70adb50640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.63:6789/0 conn(0x7f70a80a0da0 legacy=0x7f70a8134870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031276412 0 0) 0x7f70a8139fa0 con 0x7f70a809e230 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): my global_id is 5754 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f708c003530 con 0x7f70a809e230 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 229158795 0 0) 0x7f70a80a0290 con 0x7f70a809aa30 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): my global_id is 5735 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70a8139fa0 con 0x7f70a809aa30 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2701165922 0 0) 0x7f708c003530 con 0x7f70a809e230 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70a80a0290 con 0x7f70a809e230 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7098001c20 con 0x7f70a809e230 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): dump: 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2693242942 0 0) 0x7f70a8139fa0 con 0x7f70a809aa30 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f708c003530 con 0x7f70a809aa30 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70a4004100 con 0x7f70a809aa30 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): dump: 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3421905906 0 0) 0x7f70a80a0290 con 0x7f70a809e230 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.6:6790/0 conn(0x7f70a809aa30 legacy=0x7f70a8133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.63:6789/0 conn(0x7f70a80a0da0 legacy=0x7f70a8134870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 monclient: found mon.a 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a80740c0 con 0x7f70a809e230 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: _finish_auth 0 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 5 monclient: authenticate success, global_id 5754 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: _renew_subs 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70a809d940 con 0x7f70a809e230 2024-09-16T05:38:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: _renew_subs 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f70a806e7d0 con 0x7f70a809e230 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70980018d0 con 0x7f70a809e230 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7098004d20 con 0x7f70a809e230 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709f7fe640 4 set_mon_vals no callback set 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: dump: 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f70880065d0 con 0x7f7088002870 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f709800fac0 con 0x7f70a809e230 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7f7098045080 con 0x7f70a809e230 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940554852 0 0) 0x7f70880065d0 con 0x7f7088002870 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient(hunting): my global_id is 5757 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70a80a0290 con 0x7f7088002870 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2715781749 0 0) 0x7f70a80a0290 con 0x7f7088002870 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70880065d0 con 0x7f7088002870 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70a40047c0 con 0x7f7088002870 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f709dffb640 10 monclient: dump: 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.149+0000 7f70adb50640 10 monclient: get_auth_request con 0x7f708c0369b0 auth_method 0 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.153+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 578643228 0 0) 0x7f70880065d0 con 0x7f7088002870 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.153+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f708800a160 con 0x7f7088002870 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.153+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f70a4004a70 con 0x7f7088002870 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.153+0000 7f709dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.153+0000 7f709dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.153+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.6:6789/0 conn(0x7f7088002870 legacy=0x7f7088004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.153+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.6:6789/0 conn(0x7f7088002870 legacy=0x7f7088004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f70af5b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:38:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7088009e30 con 0x7f70880090f0 2024-09-16T05:38:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2604152774 0 0) 0x7f7088009e30 con 0x7f70880090f0 2024-09-16T05:38:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient(hunting): my global_id is 5760 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70880065d0 con 0x7f70880090f0 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 352285240 0 0) 0x7f70880065d0 con 0x7f70880090f0 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7088009e30 con 0x7f70880090f0 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70a4005070 con 0x7f70880090f0 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient: dump: 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1163703375 0 0) 0x7f7088009e30 con 0x7f70880090f0 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f708800c3a0 con 0x7f70880090f0 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4287 (unknown 0 0 1028719510) 0x7f70a4006480 con 0x7f70880090f0 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:38:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.6:6789/0 conn(0x7f70880090f0 legacy=0x7f7088009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f709dffb640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.6:6789/0 conn(0x7f70880090f0 legacy=0x7f7088009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:55.178 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:38:55.178 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:55.178 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:55.178 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:38:55.178 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 24, 2024-09-16T05:38:55.178 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 264213, 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:55.179 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:55.180 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:38:55.181 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:38:55.182 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:55.183 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:55.184 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:55.185 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:55.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:38:55.187 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:55.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:55.189 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:38:55.190 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.63:6812/3643207211 conn(0x7f708c0369b0 legacy=0x7f708c038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f70af5b3640 10 monclient: shutdown 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f70af5b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 >> v1:172.21.15.6:6789/0 conn(0x7f70a809e230 legacy=0x7f70a8134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.157+0000 7f70adb50640 1 -- 172.21.15.6:0/3316951603 reap_dead start 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.161+0000 7f70adb50640 1 -- 172.21.15.6:0/3316951603 reap_dead start 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.161+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 shutdown_connections 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.161+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 >> 172.21.15.6:0/3316951603 conn(0x7f70a80a1650 msgr2=0x7f70a80a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.161+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 shutdown_connections 2024-09-16T05:38:55.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.161+0000 7f70af5b3640 1 -- 172.21.15.6:0/3316951603 wait complete. 2024-09-16T05:38:55.193 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:38:55.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:38:55.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient: get_monmap_and_config 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient: build_initial_monmap 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient: monmap: 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:38:55.308359+0000 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:38:55.308359+0000 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.308 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:55.309 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:55.309 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:55.309 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 1 Processor -- start 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 1 -- start start 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient: init 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient(hunting): picked mon.noname-c con 0x7f64a40a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient(hunting): picked mon.noname-b con 0x7f64a40a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient(hunting): picked mon.noname-a con 0x7f64a40a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a406e520 con 0x7f64a40a1a00 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a40a0780 con 0x7f64a40a37a0 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a40a4c70 con 0x7f64a40a4290 2024-09-16T05:38:55.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:55.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64aa0d2640 10 monclient(hunting): authenticate will time out at 1197.115667s 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64a37fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f64a40a37a0 legacy=0x7f64a40a3b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64a2ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f64a40a4290 0x7f64a413bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:36070/0 (socket says 172.21.15.6:36070) 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.305+0000 7f64a2ffd640 1 -- 172.21.15.6:0/1953056860 learned_addr learned my addr 172.21.15.6:0/1953056860 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190987263 0 0) 0x7f64a40a4c70 con 0x7f64a40a4290 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient(hunting): my global_id is 5738 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f648c003530 con 0x7f64a40a4290 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2304319895 0 0) 0x7f64a406e520 con 0x7f64a40a1a00 2024-09-16T05:38:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient(hunting): my global_id is 5763 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64a40a4c70 con 0x7f64a40a1a00 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2667112294 0 0) 0x7f64a40a4c70 con 0x7f64a40a1a00 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 >> v1:172.21.15.6:6790/0 conn(0x7f64a40a4290 legacy=0x7f64a413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 >> v1:172.21.15.63:6789/0 conn(0x7f64a40a37a0 legacy=0x7f64a40a3b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 monclient: found mon.noname-a 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a40a65f0 con 0x7f64a40a1a00 2024-09-16T05:38:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: _finish_auth 0 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f649c001600 con 0x7f64a40a1a00 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: dump: 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 5 monclient: authenticate success, global_id 5763 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f649c0018e0 con 0x7f64a40a1a00 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 1 -- 172.21.15.6:0/1953056860 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f649c001cf0 con 0x7f64a40a1a00 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a27fc640 10 monclient: dump: 2024-09-16T05:38:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 4 set_mon_vals no callback set 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient: shutdown 2024-09-16T05:38:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- 172.21.15.6:0/1953056860 >> v1:172.21.15.6:6789/0 conn(0x7f64a40a1a00 legacy=0x7f64a413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- 172.21.15.6:0/1953056860 shutdown_connections 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- 172.21.15.6:0/1953056860 >> 172.21.15.6:0/1953056860 conn(0x7f64a40a58a0 msgr2=0x7f64a40a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- 172.21.15.6:0/1953056860 shutdown_connections 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- 172.21.15.6:0/1953056860 wait complete. 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient: build_initial_monmap 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient: monmap: 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:55.322 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 Processor -- start 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- start start 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient: init 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient(hunting): picked mon.noname-c con 0x7f64a40a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient(hunting): picked mon.noname-a con 0x7f64a40a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient(hunting): picked mon.noname-b con 0x7f64a40a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a4137ea0 con 0x7f64a40a37a0 2024-09-16T05:38:55.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a4136220 con 0x7f64a40a4290 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a40a8100 con 0x7f64a40a1a00 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64aa0d2640 10 monclient(hunting): authenticate will time out at 1197.119010s 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a2ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f64a40a37a0 0x7f64a41343b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58266/0 (socket says 172.21.15.6:58266) 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a2ffd640 1 -- 172.21.15.6:0/354489433 learned_addr learned my addr 172.21.15.6:0/354489433 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f64a3fff640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.63:6789/0 conn(0x7f64a40a4290 legacy=0x7f64a4134ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 605256137 0 0) 0x7f64a40a8100 con 0x7f64a40a1a00 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): my global_id is 5741 2024-09-16T05:38:55.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6490003530 con 0x7f64a40a1a00 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1182716770 0 0) 0x7f64a4137ea0 con 0x7f64a40a37a0 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): my global_id is 5766 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64a40a8100 con 0x7f64a40a37a0 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1184160881 0 0) 0x7f6490003530 con 0x7f64a40a1a00 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64a4137ea0 con 0x7f64a40a1a00 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6494004480 con 0x7f64a40a1a00 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:55.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): dump: 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3325512350 0 0) 0x7f64a40a8100 con 0x7f64a40a37a0 2024-09-16T05:38:55.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6490003530 con 0x7f64a40a37a0 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6498001300 con 0x7f64a40a37a0 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient(hunting): dump: 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.327 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2921652873 0 0) 0x7f64a4137ea0 con 0x7f64a40a1a00 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.63:6789/0 conn(0x7f64a40a4290 legacy=0x7f64a4134ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.6:6789/0 conn(0x7f64a40a37a0 legacy=0x7f64a41343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.309+0000 7f648bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 monclient: found mon.c 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a4137670 con 0x7f64a40a1a00 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: _finish_auth 0 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 5 monclient: authenticate success, global_id 5741 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 10 monclient: _renew_subs 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64a4137390 con 0x7f64a40a1a00 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 10 monclient: _renew_subs 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f64a41d4d60 con 0x7f64a40a1a00 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6494004620 con 0x7f64a40a1a00 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f64940049f0 con 0x7f64a40a1a00 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:55.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: dump: 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64a1ffb640 4 set_mon_vals no callback set 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:55.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f64840065d0 con 0x7f6484002870 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6494004c10 con 0x7f64a40a1a00 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(145..145 src has 1..145) ==== 7467+0+0 (unknown 143292941 0 0) 0x7f6494045260 con 0x7f64a40a1a00 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892009571 0 0) 0x7f64840065d0 con 0x7f6484002870 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient(hunting): my global_id is 5744 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64a4137ea0 con 0x7f6484002870 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2695806370 0 0) 0x7f64a4137ea0 con 0x7f6484002870 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64840065d0 con 0x7f6484002870 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6498004b80 con 0x7f6484002870 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: dump: 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f64a3fff640 10 monclient: get_auth_request con 0x7f6490036a00 auth_method 0 2024-09-16T05:38:55.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2901763431 0 0) 0x7f64840065d0 con 0x7f6484002870 2024-09-16T05:38:55.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f648400a160 con 0x7f6484002870 2024-09-16T05:38:55.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6498004e70 con 0x7f6484002870 2024-09-16T05:38:55.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:55.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:55.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.6:6790/0 conn(0x7f6484002870 legacy=0x7f6484004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.313+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.6:6790/0 conn(0x7f6484002870 legacy=0x7f6484004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:55.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:38:55.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6484009e10 con 0x7f64840090f0 2024-09-16T05:38:55.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2725933447 0 0) 0x7f6484009e10 con 0x7f64840090f0 2024-09-16T05:38:55.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:38:55.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:55.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient(hunting): my global_id is 5747 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64840065d0 con 0x7f64840090f0 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3391681660 0 0) 0x7f64840065d0 con 0x7f64840090f0 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6484009e10 con 0x7f64840090f0 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f64980053f0 con 0x7f64840090f0 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient: dump: 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:55.335 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2751055382 0 0) 0x7f6484009e10 con 0x7f64840090f0 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f648400c390 con 0x7f64840090f0 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4118 (unknown 0 0 3421465088) 0x7f6498006740 con 0x7f64840090f0 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.6:6790/0 conn(0x7f64840090f0 legacy=0x7f6484009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f648bfff640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.6:6790/0 conn(0x7f64840090f0 legacy=0x7f6484009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 24, 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 264377, 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.337 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:55.338 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:55.339 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:38:55.340 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:38:55.341 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:38:55.342 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:55.343 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.344 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:38:55.345 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:38:55.346 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:38:55.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:38:55.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:38:55.349 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:38:55.349 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:38:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6490036a00 legacy=0x7f6490038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 10 monclient: shutdown 2024-09-16T05:38:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 >> v1:172.21.15.6:6790/0 conn(0x7f64a40a1a00 legacy=0x7f64a4133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64a3fff640 1 -- 172.21.15.6:0/354489433 reap_dead start 2024-09-16T05:38:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64a3fff640 1 -- 172.21.15.6:0/354489433 reap_dead start 2024-09-16T05:38:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 shutdown_connections 2024-09-16T05:38:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 >> 172.21.15.6:0/354489433 conn(0x7f64a40a58a0 msgr2=0x7f64a40a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 shutdown_connections 2024-09-16T05:38:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:38:55.321+0000 7f64aa0d2640 1 -- 172.21.15.6:0/354489433 wait complete. 2024-09-16T05:38:55.351 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:38:55.351 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:38:55.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.754+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 v146) ==== 107+0+0 (unknown 4072984537 0 0) 0x7f7eb4018fb0 con 0x7f7ec013ccf0 2024-09-16T05:38:55.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.754+0000 7f7ea7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T05:38:55.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.754+0000 7f7ea7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:55.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.862+0000 7f7ec5b92640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T05:38:55.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.862+0000 7f7ec5b92640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T05:38:55.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.862+0000 7f7ec5b92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:55.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.862+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) -- 0x7f7ea0005ac0 con 0x7f7ec013ccf0 2024-09-16T05:38:55.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.862+0000 7f7ea7fff640 1 -- 172.21.15.63:0/2291547042 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 weight-set for pool 'cool' already created v146) ==== 149+0+0 (unknown 2389658304 0 0) 0x7f7eb401d870 con 0x7f7ec013ccf0 2024-09-16T05:38:55.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.862+0000 7f7ea7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T05:38:55.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.862+0000 7f7ea7fff640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cool' already created 2024-09-16T05:38:55.868 INFO:tasks.workunit.client.0.smithi063.stderr:weight-set for pool 'cool' already created 2024-09-16T05:38:55.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.866+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7eac036d40 legacy=0x7f7eac039200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.866+0000 7f7ec5b92640 10 monclient: shutdown 2024-09-16T05:38:55.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.866+0000 7f7ec5b92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:55.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.866+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 >> v1:172.21.15.6:6790/0 conn(0x7f7ec013ccf0 legacy=0x7f7eac0436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:55.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.866+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 shutdown_connections 2024-09-16T05:38:55.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.866+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 >> 172.21.15.63:0/2291547042 conn(0x7f7ec00a1e90 msgr2=0x7f7ec009f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:55.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.866+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 shutdown_connections 2024-09-16T05:38:55.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:55.866+0000 7f7ec5b92640 1 -- 172.21.15.63:0/2291547042 wait complete. 2024-09-16T05:38:55.884 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set create cold positional 2024-09-16T05:38:56.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient: get_monmap_and_config 2024-09-16T05:38:56.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient: build_initial_monmap 2024-09-16T05:38:56.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient: monmap: 2024-09-16T05:38:56.010 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:56.010 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:56.010 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:56.009335+0000 2024-09-16T05:38:56.011 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:56.009335+0000 2024-09-16T05:38:56.011 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:56.011 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:56.011 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:56.011 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:56.011 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:56.012 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 1 Processor -- start 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 1 -- start start 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient: init 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient(hunting): picked mon.noname-c con 0x7fadfc0a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient(hunting): picked mon.noname-b con 0x7fadfc0a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient(hunting): picked mon.noname-a con 0x7fadfc0a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadfc0a53b0 con 0x7fadfc0a4600 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadfc0a0290 con 0x7fadfc0a3b10 2024-09-16T05:38:56.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadfc0a0dc0 con 0x7fadfc0a07a0 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae0477b640 10 monclient(hunting): authenticate will time out at 1192.787004s 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae02517640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fadfc0a3b10 legacy=0x7fadfc0a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01d16640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fadfc0a07a0 0x7fadfc13be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47410/0 (socket says 172.21.15.63:47410) 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae02d18640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fadfc0a4600 0x7fadfc13f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44166/0 (socket says 172.21.15.63:44166) 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01d16640 1 -- 172.21.15.63:0/2116650919 learned_addr learned my addr 172.21.15.63:0/2116650919 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1548067701 0 0) 0x7fadfc0a0dc0 con 0x7fadfc0a07a0 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 10 monclient(hunting): _negotiate 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 10 monclient(hunting): my global_id is 5750 2024-09-16T05:38:56.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fade8003530 con 0x7fadfc0a07a0 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497373264 0 0) 0x7fadfc0a53b0 con 0x7fadfc0a4600 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 10 monclient(hunting): _negotiate 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.006+0000 7fae01515640 10 monclient(hunting): my global_id is 5769 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadfc0a0dc0 con 0x7fadfc0a4600 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4282741676 0 0) 0x7fadfc0a0dc0 con 0x7fadfc0a4600 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 >> v1:172.21.15.6:6790/0 conn(0x7fadfc0a07a0 legacy=0x7fadfc13be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 >> v1:172.21.15.63:6789/0 conn(0x7fadfc0a3b10 legacy=0x7fadfc0a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: _finish_hunting 0 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 monclient: found mon.noname-a 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:56.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadfc0a10a0 con 0x7fadfc0a4600 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: _finish_auth 0 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: _check_auth_tickets 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fadf40018b0 con 0x7fadfc0a4600 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: dump: 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:56.025 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 5 monclient: authenticate success, global_id 5769 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fadf4001b20 con 0x7fadfc0a4600 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 1 -- 172.21.15.63:0/2116650919 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fadf4001f30 con 0x7fadfc0a4600 2024-09-16T05:38:56.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 4 set_mon_vals no callback set 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae01515640 10 monclient: dump: 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:56.027 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient: shutdown 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- 172.21.15.63:0/2116650919 >> v1:172.21.15.6:6789/0 conn(0x7fadfc0a4600 legacy=0x7fadfc13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- 172.21.15.63:0/2116650919 shutdown_connections 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- 172.21.15.63:0/2116650919 >> 172.21.15.63:0/2116650919 conn(0x7fadfc075470 msgr2=0x7fadfc075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- 172.21.15.63:0/2116650919 shutdown_connections 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- 172.21.15.63:0/2116650919 wait complete. 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient: build_initial_monmap 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient: monmap: 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:56.028 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 Processor -- start 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- start start 2024-09-16T05:38:56.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient: init 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient(hunting): picked mon.noname-c con 0x7fadfc0a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient(hunting): picked mon.noname-a con 0x7fadfc0a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient(hunting): picked mon.noname-b con 0x7fadfc0a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadfc127840 con 0x7fadfc0a3b10 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadfc0a0500 con 0x7fadfc0a4600 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadfc127280 con 0x7fadfc0a07a0 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae0477b640 10 monclient(hunting): authenticate will time out at 1192.790849s 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae02d18640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fadfc0a4600 legacy=0x7fadfc1d39e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae02517640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fadfc0a07a0 0x7fadfc136e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47416/0 (socket says 172.21.15.63:47416) 2024-09-16T05:38:56.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fae02517640 1 -- 172.21.15.63:0/2344571646 learned_addr learned my addr 172.21.15.63:0/2344571646 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 279276443 0 0) 0x7fadfc127840 con 0x7fadfc0a3b10 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fadf2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fadf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fadf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fadf2ffd640 10 monclient(hunting): my global_id is 5772 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.010+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fade0003530 con 0x7fadfc0a3b10 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 958407421 0 0) 0x7fade0003530 con 0x7fadfc0a3b10 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fadfc127840 con 0x7fadfc0a3b10 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fadec001a40 con 0x7fadfc0a3b10 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 10 monclient(hunting): dump: 2024-09-16T05:38:56.031 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 889279338 0 0) 0x7fadfc127840 con 0x7fadfc0a3b10 2024-09-16T05:38:56.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 >> v1:172.21.15.6:6790/0 conn(0x7fadfc0a07a0 legacy=0x7fadfc136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 >> v1:172.21.15.63:6789/0 conn(0x7fadfc0a4600 legacy=0x7fadfc1d39e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 1 monclient: found mon.a 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadfc127560 con 0x7fadfc0a3b10 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fadf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 5 monclient: authenticate success, global_id 5772 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 10 monclient: _renew_subs 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:56.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fadfc1377c0 con 0x7fadfc0a3b10 2024-09-16T05:38:56.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 10 monclient: _renew_subs 2024-09-16T05:38:56.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:56.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fadfc1caf70 con 0x7fadfc0a3b10 2024-09-16T05:38:56.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:56.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:56.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:56.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.014+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadfc0a8340 con 0x7fadfc0a3b10 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.018+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fadec001ca0 con 0x7fadfc0a3b10 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.018+0000 7fadf2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.018+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fadec004c40 con 0x7fadfc0a3b10 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.018+0000 7fadf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.018+0000 7fadf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.018+0000 7fadf2ffd640 10 monclient: dump: 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:56.035 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.018+0000 7fadf3fff640 4 set_mon_vals no callback set 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.022+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fadec00f9e0 con 0x7fadfc0a3b10 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.022+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(147..147 src has 1..147) ==== 7799+0+0 (unknown 1059596350 0 0) 0x7fadec0450e0 con 0x7fadfc0a3b10 2024-09-16T05:38:56.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.022+0000 7fae02517640 10 monclient: get_auth_request con 0x7fade0036a90 auth_method 0 2024-09-16T05:38:56.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.034+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fadec018fb0 con 0x7fadfc0a3b10 2024-09-16T05:38:56.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.034+0000 7fadf2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:56.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.034+0000 7fadf2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.210+0000 7fae0477b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T05:38:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.210+0000 7fae0477b640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T05:38:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.210+0000 7fae0477b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:56.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:56.210+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) -- 0x7fadfc073c40 con 0x7fadfc0a3b10 2024-09-16T05:38:57.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.018+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 v148) ==== 113+0+0 (unknown 1331948489 0 0) 0x7fadec01d870 con 0x7fadfc0a3b10 2024-09-16T05:38:57.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.018+0000 7fadf2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T05:38:57.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.018+0000 7fadf2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:38:57.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.126+0000 7fae0477b640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T05:38:57.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.126+0000 7fae0477b640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T05:38:57.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.126+0000 7fae0477b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:38:57.131 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.126+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) -- 0x7fadfc1cb260 con 0x7fadfc0a3b10 2024-09-16T05:38:57.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.126+0000 7fadf2ffd640 1 -- 172.21.15.63:0/2344571646 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 weight-set for pool 'cold' already created v148) ==== 155+0+0 (unknown 3429023124 0 0) 0x7fadec03ff30 con 0x7fadfc0a3b10 2024-09-16T05:38:57.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.126+0000 7fadf2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T05:38:57.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.126+0000 7fadf2ffd640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cold' already created 2024-09-16T05:38:57.132 INFO:tasks.workunit.client.0.smithi063.stderr:weight-set for pool 'cold' already created 2024-09-16T05:38:57.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.130+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 >> v1:172.21.15.63:6812/3643207211 conn(0x7fade0036a90 legacy=0x7fade0038f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:57.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.130+0000 7fae0477b640 10 monclient: shutdown 2024-09-16T05:38:57.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.130+0000 7fae0477b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:57.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.130+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 >> v1:172.21.15.6:6789/0 conn(0x7fadfc0a3b10 legacy=0x7fadfc1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:57.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.130+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 shutdown_connections 2024-09-16T05:38:57.134 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.130+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 >> 172.21.15.63:0/2344571646 conn(0x7fadfc075470 msgr2=0x7fadfc0a7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:57.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.130+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 shutdown_connections 2024-09-16T05:38:57.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.130+0000 7fae0477b640 1 -- 172.21.15.63:0/2344571646 wait complete. 2024-09-16T05:38:57.149 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.266+0000 7f020056c640 10 monclient: get_monmap_and_config 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.266+0000 7f020056c640 10 monclient: build_initial_monmap 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.266+0000 7f020056c640 10 monclient: monmap: 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:57.271036+0000 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:57.271036+0000 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:57.271 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:57.272 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:57.272 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:57.272 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:57.272 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:57.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.266+0000 7f020056c640 1 Processor -- start 2024-09-16T05:38:57.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.266+0000 7f020056c640 1 -- start start 2024-09-16T05:38:57.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.266+0000 7f020056c640 10 monclient: init 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient(hunting): picked mon.noname-b con 0x7f01f80a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient(hunting): picked mon.noname-c con 0x7f01f80a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient(hunting): picked mon.noname-a con 0x7f01f80a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f80a4fd0 con 0x7f01f80a4790 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f80a5b40 con 0x7f01f80a08a0 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f80a1a70 con 0x7f01f80a3ca0 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient(hunting): authenticate will time out at 1194.048215s 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fdb07640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f01f80a08a0 legacy=0x7f01f813beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:57.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fe308640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f01f80a3ca0 0x7f01f80a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47428/0 (socket says 172.21.15.63:47428) 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01feb09640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f01f80a4790 0x7f01f813f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44178/0 (socket says 172.21.15.63:44178) 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fe308640 1 -- 172.21.15.63:0/4007169812 learned_addr learned my addr 172.21.15.63:0/4007169812 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652347255 0 0) 0x7f01f80a4fd0 con 0x7f01f80a4790 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient(hunting): _negotiate 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient(hunting): my global_id is 5775 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e4003530 con 0x7f01f80a4790 2024-09-16T05:38:57.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3290198504 0 0) 0x7f01f80a1a70 con 0x7f01f80a3ca0 2024-09-16T05:38:57.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient(hunting): _negotiate 2024-09-16T05:38:57.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:57.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:57.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient(hunting): my global_id is 5753 2024-09-16T05:38:57.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01f80a4fd0 con 0x7f01f80a3ca0 2024-09-16T05:38:57.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 789551068 0 0) 0x7f01f80a4fd0 con 0x7f01f80a3ca0 2024-09-16T05:38:57.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 >> v1:172.21.15.63:6789/0 conn(0x7f01f80a08a0 legacy=0x7f01f813beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 >> v1:172.21.15.6:6789/0 conn(0x7f01f80a4790 legacy=0x7f01f813f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: _finish_hunting 0 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 monclient: found mon.noname-b 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f80a53e0 con 0x7f01f80a3ca0 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: _finish_auth 0 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: _check_auth_tickets 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01ec001600 con 0x7f01f80a3ca0 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:57.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: dump: 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 5 monclient: authenticate success, global_id 5753 2024-09-16T05:38:57.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01ec001860 con 0x7f01f80a3ca0 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 1 -- 172.21.15.63:0/4007169812 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01ec001c30 con 0x7f01f80a3ca0 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 4 set_mon_vals no callback set 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f01fd306640 10 monclient: dump: 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.283 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient: shutdown 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- 172.21.15.63:0/4007169812 >> v1:172.21.15.6:6790/0 conn(0x7f01f80a3ca0 legacy=0x7f01f80a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- 172.21.15.63:0/4007169812 shutdown_connections 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- 172.21.15.63:0/4007169812 >> 172.21.15.63:0/4007169812 conn(0x7f01f80a67c0 msgr2=0x7f01f8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- 172.21.15.63:0/4007169812 shutdown_connections 2024-09-16T05:38:57.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- 172.21.15.63:0/4007169812 wait complete. 2024-09-16T05:38:57.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient: build_initial_monmap 2024-09-16T05:38:57.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient: monmap: 2024-09-16T05:38:57.285 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:57.285 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:57.285 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:57.285 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 Processor -- start 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 1 -- start start 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.270+0000 7f020056c640 10 monclient: init 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:38:57.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient(hunting): picked mon.noname-b con 0x7f01f80a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient(hunting): picked mon.noname-a con 0x7f01f80a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient(hunting): picked mon.noname-c con 0x7f01f80a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f80a7f90 con 0x7f01f80a3ca0 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f8125140 con 0x7f01f80a08a0 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01fe308640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f01f80a08a0 legacy=0x7f01f8136d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f8123f70 con 0x7f01f80a4790 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient(hunting): authenticate will time out at 1194.052150s 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01fdb07640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f01f80a3ca0 0x7f01f81d01c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44182/0 (socket says 172.21.15.63:44182) 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01fdb07640 1 -- 172.21.15.63:0/2550082406 learned_addr learned my addr 172.21.15.63:0/2550082406 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553571205 0 0) 0x7f01f8123f70 con 0x7f01f80a4790 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:57.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): my global_id is 5756 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01dc003530 con 0x7f01f80a4790 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261542662 0 0) 0x7f01f4003a50 con 0x7f01f80a3ca0 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): my global_id is 5778 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01f8123f70 con 0x7f01f80a3ca0 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1854603211 0 0) 0x7f01dc003530 con 0x7f01f80a4790 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01f4003a50 con 0x7f01f80a4790 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01f0001bb0 con 0x7f01f80a4790 2024-09-16T05:38:57.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): dump: 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:57.289 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2253253408 0 0) 0x7f01f8123f70 con 0x7f01f80a3ca0 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01dc003530 con 0x7f01f80a3ca0 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01f4004260 con 0x7f01f80a3ca0 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient(hunting): dump: 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:57.290 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2689720870 0 0) 0x7f01f4003a50 con 0x7f01f80a4790 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 >> v1:172.21.15.63:6789/0 conn(0x7f01f80a08a0 legacy=0x7f01f8136d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 >> v1:172.21.15.6:6789/0 conn(0x7f01f80a3ca0 legacy=0x7f01f81d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 monclient: found mon.c 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:57.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:57.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f80a84c0 con 0x7f01f80a4790 2024-09-16T05:38:57.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient: _finish_auth 0 2024-09-16T05:38:57.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:57.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:57.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 5 monclient: authenticate success, global_id 5756 2024-09-16T05:38:57.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient: _renew_subs 2024-09-16T05:38:57.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:57.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01f8125420 con 0x7f01f80a4790 2024-09-16T05:38:57.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient: _renew_subs 2024-09-16T05:38:57.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:57.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f01f8137440 con 0x7f01f80a4790 2024-09-16T05:38:57.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01f0001ff0 con 0x7f01f80a4790 2024-09-16T05:38:57.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:57.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01f0004fc0 con 0x7f01f80a4790 2024-09-16T05:38:57.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01fcb05640 4 set_mon_vals no callback set 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 10 monclient: dump: 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:57.294 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e0005090 con 0x7f01f80a4790 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f01f000fd60 con 0x7f01f80a4790 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(148..148 src has 1..148) ==== 8271+0+0 (unknown 1277696053 0 0) 0x7f01f0045750 con 0x7f01f80a4790 2024-09-16T05:38:57.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.274+0000 7f01fe308640 10 monclient: get_auth_request con 0x7f01dc036ac0 auth_method 0 2024-09-16T05:38:57.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.290+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f01f00157d0 con 0x7f01f80a4790 2024-09-16T05:38:57.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.290+0000 7f01eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:57.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.290+0000 7f01eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:57.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.474+0000 7f020056c640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:38:57.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.474+0000 7f020056c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:38:57.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.474+0000 7f020056c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:57.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:57.474+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7f01e0005b90 con 0x7f01f80a4790 2024-09-16T05:38:58.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:58.034+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v149) ==== 123+0+0 (unknown 28469974 0 0) 0x7f01f00150d0 con 0x7f01f80a4790 2024-09-16T05:38:58.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:58.034+0000 7f01eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:38:58.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:58.034+0000 7f01eaffd640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T05:38:58.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:58.138+0000 7f020056c640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:38:58.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:58.138+0000 7f020056c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:38:58.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:58.138+0000 7f020056c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:58.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:58.138+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7f01e0005610 con 0x7f01f80a4790 2024-09-16T05:38:59.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.034+0000 7f01eaffd640 1 -- 172.21.15.63:0/2550082406 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v150) ==== 149+0+0 (unknown 71783875 0 0) 0x7f01f001dc80 con 0x7f01f80a4790 2024-09-16T05:38:59.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.034+0000 7f01eaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:38:59.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.034+0000 7f01eaffd640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T05:38:59.039 INFO:tasks.workunit.client.0.smithi063.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T05:38:59.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.038+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 >> v1:172.21.15.63:6812/3643207211 conn(0x7f01dc036ac0 legacy=0x7f01dc038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:59.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.038+0000 7f020056c640 10 monclient: shutdown 2024-09-16T05:38:59.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.038+0000 7f020056c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:59.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.038+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 >> v1:172.21.15.6:6790/0 conn(0x7f01f80a4790 legacy=0x7f01f81d38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:59.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.038+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 shutdown_connections 2024-09-16T05:38:59.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.038+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 >> 172.21.15.63:0/2550082406 conn(0x7f01f80a67c0 msgr2=0x7f01f80a8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:59.043 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.038+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 shutdown_connections 2024-09-16T05:38:59.043 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.038+0000 7f020056c640 1 -- 172.21.15.63:0/2550082406 wait complete. 2024-09-16T05:38:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight-compat osd.0 10.5 2024-09-16T05:38:59.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.190+0000 7f29206e4640 10 monclient: get_monmap_and_config 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.190+0000 7f29206e4640 10 monclient: build_initial_monmap 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.190+0000 7f29206e4640 10 monclient: monmap: 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:38:59.194728+0000 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:38:59.194728+0000 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:38:59.195 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:59.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.190+0000 7f29206e4640 1 Processor -- start 2024-09-16T05:38:59.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.190+0000 7f29206e4640 1 -- start start 2024-09-16T05:38:59.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.190+0000 7f29206e4640 10 monclient: init 2024-09-16T05:38:59.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:59.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:59.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient(hunting): picked mon.noname-c con 0x7f29180a17b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:59.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient(hunting): picked mon.noname-b con 0x7f291809c190 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient(hunting): picked mon.noname-a con 0x7f29180a22a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291809aaf0 con 0x7f29180a22a0 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291809aff0 con 0x7f29180a17b0 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291809b2d0 con 0x7f291809c190 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient(hunting): authenticate will time out at 1195.972556s 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291e480640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f29180a17b0 legacy=0x7f29180a1b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291ec81640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f29180a22a0 0x7f291813f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44198/0 (socket says 172.21.15.63:44198) 2024-09-16T05:38:59.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291ec81640 1 -- 172.21.15.63:0/480172936 learned_addr learned my addr 172.21.15.63:0/480172936 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:59.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223751701 0 0) 0x7f291809aaf0 con 0x7f29180a22a0 2024-09-16T05:38:59.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient(hunting): _negotiate 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient(hunting): my global_id is 5781 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2904003530 con 0x7f29180a22a0 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3463388491 0 0) 0x7f291809b2d0 con 0x7f291809c190 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient(hunting): _negotiate 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient(hunting): my global_id is 5759 2024-09-16T05:38:59.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f291809aaf0 con 0x7f291809c190 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1865243796 0 0) 0x7f2904003530 con 0x7f29180a22a0 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 >> v1:172.21.15.6:6790/0 conn(0x7f291809c190 legacy=0x7f291813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 >> v1:172.21.15.63:6789/0 conn(0x7f29180a17b0 legacy=0x7f29180a1b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: _finish_hunting 0 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 monclient: found mon.noname-a 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291809b5b0 con 0x7f29180a22a0 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: _finish_auth 0 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: _check_auth_tickets 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:59.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f29100019a0 con 0x7f29180a22a0 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: dump: 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:59.203 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 5 monclient: authenticate success, global_id 5781 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2910001c00 con 0x7f29180a22a0 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 1 -- 172.21.15.63:0/480172936 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2910001f90 con 0x7f29180a22a0 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f291d47e640 10 monclient: dump: 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:59.204 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient: get_monmap_and_config success 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 4 set_mon_vals no callback set 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient: shutdown 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:38:59.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 1 -- 172.21.15.63:0/480172936 >> v1:172.21.15.6:6789/0 conn(0x7f29180a22a0 legacy=0x7f291813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 1 -- 172.21.15.63:0/480172936 shutdown_connections 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 1 -- 172.21.15.63:0/480172936 >> 172.21.15.63:0/480172936 conn(0x7f29180a2f90 msgr2=0x7f2918075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 1 -- 172.21.15.63:0/480172936 shutdown_connections 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 1 -- 172.21.15.63:0/480172936 wait complete. 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient: build_initial_monmap 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.194+0000 7f29206e4640 10 monclient: monmap: 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:59.206 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:38:59.207 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:38:59.207 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:38:59.207 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:59.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 1 Processor -- start 2024-09-16T05:38:59.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 1 -- start start 2024-09-16T05:38:59.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: init 2024-09-16T05:38:59.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: _reopen_session rank -1 2024-09-16T05:38:59.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:38:59.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient(hunting): picked mon.noname-b con 0x7f291809c190 addr v1:172.21.15.63:6789/0 2024-09-16T05:38:59.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient(hunting): picked mon.noname-c con 0x7f29180a17b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient(hunting): picked mon.noname-a con 0x7f29180a22a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2918127180 con 0x7f29180a22a0 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2918125610 con 0x7f291809c190 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f291809d6b0 con 0x7f29180a17b0 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient(hunting): _renew_subs 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient(hunting): authenticate will time out at 1195.976861s 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f291e480640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f291809c190 legacy=0x7f2918136a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f291ec81640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f29180a22a0 0x7f29181d37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44210/0 (socket says 172.21.15.63:44210) 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f291ec81640 1 -- 172.21.15.63:0/462067988 learned_addr learned my addr 172.21.15.63:0/462067988 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363124433 0 0) 0x7f2918127180 con 0x7f29180a22a0 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:59.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): my global_id is 5784 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28fc003530 con 0x7f29180a22a0 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1146113239 0 0) 0x7f291809d6b0 con 0x7f29180a17b0 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): _negotiate 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): my global_id is 5762 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2918127180 con 0x7f29180a17b0 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2374380312 0 0) 0x7f28fc003530 con 0x7f29180a22a0 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f291809d6b0 con 0x7f29180a22a0 2024-09-16T05:38:59.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f29100018a0 con 0x7f29180a22a0 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): dump: 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:59.213 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 459165953 0 0) 0x7f2918127180 con 0x7f29180a17b0 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28fc003530 con 0x7f29180a17b0 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2908001880 con 0x7f29180a17b0 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient(hunting): dump: 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:59.214 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1538224644 0 0) 0x7f28fc003530 con 0x7f29180a17b0 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 >> v1:172.21.15.63:6789/0 conn(0x7f291809c190 legacy=0x7f2918136a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 >> v1:172.21.15.6:6789/0 conn(0x7f29180a22a0 legacy=0x7f29181d37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient: _finish_hunting 0 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 monclient: found mon.c 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291809d310 con 0x7f29180a17b0 2024-09-16T05:38:59.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient: _finish_auth 0 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient: _check_auth_tickets 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 5 monclient: authenticate success, global_id 5762 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: _renew_subs 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2918126ea0 con 0x7f29180a17b0 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: _renew_subs 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2918137180 con 0x7f29180a17b0 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2908003600 con 0x7f29180a17b0 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:38:59.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2908003950 con 0x7f29180a17b0 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f290effd640 10 monclient: dump: 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:38:59.217 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f291cc7d640 4 set_mon_vals no callback set 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.198+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2900005090 con 0x7f29180a17b0 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.202+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f290800e6f0 con 0x7f29180a17b0 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.202+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(150..150 src has 1..150) ==== 8239+0+0 (unknown 2254728883 0 0) 0x7f29080440a0 con 0x7f29180a17b0 2024-09-16T05:38:59.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.202+0000 7f291e480640 10 monclient: get_auth_request con 0x7f28fc036a10 auth_method 0 2024-09-16T05:38:59.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.214+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2908014160 con 0x7f29180a17b0 2024-09-16T05:38:59.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.214+0000 7f290effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:38:59.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.214+0000 7f290effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:38:59.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.406+0000 7f29206e4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T05:38:59.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.406+0000 7f29206e4640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T05:38:59.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.406+0000 7f29206e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:38:59.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:38:59.406+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) -- 0x7f2900005380 con 0x7f29180a17b0 2024-09-16T05:39:00.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:00.054+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v151) ==== 119+0+0 (unknown 3809373274 0 0) 0x7f2908013a60 con 0x7f29180a17b0 2024-09-16T05:39:00.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:00.054+0000 7f290effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T05:39:00.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:00.054+0000 7f290effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:00.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:00.174+0000 7f29206e4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T05:39:00.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:00.174+0000 7f29206e4640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T05:39:00.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:00.174+0000 7f29206e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:00.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:00.174+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) -- 0x7f2900005ac0 con 0x7f29180a17b0 2024-09-16T05:39:01.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.022+0000 7f290effd640 1 -- 172.21.15.63:0/462067988 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v152) ==== 119+0+0 (unknown 3261305473 0 0) 0x7f2908017db0 con 0x7f29180a17b0 2024-09-16T05:39:01.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.022+0000 7f290effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T05:39:01.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.022+0000 7f290effd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:01.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.026+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 >> v1:172.21.15.63:6812/3643207211 conn(0x7f28fc036a10 legacy=0x7f28fc038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:01.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.026+0000 7f29206e4640 10 monclient: shutdown 2024-09-16T05:39:01.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.026+0000 7f29206e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:01.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.026+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 >> v1:172.21.15.6:6790/0 conn(0x7f29180a17b0 legacy=0x7f29181d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:01.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.026+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 shutdown_connections 2024-09-16T05:39:01.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.026+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 >> 172.21.15.63:0/462067988 conn(0x7f29180a2f90 msgr2=0x7f29180a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:01.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.026+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 shutdown_connections 2024-09-16T05:39:01.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.026+0000 7f29206e4640 1 -- 172.21.15.63:0/462067988 wait complete. 2024-09-16T05:39:01.059 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight cool osd.0 11.5 2024-09-16T05:39:01.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient: get_monmap_and_config 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient: build_initial_monmap 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient: monmap: 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:01.194395+0000 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:01.194395+0000 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:01.195 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:01.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 1 Processor -- start 2024-09-16T05:39:01.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 1 -- start start 2024-09-16T05:39:01.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient: init 2024-09-16T05:39:01.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:01.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:01.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient(hunting): picked mon.noname-b con 0x7f3b001339b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:01.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient(hunting): picked mon.noname-a con 0x7f3b0009fc50 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:01.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient(hunting): picked mon.noname-c con 0x7f3b0013d140 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:01.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b000a0710 con 0x7f3b0009fc50 2024-09-16T05:39:01.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b000a09f0 con 0x7f3b0013d140 2024-09-16T05:39:01.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b000a0cd0 con 0x7f3b001339b0 2024-09-16T05:39:01.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:01.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3b048fa640 10 monclient(hunting): authenticate will time out at 1197.971331s 2024-09-16T05:39:01.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3afed9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3b0013d140 legacy=0x7f3b0013f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:01.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3afe59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3b001339b0 0x7f3b00133d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47456/0 (socket says 172.21.15.63:47456) 2024-09-16T05:39:01.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3afe59c640 1 -- 172.21.15.63:0/1004429927 learned_addr learned my addr 172.21.15.63:0/1004429927 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.190+0000 7f3afdd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3b0009fc50 0x7f3b0013ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44216/0 (socket says 172.21.15.63:44216) 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3976655445 0 0) 0x7f3b000a0710 con 0x7f3b0009fc50 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient(hunting): my global_id is 5787 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ae8003530 con 0x7f3b0009fc50 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 509881158 0 0) 0x7f3b000a0cd0 con 0x7f3b001339b0 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:01.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient(hunting): my global_id is 5765 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b000a0710 con 0x7f3b001339b0 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4075247645 0 0) 0x7f3ae8003530 con 0x7f3b0009fc50 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 >> v1:172.21.15.6:6790/0 conn(0x7f3b001339b0 legacy=0x7f3b00133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 >> v1:172.21.15.63:6789/0 conn(0x7f3b0013d140 legacy=0x7f3b0013f530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: _finish_hunting 0 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 monclient: found mon.noname-a 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b000a15b0 con 0x7f3b0009fc50 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: _finish_auth 0 2024-09-16T05:39:01.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: _check_auth_tickets 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3af0001550 con 0x7f3b0009fc50 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: dump: 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:01.206 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 5 monclient: authenticate success, global_id 5787 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3af00017b0 con 0x7f3b0009fc50 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 1 -- 172.21.15.63:0/1004429927 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3af0001bc0 con 0x7f3b0009fc50 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afd59a640 10 monclient: dump: 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:01.207 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:01.208 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.208 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.208 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:01.208 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:01.208 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:01.208 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:01.208 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:01.208 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 4 set_mon_vals no callback set 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient: shutdown 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- 172.21.15.63:0/1004429927 >> v1:172.21.15.6:6789/0 conn(0x7f3b0009fc50 legacy=0x7f3b0013ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- 172.21.15.63:0/1004429927 shutdown_connections 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- 172.21.15.63:0/1004429927 >> 172.21.15.63:0/1004429927 conn(0x7f3b000a1e90 msgr2=0x7f3b00075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- 172.21.15.63:0/1004429927 shutdown_connections 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- 172.21.15.63:0/1004429927 wait complete. 2024-09-16T05:39:01.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient: build_initial_monmap 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient: monmap: 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 Processor -- start 2024-09-16T05:39:01.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- start start 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient: init 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient(hunting): picked mon.noname-a con 0x7f3b0009fc50 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient(hunting): picked mon.noname-c con 0x7f3b001339b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient(hunting): picked mon.noname-b con 0x7f3b0013d140 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0013f3f0 con 0x7f3b0009fc50 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0012d9a0 con 0x7f3b0013d140 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0012f860 con 0x7f3b001339b0 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3b048fa640 10 monclient(hunting): authenticate will time out at 1197.975310s 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afed9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3b0013d140 legacy=0x7f3b001d3c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:01.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afe59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3b0009fc50 0x7f3b0013e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44230/0 (socket says 172.21.15.63:44230) 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afdd9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3b001339b0 0x7f3b0013ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47470/0 (socket says 172.21.15.63:47470) 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.194+0000 7f3afe59c640 1 -- 172.21.15.63:0/122654566 learned_addr learned my addr 172.21.15.63:0/122654566 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1880100258 0 0) 0x7f3b0013f3f0 con 0x7f3b0009fc50 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): my global_id is 5790 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ae0003530 con 0x7f3b0009fc50 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710828960 0 0) 0x7f3b0012f860 con 0x7f3b001339b0 2024-09-16T05:39:01.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): my global_id is 5768 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b0013f3f0 con 0x7f3b001339b0 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1527449793 0 0) 0x7f3ae0003530 con 0x7f3b0009fc50 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b0012f860 con 0x7f3b0009fc50 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3af40019a0 con 0x7f3b0009fc50 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): dump: 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:01.213 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1878824527 0 0) 0x7f3b0013f3f0 con 0x7f3b001339b0 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ae0003530 con 0x7f3b001339b0 2024-09-16T05:39:01.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3af0001b00 con 0x7f3b001339b0 2024-09-16T05:39:01.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:01.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:01.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient(hunting): dump: 2024-09-16T05:39:01.215 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:01.215 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:01.215 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.215 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.215 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2668528263 0 0) 0x7f3b0012f860 con 0x7f3b0009fc50 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 >> v1:172.21.15.6:6790/0 conn(0x7f3b001339b0 legacy=0x7f3b0013ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 >> v1:172.21.15.63:6789/0 conn(0x7f3b0013d140 legacy=0x7f3b001d3c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:01.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 monclient: found mon.a 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b00140870 con 0x7f3b0009fc50 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 5 monclient: authenticate success, global_id 5790 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 10 monclient: _renew_subs 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0009c0f0 con 0x7f3b0009fc50 2024-09-16T05:39:01.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 10 monclient: _renew_subs 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3b0012f580 con 0x7f3b0009fc50 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3af4001c80 con 0x7f3b0009fc50 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3af4004c40 con 0x7f3b0009fc50 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeffff640 4 set_mon_vals no callback set 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 10 monclient: dump: 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:01.218 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:01.219 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.219 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:01.219 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:01.219 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:01.219 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:01.219 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:01.219 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:01.219 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:01.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:01.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:01.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:01.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae4005090 con 0x7f3b0009fc50 2024-09-16T05:39:01.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3af400f9e0 con 0x7f3b0009fc50 2024-09-16T05:39:01.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(152..152 src has 1..152) ==== 8396+0+0 (unknown 3499608002 0 0) 0x7f3af4045570 con 0x7f3b0009fc50 2024-09-16T05:39:01.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.198+0000 7f3afdd9b640 10 monclient: get_auth_request con 0x7f3ae0036a00 auth_method 0 2024-09-16T05:39:01.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.210+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3af4015530 con 0x7f3b0009fc50 2024-09-16T05:39:01.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.210+0000 7f3aeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:01.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.210+0000 7f3aeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:01.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.386+0000 7f3b048fa640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T05:39:01.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.386+0000 7f3b048fa640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T05:39:01.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.386+0000 7f3b048fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:01.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:01.386+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) -- 0x7f3ae4005380 con 0x7f3b0009fc50 2024-09-16T05:39:02.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:02.046+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v153) ==== 128+0+0 (unknown 1139851515 0 0) 0x7f3af4014e30 con 0x7f3b0009fc50 2024-09-16T05:39:02.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:02.046+0000 7f3aeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T05:39:02.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:02.046+0000 7f3aeeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:02.161 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:02.158+0000 7f3b048fa640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T05:39:02.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:02.158+0000 7f3b048fa640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T05:39:02.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:02.158+0000 7f3b048fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:02.162 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:02.158+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) -- 0x7f3ae4005c70 con 0x7f3b0009fc50 2024-09-16T05:39:03.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.082+0000 7f3aeeffd640 1 -- 172.21.15.63:0/122654566 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v154) ==== 128+0+0 (unknown 4182217501 0 0) 0x7f3af4019180 con 0x7f3b0009fc50 2024-09-16T05:39:03.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.082+0000 7f3aeeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T05:39:03.087 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.082+0000 7f3aeeffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:03.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.086+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3ae0036a00 legacy=0x7f3ae0038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:03.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.086+0000 7f3b048fa640 10 monclient: shutdown 2024-09-16T05:39:03.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.086+0000 7f3b048fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:03.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.086+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 >> v1:172.21.15.6:6789/0 conn(0x7f3b0009fc50 legacy=0x7f3b0013e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:03.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.086+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 shutdown_connections 2024-09-16T05:39:03.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.086+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 >> 172.21.15.63:0/122654566 conn(0x7f3b000a1e90 msgr2=0x7f3b00072140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:03.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.086+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 shutdown_connections 2024-09-16T05:39:03.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.086+0000 7f3b048fa640 1 -- 172.21.15.63:0/122654566 wait complete. 2024-09-16T05:39:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight cold osd.0 12.5 12.4 2024-09-16T05:39:03.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient: get_monmap_and_config 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient: build_initial_monmap 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient: monmap: 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:03.241599+0000 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:03.241599+0000 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:03.242 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:03.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 1 Processor -- start 2024-09-16T05:39:03.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 1 -- start start 2024-09-16T05:39:03.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient: init 2024-09-16T05:39:03.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:03.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:03.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient(hunting): picked mon.noname-c con 0x7fc63409ba70 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:03.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient(hunting): picked mon.noname-b con 0x7fc63409efd0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:03.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient(hunting): picked mon.noname-a con 0x7fc6340a11e0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63409da80 con 0x7fc6340a11e0 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63409e110 con 0x7fc63409ba70 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6340a0e40 con 0x7fc63409efd0 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc638b5c640 10 monclient(hunting): authenticate will time out at 1200.018683s 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc63409ba70 legacy=0x7fc6340a0020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc631d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc63409efd0 0x7fc6340a0730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47476/0 (socket says 172.21.15.63:47476) 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc631d9b640 1 -- 172.21.15.63:0/3717142298 learned_addr learned my addr 172.21.15.63:0/3717142298 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc632d9d640 1 --1- 172.21.15.63:0/3717142298 >> v1:172.21.15.6:6789/0 conn(0x7fc6340a11e0 0x7fc63413b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44240/0 (socket says 172.21.15.63:44240) 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 13894786 0 0) 0x7fc6340a0e40 con 0x7fc63409efd0 2024-09-16T05:39:03.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 10 monclient(hunting): my global_id is 5771 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc61c003530 con 0x7fc63409efd0 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 652427643 0 0) 0x7fc63409da80 con 0x7fc6340a11e0 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 10 monclient(hunting): my global_id is 5793 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.238+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6340a0e40 con 0x7fc6340a11e0 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 12070835 0 0) 0x7fc61c003530 con 0x7fc63409efd0 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 >> v1:172.21.15.63:6789/0 conn(0x7fc63409ba70 legacy=0x7fc6340a0020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:03.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 >> v1:172.21.15.6:6789/0 conn(0x7fc6340a11e0 legacy=0x7fc63413b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: _finish_hunting 0 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 1 monclient: found mon.noname-b 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63409d450 con 0x7fc63409efd0 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: _finish_auth 0 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: _check_auth_tickets 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 5 monclient: authenticate success, global_id 5771 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:03.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc628001620 con 0x7fc63409efd0 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: dump: 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:03.249 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6280018c0 con 0x7fc63409efd0 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 1 -- 172.21.15.63:0/3717142298 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc628001cd0 con 0x7fc63409efd0 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63159a640 10 monclient: dump: 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.250 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:03.251 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:03.251 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:03.251 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:03.251 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:03.251 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:03.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:03.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 4 set_mon_vals no callback set 2024-09-16T05:39:03.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient: shutdown 2024-09-16T05:39:03.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:03.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- 172.21.15.63:0/3717142298 >> v1:172.21.15.6:6790/0 conn(0x7fc63409efd0 legacy=0x7fc6340a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- 172.21.15.63:0/3717142298 shutdown_connections 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- 172.21.15.63:0/3717142298 >> 172.21.15.63:0/3717142298 conn(0x7fc634075470 msgr2=0x7fc634075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- 172.21.15.63:0/3717142298 shutdown_connections 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- 172.21.15.63:0/3717142298 wait complete. 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient: build_initial_monmap 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient: monmap: 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:03.257 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 Processor -- start 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- start start 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient: init 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient(hunting): picked mon.noname-b con 0x7fc63409ba70 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:03.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient(hunting): picked mon.noname-a con 0x7fc63409efd0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient(hunting): picked mon.noname-c con 0x7fc6340a11e0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6341c8580 con 0x7fc63409efd0 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6341a9300 con 0x7fc63409ba70 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63409c0f0 con 0x7fc6340a11e0 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc63259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc63409ba70 legacy=0x7fc6341b9920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc638b5c640 10 monclient(hunting): authenticate will time out at 1200.022886s 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc632d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc6340a11e0 0x7fc6341d3b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47492/0 (socket says 172.21.15.63:47492) 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc632d9d640 1 -- 172.21.15.63:0/2300661854 learned_addr learned my addr 172.21.15.63:0/2300661854 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc631d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc63409efd0 0x7fc6341d03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44246/0 (socket says 172.21.15.63:44246) 2024-09-16T05:39:03.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2813593962 0 0) 0x7fc63409c0f0 con 0x7fc6340a11e0 2024-09-16T05:39:03.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:03.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:03.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:03.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 10 monclient(hunting): my global_id is 5774 2024-09-16T05:39:03.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc614003530 con 0x7fc6340a11e0 2024-09-16T05:39:03.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070409863 0 0) 0x7fc63409d450 con 0x7fc63409efd0 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 10 monclient(hunting): my global_id is 5796 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.242+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc63409c0f0 con 0x7fc63409efd0 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1659451891 0 0) 0x7fc63409c0f0 con 0x7fc63409efd0 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc63409d450 con 0x7fc63409efd0 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4057211563 0 0) 0x7fc614003530 con 0x7fc6340a11e0 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc63409c0f0 con 0x7fc6340a11e0 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc628001c10 con 0x7fc63409efd0 2024-09-16T05:39:03.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:03.262 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc62c001bb0 con 0x7fc6340a11e0 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:03.263 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:03.264 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:03.264 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:03.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1864897628 0 0) 0x7fc63409d450 con 0x7fc63409efd0 2024-09-16T05:39:03.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 >> v1:172.21.15.6:6790/0 conn(0x7fc6340a11e0 legacy=0x7fc6341d3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:03.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 >> v1:172.21.15.63:6789/0 conn(0x7fc63409ba70 legacy=0x7fc6341b9920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:03.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:03.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 monclient: found mon.a 2024-09-16T05:39:03.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:03.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:03.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63406f550 con 0x7fc63409efd0 2024-09-16T05:39:03.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:03.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:03.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:03.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 5 monclient: authenticate success, global_id 5796 2024-09-16T05:39:03.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 10 monclient: _renew_subs 2024-09-16T05:39:03.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:03.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6341a7b70 con 0x7fc63409efd0 2024-09-16T05:39:03.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 10 monclient: _renew_subs 2024-09-16T05:39:03.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:03.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc6341ba030 con 0x7fc63409efd0 2024-09-16T05:39:03.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6280018c0 con 0x7fc63409efd0 2024-09-16T05:39:03.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:03.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc630d99640 4 set_mon_vals no callback set 2024-09-16T05:39:03.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc628004c00 con 0x7fc63409efd0 2024-09-16T05:39:03.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:03.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:03.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 10 monclient: dump: 2024-09-16T05:39:03.267 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:03.268 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:03.268 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.268 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:03.268 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:03.268 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:03.269 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:03.269 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:03.269 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:03.269 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:03.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:03.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:03.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:03.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc618005090 con 0x7fc63409efd0 2024-09-16T05:39:03.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc62800f9a0 con 0x7fc63409efd0 2024-09-16T05:39:03.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(154..154 src has 1..154) ==== 8396+0+0 (unknown 269765080 0 0) 0x7fc6280453e0 con 0x7fc63409efd0 2024-09-16T05:39:03.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.246+0000 7fc63259c640 10 monclient: get_auth_request con 0x7fc6140369c0 auth_method 0 2024-09-16T05:39:03.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.258+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc6341c8580 con 0x7fc63409efd0 2024-09-16T05:39:03.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.258+0000 7fc622ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:03.271 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.258+0000 7fc622ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:03.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.430+0000 7fc638b5c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T05:39:03.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.434+0000 7fc638b5c640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T05:39:03.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.434+0000 7fc638b5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:03.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:03.434+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) -- 0x7fc618005380 con 0x7fc63409efd0 2024-09-16T05:39:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:04.094+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v155) ==== 134+0+0 (unknown 2247049321 0 0) 0x7fc628015540 con 0x7fc63409efd0 2024-09-16T05:39:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:04.094+0000 7fc622ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T05:39:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:04.094+0000 7fc622ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:04.202+0000 7fc638b5c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T05:39:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:04.202+0000 7fc638b5c640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T05:39:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:04.202+0000 7fc638b5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:04.202+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) -- 0x7fc618001180 con 0x7fc63409efd0 2024-09-16T05:39:05.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.094+0000 7fc622ffd640 1 -- 172.21.15.63:0/2300661854 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v156) ==== 134+0+0 (unknown 1497442105 0 0) 0x7fc618001180 con 0x7fc63409efd0 2024-09-16T05:39:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.094+0000 7fc622ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T05:39:05.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.094+0000 7fc622ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:05.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.098+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc6140369c0 legacy=0x7fc614038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:05.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.098+0000 7fc638b5c640 10 monclient: shutdown 2024-09-16T05:39:05.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.098+0000 7fc638b5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:05.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.098+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 >> v1:172.21.15.6:6789/0 conn(0x7fc63409efd0 legacy=0x7fc6341d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:05.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.098+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 shutdown_connections 2024-09-16T05:39:05.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.098+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 >> 172.21.15.63:0/2300661854 conn(0x7fc634075470 msgr2=0x7fc6341bc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:05.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.098+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 shutdown_connections 2024-09-16T05:39:05.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.098+0000 7fc638b5c640 1 -- 172.21.15.63:0/2300661854 wait complete. 2024-09-16T05:39:05.121 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush set-device-class fish osd.0 2024-09-16T05:39:05.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient: get_monmap_and_config 2024-09-16T05:39:05.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient: build_initial_monmap 2024-09-16T05:39:05.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient: monmap: 2024-09-16T05:39:05.245 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:05.245 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:05.246 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:05.245296+0000 2024-09-16T05:39:05.246 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:05.245296+0000 2024-09-16T05:39:05.246 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:05.246 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:05.246 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:05.246 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:05.246 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:05.246 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 1 Processor -- start 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 1 -- start start 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient: init 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient(hunting): picked mon.noname-c con 0x7fbd1809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient(hunting): picked mon.noname-a con 0x7fbd180a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1809b780 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd18074200 con 0x7fbd180a0280 2024-09-16T05:39:05.248 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1de4a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbd1809ac90 legacy=0x7fbd1809b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1809f900 con 0x7fbd1809ac90 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1809e870 con 0x7fbd1809b780 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd200ae640 10 monclient(hunting): authenticate will time out at 1202.022536s 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1e64b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbd1809b780 0x7fbd1813cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44150/0 (socket says 172.21.15.63:44150) 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1d649640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbd180a0280 0x7fbd181398b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33414/0 (socket says 172.21.15.63:33414) 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1e64b640 1 -- 172.21.15.63:0/3862590957 learned_addr learned my addr 172.21.15.63:0/3862590957 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1689520913 0 0) 0x7fbd1809e870 con 0x7fbd1809b780 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 10 monclient(hunting): _negotiate 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 10 monclient(hunting): my global_id is 5777 2024-09-16T05:39:05.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd04003530 con 0x7fbd1809b780 2024-09-16T05:39:05.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96591871 0 0) 0x7fbd18074200 con 0x7fbd180a0280 2024-09-16T05:39:05.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 10 monclient(hunting): _negotiate 2024-09-16T05:39:05.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:05.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:05.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 10 monclient(hunting): my global_id is 5799 2024-09-16T05:39:05.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd1809e870 con 0x7fbd180a0280 2024-09-16T05:39:05.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.242+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2372550958 0 0) 0x7fbd04003530 con 0x7fbd1809b780 2024-09-16T05:39:05.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 >> v1:172.21.15.63:6789/0 conn(0x7fbd1809ac90 legacy=0x7fbd1809b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:05.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 >> v1:172.21.15.6:6789/0 conn(0x7fbd180a0280 legacy=0x7fbd181398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:05.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: _finish_hunting 0 2024-09-16T05:39:05.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 1 monclient: found mon.noname-b 2024-09-16T05:39:05.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:05.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:05.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd18141730 con 0x7fbd1809b780 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: _finish_auth 0 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: _check_auth_tickets 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd100018d0 con 0x7fbd1809b780 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 5 monclient: authenticate success, global_id 5777 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: dump: 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.256 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd10001b30 con 0x7fbd1809b780 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 4 set_mon_vals no callback set 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 1 -- 172.21.15.63:0/3862590957 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd10001ec0 con 0x7fbd1809b780 2024-09-16T05:39:05.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1ce48640 10 monclient: dump: 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:05.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient: shutdown 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- 172.21.15.63:0/3862590957 >> v1:172.21.15.6:6790/0 conn(0x7fbd1809b780 legacy=0x7fbd1813cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- 172.21.15.63:0/3862590957 shutdown_connections 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- 172.21.15.63:0/3862590957 >> 172.21.15.63:0/3862590957 conn(0x7fbd180a5290 msgr2=0x7fbd180a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- 172.21.15.63:0/3862590957 shutdown_connections 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- 172.21.15.63:0/3862590957 wait complete. 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient: build_initial_monmap 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient: monmap: 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:05.259 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 Processor -- start 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- start start 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient: init 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:05.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1809b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient(hunting): picked mon.noname-c con 0x7fbd180a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1813fe90 con 0x7fbd1809b780 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1809d250 con 0x7fbd1809ac90 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1809c0f0 con 0x7fbd180a0280 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd200ae640 10 monclient(hunting): authenticate will time out at 1202.026459s 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1de4a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbd1809ac90 legacy=0x7fbd1809c430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1d649640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbd1809b780 0x7fbd1809cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33424/0 (socket says 172.21.15.63:33424) 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1e64b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbd180a0280 0x7fbd181d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44154/0 (socket says 172.21.15.63:44154) 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd1d649640 1 -- 172.21.15.63:0/3326850425 learned_addr learned my addr 172.21.15.63:0/3326850425 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364863140 0 0) 0x7fbd1809c0f0 con 0x7fbd180a0280 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:05.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): my global_id is 5780 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcfc003530 con 0x7fbd180a0280 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224347035 0 0) 0x7fbd1813fe90 con 0x7fbd1809b780 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): my global_id is 5802 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd1809c0f0 con 0x7fbd1809b780 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1358585106 0 0) 0x7fbcfc003530 con 0x7fbd180a0280 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd1813fe90 con 0x7fbd180a0280 2024-09-16T05:39:05.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd100017d0 con 0x7fbd180a0280 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.246+0000 7fbd0e7fc640 10 monclient(hunting): dump: 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:05.263 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3953294505 0 0) 0x7fbd1809c0f0 con 0x7fbd1809b780 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcfc003530 con 0x7fbd1809b780 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd14001a20 con 0x7fbd1809b780 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient(hunting): dump: 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.264 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3855600903 0 0) 0x7fbd1813fe90 con 0x7fbd180a0280 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 >> v1:172.21.15.63:6789/0 conn(0x7fbd1809ac90 legacy=0x7fbd1809c430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 >> v1:172.21.15.6:6789/0 conn(0x7fbd1809b780 legacy=0x7fbd1809cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 monclient: found mon.c 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:05.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1813fbb0 con 0x7fbd180a0280 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 5 monclient: authenticate success, global_id 5780 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 10 monclient: _renew_subs 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1813ed20 con 0x7fbd180a0280 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 10 monclient: _renew_subs 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd181341c0 con 0x7fbd180a0280 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd10001af0 con 0x7fbd180a0280 2024-09-16T05:39:05.266 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbd10004fc0 con 0x7fbd180a0280 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0ffff640 4 set_mon_vals no callback set 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 10 monclient: dump: 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:05.267 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd00005090 con 0x7fbd180a0280 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbd1000fd60 con 0x7fbd180a0280 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(156..156 src has 1..156) ==== 8396+0+0 (unknown 1417456903 0 0) 0x7fbd10045580 con 0x7fbd180a0280 2024-09-16T05:39:05.268 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.250+0000 7fbd1de4a640 10 monclient: get_auth_request con 0x7fbcfc036980 auth_method 0 2024-09-16T05:39:05.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.262+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbd10015540 con 0x7fbd180a0280 2024-09-16T05:39:05.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.262+0000 7fbd0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:05.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.262+0000 7fbd0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:05.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.434+0000 7fbd200ae640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T05:39:05.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.434+0000 7fbd200ae640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T05:39:05.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.434+0000 7fbd200ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:05.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:05.434+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) -- 0x7fbd00005380 con 0x7fbd180a0280 2024-09-16T05:39:06.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:06.026+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'fish' v157) ==== 137+0+0 (unknown 2208348508 0 0) 0x7fbd10019190 con 0x7fbd180a0280 2024-09-16T05:39:06.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:06.026+0000 7fbd0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T05:39:06.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:06.026+0000 7fbd0e7fc640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'fish' 2024-09-16T05:39:06.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:06.130+0000 7fbd200ae640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T05:39:06.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:06.130+0000 7fbd200ae640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T05:39:06.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:06.130+0000 7fbd200ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:06.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:06.130+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) -- 0x7fbd000010a0 con 0x7fbd180a0280 2024-09-16T05:39:07.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.126+0000 7fbd0e7fc640 1 -- 172.21.15.63:0/3326850425 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' v158) ==== 241+0+0 (unknown 2342971506 0 0) 0x7fbd1001da50 con 0x7fbd180a0280 2024-09-16T05:39:07.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.126+0000 7fbd0e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T05:39:07.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.126+0000 7fbd0e7fc640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-16T05:39:07.133 INFO:tasks.workunit.client.0.smithi063.stderr:osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-16T05:39:07.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.134+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbcfc036980 legacy=0x7fbcfc038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.134+0000 7fbd200ae640 10 monclient: shutdown 2024-09-16T05:39:07.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.134+0000 7fbd200ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:07.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.134+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 >> v1:172.21.15.6:6790/0 conn(0x7fbd180a0280 legacy=0x7fbd181d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.134+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 shutdown_connections 2024-09-16T05:39:07.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.134+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 >> 172.21.15.63:0/3326850425 conn(0x7fbd180a5290 msgr2=0x7fbd18137080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:07.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.134+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 shutdown_connections 2024-09-16T05:39:07.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.134+0000 7fbd200ae640 1 -- 172.21.15.63:0/3326850425 wait complete. 2024-09-16T05:39:07.158 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:07.158 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep fish 2024-09-16T05:39:07.158 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:07.158 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '10\.' 2024-09-16T05:39:07.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient: get_monmap_and_config 2024-09-16T05:39:07.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient: build_initial_monmap 2024-09-16T05:39:07.283 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient: monmap: 2024-09-16T05:39:07.283 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:07.283 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.284 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:07.282323+0000 2024-09-16T05:39:07.284 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:07.282323+0000 2024-09-16T05:39:07.284 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:07.284 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.284 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:07.285 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:07.285 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:07.285 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 1 Processor -- start 2024-09-16T05:39:07.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 1 -- start start 2024-09-16T05:39:07.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient: init 2024-09-16T05:39:07.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:07.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:07.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient(hunting): picked mon.noname-b con 0x7f53b00a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:07.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient(hunting): picked mon.noname-a con 0x7f53b00a24f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:07.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.278+0000 7f53b5059640 10 monclient(hunting): picked mon.noname-c con 0x7f53b009bb00 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:07.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b009ae40 con 0x7f53b00a24f0 2024-09-16T05:39:07.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b009b120 con 0x7f53b009bb00 2024-09-16T05:39:07.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b009b400 con 0x7f53b00a1a00 2024-09-16T05:39:07.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:07.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient(hunting): authenticate will time out at 1204.060046s 2024-09-16T05:39:07.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53af59e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f53b009bb00 legacy=0x7f53b013f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:07.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53aed9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f53b00a1a00 0x7f53b00a1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44164/0 (socket says 172.21.15.63:44164) 2024-09-16T05:39:07.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53aed9d640 1 -- 172.21.15.63:0/1111672210 learned_addr learned my addr 172.21.15.63:0/1111672210 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:07.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53ae59c640 1 --1- 172.21.15.63:0/1111672210 >> v1:172.21.15.6:6789/0 conn(0x7f53b00a24f0 0x7f53b013bc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33434/0 (socket says 172.21.15.63:33434) 2024-09-16T05:39:07.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4064520807 0 0) 0x7f53b009ae40 con 0x7f53b00a24f0 2024-09-16T05:39:07.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient(hunting): _negotiate 2024-09-16T05:39:07.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:07.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:07.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient(hunting): my global_id is 5805 2024-09-16T05:39:07.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5398003530 con 0x7f53b00a24f0 2024-09-16T05:39:07.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184343726 0 0) 0x7f53b009b400 con 0x7f53b00a1a00 2024-09-16T05:39:07.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient(hunting): _negotiate 2024-09-16T05:39:07.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:07.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:07.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient(hunting): my global_id is 5783 2024-09-16T05:39:07.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53b009ae40 con 0x7f53b00a1a00 2024-09-16T05:39:07.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3275450900 0 0) 0x7f5398003530 con 0x7f53b00a24f0 2024-09-16T05:39:07.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 >> v1:172.21.15.6:6790/0 conn(0x7f53b00a1a00 legacy=0x7f53b00a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 >> v1:172.21.15.63:6789/0 conn(0x7f53b009bb00 legacy=0x7f53b013f360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:07.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: _finish_hunting 0 2024-09-16T05:39:07.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 monclient: found mon.noname-a 2024-09-16T05:39:07.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:07.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b009bf50 con 0x7f53b00a24f0 2024-09-16T05:39:07.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: _finish_auth 0 2024-09-16T05:39:07.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: _check_auth_tickets 2024-09-16T05:39:07.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:07.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53a4001620 con 0x7f53b00a24f0 2024-09-16T05:39:07.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:07.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:07.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: dump: 2024-09-16T05:39:07.308 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.309 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.309 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.309 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.309 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.309 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.309 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.310 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.310 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.310 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 5 monclient: authenticate success, global_id 5805 2024-09-16T05:39:07.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:07.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53a4001880 con 0x7f53b00a24f0 2024-09-16T05:39:07.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:07.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 1 -- 172.21.15.63:0/1111672210 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53a4001c50 con 0x7f53b00a24f0 2024-09-16T05:39:07.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:07.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:07.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53add9b640 10 monclient: dump: 2024-09-16T05:39:07.312 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.312 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.312 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.312 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.313 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.313 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.313 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.313 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.313 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:07.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 4 set_mon_vals no callback set 2024-09-16T05:39:07.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient: shutdown 2024-09-16T05:39:07.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:07.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- 172.21.15.63:0/1111672210 >> v1:172.21.15.6:6789/0 conn(0x7f53b00a24f0 legacy=0x7f53b013bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- 172.21.15.63:0/1111672210 shutdown_connections 2024-09-16T05:39:07.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- 172.21.15.63:0/1111672210 >> 172.21.15.63:0/1111672210 conn(0x7f53b00a3570 msgr2=0x7f53b0075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:07.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- 172.21.15.63:0/1111672210 shutdown_connections 2024-09-16T05:39:07.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- 172.21.15.63:0/1111672210 wait complete. 2024-09-16T05:39:07.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient: build_initial_monmap 2024-09-16T05:39:07.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient: monmap: 2024-09-16T05:39:07.316 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:07.316 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:07.316 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:07.316 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:07.316 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:07.317 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.317 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:07.317 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:07.317 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:07.317 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 Processor -- start 2024-09-16T05:39:07.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- start start 2024-09-16T05:39:07.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient: init 2024-09-16T05:39:07.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:07.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:07.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient(hunting): picked mon.noname-a con 0x7f53b009bb00 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:07.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient(hunting): picked mon.noname-c con 0x7f53b00a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:07.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 10 monclient(hunting): picked mon.noname-b con 0x7f53b00a24f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:07.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b013fbc0 con 0x7f53b009bb00 2024-09-16T05:39:07.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.282+0000 7f53b5059640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b01361d0 con 0x7f53b00a24f0 2024-09-16T05:39:07.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b01369d0 con 0x7f53b00a1a00 2024-09-16T05:39:07.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:07.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 monclient(hunting): authenticate will time out at 1204.064020s 2024-09-16T05:39:07.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53af59e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f53b00a24f0 legacy=0x7f53b01d39a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:07.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53aed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f53b009bb00 0x7f53b013f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33438/0 (socket says 172.21.15.63:33438) 2024-09-16T05:39:07.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53ae59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f53b00a1a00 0x7f53b01d0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44166/0 (socket says 172.21.15.63:44166) 2024-09-16T05:39:07.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53aed9d640 1 -- 172.21.15.63:0/3938141688 learned_addr learned my addr 172.21.15.63:0/3938141688 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:07.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3830971894 0 0) 0x7f53b013fbc0 con 0x7f53b009bb00 2024-09-16T05:39:07.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:07.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:07.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:07.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): my global_id is 5808 2024-09-16T05:39:07.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f539c003530 con 0x7f53b009bb00 2024-09-16T05:39:07.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3885001382 0 0) 0x7f53b01369d0 con 0x7f53b00a1a00 2024-09-16T05:39:07.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:07.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:07.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:07.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): my global_id is 5786 2024-09-16T05:39:07.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53b013fbc0 con 0x7f53b00a1a00 2024-09-16T05:39:07.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 87368950 0 0) 0x7f539c003530 con 0x7f53b009bb00 2024-09-16T05:39:07.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53b01369d0 con 0x7f53b009bb00 2024-09-16T05:39:07.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53a00018c0 con 0x7f53b009bb00 2024-09-16T05:39:07.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:07.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:07.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): dump: 2024-09-16T05:39:07.324 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.324 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.324 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.325 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.325 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.325 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.325 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.325 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.325 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.326 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:07.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3334288470 0 0) 0x7f53b013fbc0 con 0x7f53b00a1a00 2024-09-16T05:39:07.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f539c003530 con 0x7f53b00a1a00 2024-09-16T05:39:07.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53a4001520 con 0x7f53b00a1a00 2024-09-16T05:39:07.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:07.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:07.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient(hunting): dump: 2024-09-16T05:39:07.327 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.327 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.327 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.328 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.328 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.328 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.328 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.328 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.329 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.329 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 114668261 0 0) 0x7f53b01369d0 con 0x7f53b009bb00 2024-09-16T05:39:07.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 >> v1:172.21.15.6:6790/0 conn(0x7f53b00a1a00 legacy=0x7f53b01d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 >> v1:172.21.15.63:6789/0 conn(0x7f53b00a24f0 legacy=0x7f53b01d39a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:07.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:07.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 monclient: found mon.a 2024-09-16T05:39:07.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:07.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b0138690 con 0x7f53b009bb00 2024-09-16T05:39:07.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:07.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:07.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:07.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 5 monclient: authenticate success, global_id 5808 2024-09-16T05:39:07.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 monclient: _renew_subs 2024-09-16T05:39:07.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53b0138970 con 0x7f53b009bb00 2024-09-16T05:39:07.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 monclient: _renew_subs 2024-09-16T05:39:07.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f53b01375c0 con 0x7f53b009bb00 2024-09-16T05:39:07.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53a00047f0 con 0x7f53b009bb00 2024-09-16T05:39:07.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:07.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53a0004b40 con 0x7f53b009bb00 2024-09-16T05:39:07.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:07.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:07.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53ad59a640 4 set_mon_vals no callback set 2024-09-16T05:39:07.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 10 monclient: dump: 2024-09-16T05:39:07.334 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.334 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.334 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.335 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.335 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.335 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.335 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.335 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.335 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.336 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:07.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:07.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5390005090 con 0x7f53b009bb00 2024-09-16T05:39:07.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f53a000f8e0 con 0x7f53b009bb00 2024-09-16T05:39:07.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(158..158 src has 1..158) ==== 9061+0+0 (unknown 3776196854 0 0) 0x7f53a00456c0 con 0x7f53b009bb00 2024-09-16T05:39:07.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.286+0000 7f53ae59c640 10 monclient: get_auth_request con 0x7f539c036ac0 auth_method 0 2024-09-16T05:39:07.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.302+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f53a0015390 con 0x7f53b009bb00 2024-09-16T05:39:07.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.302+0000 7f53977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:07.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.302+0000 7f53977fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:07.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.474+0000 7f53b5059640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.474+0000 7f53b5059640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.474+0000 7f53b5059640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.474+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f5390005380 con 0x7f53b009bb00 2024-09-16T05:39:07.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.478+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2175 (unknown 807768366 0 2518512076) 0x7f53a0019040 con 0x7f53b009bb00 2024-09-16T05:39:07.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.478+0000 7f53977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.478+0000 7f53977fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:07.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.578+0000 7f53b5059640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.578+0000 7f53b5059640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.578+0000 7f53b5059640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.578+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f5390005770 con 0x7f53b009bb00 2024-09-16T05:39:07.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53977fe640 1 -- 172.21.15.63:0/3938141688 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2175 (unknown 807768366 0 2518512076) 0x7f53a001d900 con 0x7f53b009bb00 2024-09-16T05:39:07.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53977fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53977fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 >> v1:172.21.15.63:6812/3643207211 conn(0x7f539c036ac0 legacy=0x7f539c038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53b5059640 10 monclient: shutdown 2024-09-16T05:39:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53b5059640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 >> v1:172.21.15.6:6789/0 conn(0x7f53b009bb00 legacy=0x7f53b013f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 shutdown_connections 2024-09-16T05:39:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 >> 172.21.15.63:0/3938141688 conn(0x7f53b00a3570 msgr2=0x7f53b00a5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 shutdown_connections 2024-09-16T05:39:07.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.582+0000 7f53b5059640 1 -- 172.21.15.63:0/3938141688 wait complete. 2024-09-16T05:39:07.603 INFO:tasks.workunit.client.0.smithi063.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:07.603 INFO:tasks.workunit.client.0.smithi063.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:07.604 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:07.604 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep fish 2024-09-16T05:39:07.604 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:07.604 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '11\.' 2024-09-16T05:39:07.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient: get_monmap_and_config 2024-09-16T05:39:07.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient: build_initial_monmap 2024-09-16T05:39:07.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient: monmap: 2024-09-16T05:39:07.726 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:07.726 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.726 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:07.726066+0000 2024-09-16T05:39:07.726 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:07.726066+0000 2024-09-16T05:39:07.726 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:07.727 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.727 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:07.727 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:07.727 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:07.727 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 1 Processor -- start 2024-09-16T05:39:07.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 1 -- start start 2024-09-16T05:39:07.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient: init 2024-09-16T05:39:07.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:07.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:07.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient(hunting): picked mon.noname-c con 0x7f5a240a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:07.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient(hunting): picked mon.noname-b con 0x7f5a240a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:07.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient(hunting): picked mon.noname-a con 0x7f5a240a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a240a4fd0 con 0x7f5a240a4790 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a240a5b40 con 0x7f5a240a3ca0 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a240a1a70 con 0x7f5a240a08a0 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a2b0c3640 10 monclient(hunting): authenticate will time out at 1204.503809s 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.722+0000 7f5a28e5f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5a240a3ca0 legacy=0x7f5a240a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a23fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5a240a08a0 0x7f5a2413beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44170/0 (socket says 172.21.15.63:44170) 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a23fff640 1 -- 172.21.15.63:0/2827642653 learned_addr learned my addr 172.21.15.63:0/2827642653 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:07.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a29660640 1 --1- 172.21.15.63:0/2827642653 >> v1:172.21.15.6:6789/0 conn(0x7f5a240a4790 0x7f5a2413f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33446/0 (socket says 172.21.15.63:33446) 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3083239334 0 0) 0x7f5a240a4fd0 con 0x7f5a240a4790 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient(hunting): my global_id is 5811 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a14003530 con 0x7f5a240a4790 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009493623 0 0) 0x7f5a240a1a70 con 0x7f5a240a08a0 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:07.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:07.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient(hunting): my global_id is 5789 2024-09-16T05:39:07.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a240a4fd0 con 0x7f5a240a08a0 2024-09-16T05:39:07.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3627978984 0 0) 0x7f5a14003530 con 0x7f5a240a4790 2024-09-16T05:39:07.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 >> v1:172.21.15.6:6790/0 conn(0x7f5a240a08a0 legacy=0x7f5a2413beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 >> v1:172.21.15.63:6789/0 conn(0x7f5a240a3ca0 legacy=0x7f5a240a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:07.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:07.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 monclient: found mon.noname-a 2024-09-16T05:39:07.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:07.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a240a53e0 con 0x7f5a240a4790 2024-09-16T05:39:07.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a1c0019a0 con 0x7f5a240a4790 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 5 monclient: authenticate success, global_id 5811 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: dump: 2024-09-16T05:39:07.738 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.739 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.739 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.739 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.739 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.740 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.740 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.741 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.741 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.741 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:07.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a1c001c00 con 0x7f5a240a4790 2024-09-16T05:39:07.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:07.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 1 -- 172.21.15.63:0/2827642653 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a1c001f50 con 0x7f5a240a4790 2024-09-16T05:39:07.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:07.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:07.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a237fe640 10 monclient: dump: 2024-09-16T05:39:07.743 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.743 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.743 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.743 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.743 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.743 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.744 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.744 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.744 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.744 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:07.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 4 set_mon_vals no callback set 2024-09-16T05:39:07.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient: shutdown 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/2827642653 >> v1:172.21.15.6:6789/0 conn(0x7f5a240a4790 legacy=0x7f5a2413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/2827642653 shutdown_connections 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/2827642653 >> 172.21.15.63:0/2827642653 conn(0x7f5a240a67c0 msgr2=0x7f5a24075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/2827642653 shutdown_connections 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/2827642653 wait complete. 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient: build_initial_monmap 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient: monmap: 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:07.746 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 Processor -- start 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- start start 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient: init 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:07.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient(hunting): picked mon.noname-b con 0x7f5a240a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient(hunting): picked mon.noname-c con 0x7f5a240a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient(hunting): picked mon.noname-a con 0x7f5a240a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a24126db0 con 0x7f5a240a4790 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a24125240 con 0x7f5a240a08a0 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a24127090 con 0x7f5a240a3ca0 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a28e5f640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5a240a08a0 legacy=0x7f5a241cb0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.726+0000 7f5a2b0c3640 10 monclient(hunting): authenticate will time out at 1204.507803s 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a23fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5a240a3ca0 0x7f5a24136680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44180/0 (socket says 172.21.15.63:44180) 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a23fff640 1 -- 172.21.15.63:0/3828530380 learned_addr learned my addr 172.21.15.63:0/3828530380 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 210863585 0 0) 0x7f5a24126db0 con 0x7f5a240a4790 2024-09-16T05:39:07.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): my global_id is 5814 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a08003530 con 0x7f5a240a4790 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087165502 0 0) 0x7f5a24127090 con 0x7f5a240a3ca0 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): my global_id is 5792 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a24126db0 con 0x7f5a240a3ca0 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3016536804 0 0) 0x7f5a08003530 con 0x7f5a240a4790 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a24127090 con 0x7f5a240a4790 2024-09-16T05:39:07.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a1c0018a0 con 0x7f5a240a4790 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): dump: 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.750 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2357782424 0 0) 0x7f5a24126db0 con 0x7f5a240a3ca0 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a08003530 con 0x7f5a240a3ca0 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a10001820 con 0x7f5a240a3ca0 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient(hunting): dump: 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.751 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 334513157 0 0) 0x7f5a24127090 con 0x7f5a240a4790 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 >> v1:172.21.15.6:6790/0 conn(0x7f5a240a3ca0 legacy=0x7f5a24136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 >> v1:172.21.15.63:6789/0 conn(0x7f5a240a08a0 legacy=0x7f5a241cb0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 monclient: found mon.a 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a240a0500 con 0x7f5a240a4790 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:07.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 5 monclient: authenticate success, global_id 5814 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 10 monclient: _renew_subs 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a240a8970 con 0x7f5a240a4790 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 10 monclient: _renew_subs 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a24136d90 con 0x7f5a240a4790 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a1c001bc0 con 0x7f5a240a4790 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5a1c004fc0 con 0x7f5a240a4790 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:07.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a22ffd640 4 set_mon_vals no callback set 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 10 monclient: dump: 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:07.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:07.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:07.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a04005090 con 0x7f5a240a4790 2024-09-16T05:39:07.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5a1c00fd60 con 0x7f5a240a4790 2024-09-16T05:39:07.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(158..158 src has 1..158) ==== 9061+0+0 (unknown 3776196854 0 0) 0x7f5a1c0459f0 con 0x7f5a240a4790 2024-09-16T05:39:07.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.730+0000 7f5a28e5f640 10 monclient: get_auth_request con 0x7f5a08036a20 auth_method 0 2024-09-16T05:39:07.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.742+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5a1c015760 con 0x7f5a240a4790 2024-09-16T05:39:07.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.742+0000 7f5a217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:07.756 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.742+0000 7f5a217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:07.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.918+0000 7f5a2b0c3640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.918+0000 7f5a2b0c3640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.918+0000 7f5a2b0c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:07.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.918+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f5a04005380 con 0x7f5a240a4790 2024-09-16T05:39:07.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.922+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2175 (unknown 807768366 0 2518512076) 0x7f5a1c015060 con 0x7f5a240a4790 2024-09-16T05:39:07.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.922+0000 7f5a217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:07.925 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:07.922+0000 7f5a217fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:08.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.022+0000 7f5a2b0c3640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.022+0000 7f5a2b0c3640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.022+0000 7f5a2b0c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.022+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f5a04005770 con 0x7f5a240a4790 2024-09-16T05:39:08.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a217fa640 1 -- 172.21.15.63:0/3828530380 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2175 (unknown 807768366 0 2518512076) 0x7f5a1c01dc10 con 0x7f5a240a4790 2024-09-16T05:39:08.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a217fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a217fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:08.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5a08036a20 legacy=0x7f5a08038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a2b0c3640 10 monclient: shutdown 2024-09-16T05:39:08.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a2b0c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:08.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 >> v1:172.21.15.6:6789/0 conn(0x7f5a240a4790 legacy=0x7f5a241d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 shutdown_connections 2024-09-16T05:39:08.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 >> 172.21.15.63:0/3828530380 conn(0x7f5a240a67c0 msgr2=0x7f5a241373d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:08.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 shutdown_connections 2024-09-16T05:39:08.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.026+0000 7f5a2b0c3640 1 -- 172.21.15.63:0/3828530380 wait complete. 2024-09-16T05:39:08.046 INFO:tasks.workunit.client.0.smithi063.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:08.046 INFO:tasks.workunit.client.0.smithi063.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:08.046 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep fish 2024-09-16T05:39:08.047 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '12\.' 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.162+0000 7fcc0d128640 10 monclient: get_monmap_and_config 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.162+0000 7fcc0d128640 10 monclient: build_initial_monmap 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.162+0000 7fcc0d128640 10 monclient: monmap: 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:08.166727+0000 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:08.166727+0000 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:08.167 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:08.168 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.162+0000 7fcc0d128640 1 Processor -- start 2024-09-16T05:39:08.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.162+0000 7fcc0d128640 1 -- start start 2024-09-16T05:39:08.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.162+0000 7fcc0d128640 10 monclient: init 2024-09-16T05:39:08.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:08.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:08.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient(hunting): picked mon.noname-a con 0x7fcc080a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:08.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient(hunting): picked mon.noname-b con 0x7fcc080a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:08.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient(hunting): picked mon.noname-c con 0x7fcc080a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc080a4fd0 con 0x7fcc080a3ca0 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc080a5b40 con 0x7fcc080a4790 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc080a1a70 con 0x7fcc080a08a0 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient(hunting): authenticate will time out at 1204.944466s 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0759e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcc080a4790 legacy=0x7fcc0813f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc06d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcc080a3ca0 0x7fcc080a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33470/0 (socket says 172.21.15.63:33470) 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc06d9d640 1 -- 172.21.15.63:0/4258434770 learned_addr learned my addr 172.21.15.63:0/4258434770 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:08.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcc080a08a0 0x7fcc0813beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44186/0 (socket says 172.21.15.63:44186) 2024-09-16T05:39:08.175 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 766730006 0 0) 0x7fcc080a4fd0 con 0x7fcc080a3ca0 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient(hunting): my global_id is 5817 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf0003530 con 0x7fcc080a3ca0 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008376346 0 0) 0x7fcc080a1a70 con 0x7fcc080a08a0 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient(hunting): my global_id is 5795 2024-09-16T05:39:08.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc080a4fd0 con 0x7fcc080a08a0 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1964224089 0 0) 0x7fcbf0003530 con 0x7fcc080a3ca0 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 >> v1:172.21.15.6:6790/0 conn(0x7fcc080a08a0 legacy=0x7fcc0813beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 >> v1:172.21.15.63:6789/0 conn(0x7fcc080a4790 legacy=0x7fcc0813f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 monclient: found mon.noname-a 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc080a53e0 con 0x7fcc080a3ca0 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: _finish_auth 0 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcbf80019a0 con 0x7fcc080a3ca0 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:08.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: dump: 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.178 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 5 monclient: authenticate success, global_id 5817 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbf8001c50 con 0x7fcc080a3ca0 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 1 -- 172.21.15.63:0/4258434770 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcbf8002020 con 0x7fcc080a3ca0 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc05d9b640 10 monclient: dump: 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.179 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 4 set_mon_vals no callback set 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient: shutdown 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- 172.21.15.63:0/4258434770 >> v1:172.21.15.6:6789/0 conn(0x7fcc080a3ca0 legacy=0x7fcc080a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- 172.21.15.63:0/4258434770 shutdown_connections 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- 172.21.15.63:0/4258434770 >> 172.21.15.63:0/4258434770 conn(0x7fcc080a67c0 msgr2=0x7fcc08075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- 172.21.15.63:0/4258434770 shutdown_connections 2024-09-16T05:39:08.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- 172.21.15.63:0/4258434770 wait complete. 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient: build_initial_monmap 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient: monmap: 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.181 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 Processor -- start 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 1 -- start start 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.166+0000 7fcc0d128640 10 monclient: init 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient(hunting): picked mon.noname-a con 0x7fcc080a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient(hunting): picked mon.noname-b con 0x7fcc080a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient(hunting): picked mon.noname-c con 0x7fcc080a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc081247a0 con 0x7fcc080a08a0 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc080a88a0 con 0x7fcc080a3ca0 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fcc080a3ca0 legacy=0x7fcc08136390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc080a8370 con 0x7fcc080a4790 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:08.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient(hunting): authenticate will time out at 1204.948307s 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0759e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcc080a4790 0x7fcc081d3440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44196/0 (socket says 172.21.15.63:44196) 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc06d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcc080a08a0 0x7fcc081cb0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33474/0 (socket says 172.21.15.63:33474) 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0759e640 1 -- 172.21.15.63:0/784085686 learned_addr learned my addr 172.21.15.63:0/784085686 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2493560484 0 0) 0x7fcc081247a0 con 0x7fcc080a08a0 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): my global_id is 5820 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf4003530 con 0x7fcc080a08a0 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 164570549 0 0) 0x7fcc080a8370 con 0x7fcc080a4790 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:08.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): my global_id is 5798 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc081247a0 con 0x7fcc080a4790 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2114954306 0 0) 0x7fcbf4003530 con 0x7fcc080a08a0 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc080a8370 con 0x7fcc080a08a0 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcbf8001fa0 con 0x7fcc080a08a0 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): dump: 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.184 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3035218412 0 0) 0x7fcc081247a0 con 0x7fcc080a4790 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbf4003530 con 0x7fcc080a4790 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcc00004040 con 0x7fcc080a4790 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:08.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient(hunting): dump: 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 656940627 0 0) 0x7fcc080a8370 con 0x7fcc080a08a0 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 >> v1:172.21.15.6:6790/0 conn(0x7fcc080a4790 legacy=0x7fcc081d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 >> v1:172.21.15.63:6789/0 conn(0x7fcc080a3ca0 legacy=0x7fcc08136390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 monclient: found mon.a 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc08123350 con 0x7fcc080a08a0 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 5 monclient: authenticate success, global_id 5820 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient: _renew_subs 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc08136aa0 con 0x7fcc080a08a0 2024-09-16T05:39:08.187 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient: _renew_subs 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc081d4b60 con 0x7fcc080a08a0 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbf8001c50 con 0x7fcc080a08a0 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcbf8004fd0 con 0x7fcc080a08a0 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0559a640 4 set_mon_vals no callback set 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 10 monclient: dump: 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.188 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe8005090 con 0x7fcc080a08a0 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcbf800fd70 con 0x7fcc080a08a0 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(158..158 src has 1..158) ==== 9061+0+0 (unknown 3776196854 0 0) 0x7fcbf8045a70 con 0x7fcc080a08a0 2024-09-16T05:39:08.189 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.170+0000 7fcc0659c640 10 monclient: get_auth_request con 0x7fcbf4036a70 auth_method 0 2024-09-16T05:39:08.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.186+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fcbf8015780 con 0x7fcc080a08a0 2024-09-16T05:39:08.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.186+0000 7fcbef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:08.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.186+0000 7fcbef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:08.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.354+0000 7fcc0d128640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.354+0000 7fcc0d128640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.354+0000 7fcc0d128640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.354+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fcbe8005380 con 0x7fcc080a08a0 2024-09-16T05:39:08.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.354+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2175 (unknown 807768366 0 2518512076) 0x7fcbf80193d0 con 0x7fcc080a08a0 2024-09-16T05:39:08.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.354+0000 7fcbef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.354+0000 7fcbef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:08.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.458+0000 7fcc0d128640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.458+0000 7fcc0d128640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.458+0000 7fcc0d128640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.458+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fcbe8005770 con 0x7fcc080a08a0 2024-09-16T05:39:08.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcbef7fe640 1 -- 172.21.15.63:0/784085686 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2175 (unknown 807768366 0 2518512076) 0x7fcbf801dc90 con 0x7fcc080a08a0 2024-09-16T05:39:08.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcbef7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:08.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcbef7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:08.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 >> v1:172.21.15.63:6812/3643207211 conn(0x7fcbf4036a70 legacy=0x7fcbf4038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcc0d128640 10 monclient: shutdown 2024-09-16T05:39:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcc0d128640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 >> v1:172.21.15.6:6789/0 conn(0x7fcc080a08a0 legacy=0x7fcc081cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 shutdown_connections 2024-09-16T05:39:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 >> 172.21.15.63:0/784085686 conn(0x7fcc080a67c0 msgr2=0x7fcc081371e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 shutdown_connections 2024-09-16T05:39:08.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.462+0000 7fcc0d128640 1 -- 172.21.15.63:0/784085686 wait complete. 2024-09-16T05:39:08.482 INFO:tasks.workunit.client.0.smithi063.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:08.483 INFO:tasks.workunit.client.0.smithi063.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:08.483 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T05:39:08.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.598+0000 7f45d1a12640 10 monclient: get_monmap_and_config 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.598+0000 7f45d1a12640 10 monclient: build_initial_monmap 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.598+0000 7f45d1a12640 10 monclient: monmap: 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:08.603764+0000 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:08.603764+0000 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:08.604 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:08.605 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:08.605 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.598+0000 7f45d1a12640 1 Processor -- start 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- start start 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient: init 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient(hunting): picked mon.noname-b con 0x7f45cc0a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient(hunting): picked mon.noname-c con 0x7f45cc09b450 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient(hunting): picked mon.noname-a con 0x7f45cc09b8a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc09e970 con 0x7f45cc09b8a0 2024-09-16T05:39:08.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc09c370 con 0x7f45cc09b450 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc09d620 con 0x7f45cc0a08a0 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient(hunting): authenticate will time out at 1205.380733s 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45c27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f45cc09b450 legacy=0x7f45cc1389b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45cb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f45cc09b8a0 0x7f45cc13c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33476/0 (socket says 172.21.15.63:33476) 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45cb7fe640 1 -- 172.21.15.63:0/1507524294 learned_addr learned my addr 172.21.15.63:0/1507524294 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45caffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f45cc0a08a0 0x7f45cc09ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44206/0 (socket says 172.21.15.63:44206) 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3840631826 0 0) 0x7f45cc09e970 con 0x7f45cc09b8a0 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient(hunting): my global_id is 5823 2024-09-16T05:39:08.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45b8003530 con 0x7f45cc09b8a0 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2049591146 0 0) 0x7f45cc09d620 con 0x7f45cc0a08a0 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient(hunting): my global_id is 5801 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45cc09e970 con 0x7f45cc0a08a0 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1507054439 0 0) 0x7f45b8003530 con 0x7f45cc09b8a0 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 >> v1:172.21.15.6:6790/0 conn(0x7f45cc0a08a0 legacy=0x7f45cc09ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 >> v1:172.21.15.63:6789/0 conn(0x7f45cc09b450 legacy=0x7f45cc1389b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 monclient: found mon.noname-a 2024-09-16T05:39:08.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc09d900 con 0x7f45cc09b8a0 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f45c40019a0 con 0x7f45cc09b8a0 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: dump: 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.615 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 5 monclient: authenticate success, global_id 5823 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f45c4001c50 con 0x7f45cc09b8a0 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 1 -- 172.21.15.63:0/1507524294 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f45c4002020 con 0x7f45cc09b8a0 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 4 set_mon_vals no callback set 2024-09-16T05:39:08.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45ca7fc640 10 monclient: dump: 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient: shutdown 2024-09-16T05:39:08.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- 172.21.15.63:0/1507524294 >> v1:172.21.15.6:6789/0 conn(0x7f45cc09b8a0 legacy=0x7f45cc13c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- 172.21.15.63:0/1507524294 shutdown_connections 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- 172.21.15.63:0/1507524294 >> 172.21.15.63:0/1507524294 conn(0x7f45cc0a1c90 msgr2=0x7f45cc0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- 172.21.15.63:0/1507524294 shutdown_connections 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 1 -- 172.21.15.63:0/1507524294 wait complete. 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient: build_initial_monmap 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.602+0000 7f45d1a12640 10 monclient: monmap: 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:08.618 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 1 Processor -- start 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 1 -- start start 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: init 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient(hunting): picked mon.noname-a con 0x7f45cc09b450 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient(hunting): picked mon.noname-c con 0x7f45cc09b8a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient(hunting): picked mon.noname-b con 0x7f45cc0a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:08.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc121480 con 0x7f45cc09b450 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc0a4050 con 0x7f45cc0a08a0 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc09c0f0 con 0x7f45cc09b8a0 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient(hunting): authenticate will time out at 1205.384790s 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45cb7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f45cc0a08a0 legacy=0x7f45cc1cf6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c27fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f45cc09b8a0 0x7f45cc1cbf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44216/0 (socket says 172.21.15.63:44216) 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45caffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f45cc09b450 0x7f45cc131aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33492/0 (socket says 172.21.15.63:33492) 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c27fc640 1 -- 172.21.15.63:0/3329952926 learned_addr learned my addr 172.21.15.63:0/3329952926 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435679847 0 0) 0x7f45cc121480 con 0x7f45cc09b450 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:08.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): my global_id is 5826 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45ac003560 con 0x7f45cc09b450 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 350016277 0 0) 0x7f45cc09c0f0 con 0x7f45cc09b8a0 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): my global_id is 5804 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45cc121480 con 0x7f45cc09b8a0 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2941833623 0 0) 0x7f45ac003560 con 0x7f45cc09b450 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45cc09c0f0 con 0x7f45cc09b450 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f45bc001810 con 0x7f45cc09b450 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:08.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:08.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 887839744 0 0) 0x7f45cc121480 con 0x7f45cc09b8a0 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45ac003560 con 0x7f45cc09b8a0 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f45b4004160 con 0x7f45cc09b8a0 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.623 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3072523129 0 0) 0x7f45cc09c0f0 con 0x7f45cc09b450 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 >> v1:172.21.15.6:6790/0 conn(0x7f45cc09b8a0 legacy=0x7f45cc1cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 >> v1:172.21.15.63:6789/0 conn(0x7f45cc0a08a0 legacy=0x7f45cc1cf6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 monclient: found mon.a 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc11f190 con 0x7f45cc09b450 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient: _finish_auth 0 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:39:08.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 5 monclient: authenticate success, global_id 5826 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: _renew_subs 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f45cc1211a0 con 0x7f45cc09b450 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: _renew_subs 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f45cc1321b0 con 0x7f45cc09b450 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f45bc0047f0 con 0x7f45cc09b450 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f45bc004b40 con 0x7f45cc09b450 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c9ffb640 4 set_mon_vals no callback set 2024-09-16T05:39:08.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45c8ff9640 10 monclient: dump: 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:08.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.606+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b0005090 con 0x7f45cc09b450 2024-09-16T05:39:08.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.610+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f45bc00f8e0 con 0x7f45cc09b450 2024-09-16T05:39:08.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.610+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(158..158 src has 1..158) ==== 9061+0+0 (unknown 3776196854 0 0) 0x7f45bc0455f0 con 0x7f45cc09b450 2024-09-16T05:39:08.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.610+0000 7f45c27fc640 10 monclient: get_auth_request con 0x7f45ac036aa0 auth_method 0 2024-09-16T05:39:08.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.622+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f45bc015300 con 0x7f45cc09b450 2024-09-16T05:39:08.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.622+0000 7f45c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:08.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.622+0000 7f45c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:08.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.786+0000 7f45d1a12640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:08.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.786+0000 7f45d1a12640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:08.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.786+0000 7f45d1a12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:08.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:08.786+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7f45b0005b90 con 0x7f45cc09b450 2024-09-16T05:39:09.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:09.142+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v159) ==== 123+0+0 (unknown 3241280321 0 0) 0x7f45bc014c00 con 0x7f45cc09b450 2024-09-16T05:39:09.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:09.142+0000 7f45c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:09.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:09.142+0000 7f45c8ff9640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T05:39:09.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:09.246+0000 7f45d1a12640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:09.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:09.246+0000 7f45d1a12640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:09.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:09.246+0000 7f45d1a12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:09.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:09.246+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7f45b0005610 con 0x7f45cc09b450 2024-09-16T05:39:10.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.150+0000 7f45c8ff9640 1 -- 172.21.15.63:0/3329952926 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v160) ==== 149+0+0 (unknown 1916092552 0 0) 0x7f45bc018f50 con 0x7f45cc09b450 2024-09-16T05:39:10.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45c8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:10.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45c8ff9640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T05:39:10.157 INFO:tasks.workunit.client.0.smithi063.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T05:39:10.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 >> v1:172.21.15.63:6812/3643207211 conn(0x7f45ac036aa0 legacy=0x7f45ac038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:10.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45d1a12640 10 monclient: shutdown 2024-09-16T05:39:10.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45d1a12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:10.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 >> v1:172.21.15.6:6789/0 conn(0x7f45cc09b450 legacy=0x7f45cc131aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:10.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 shutdown_connections 2024-09-16T05:39:10.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 >> 172.21.15.63:0/3329952926 conn(0x7f45cc0a1c90 msgr2=0x7f45cc075410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:10.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 shutdown_connections 2024-09-16T05:39:10.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.154+0000 7f45d1a12640 1 -- 172.21.15.63:0/3329952926 wait complete. 2024-09-16T05:39:10.180 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush set-device-class globster osd.0 2024-09-16T05:39:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.298+0000 7fd1290a4640 10 monclient: get_monmap_and_config 2024-09-16T05:39:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.298+0000 7fd1290a4640 10 monclient: build_initial_monmap 2024-09-16T05:39:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.298+0000 7fd1290a4640 10 monclient: monmap: 2024-09-16T05:39:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:10.303214+0000 2024-09-16T05:39:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:10.303214+0000 2024-09-16T05:39:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:10.304 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:10.304 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 Processor -- start 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- start start 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient: init 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient(hunting): picked mon.noname-c con 0x7fd124072270 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient(hunting): picked mon.noname-b con 0x7fd12409f150 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient(hunting): picked mon.noname-a con 0x7fd12413b040 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:10.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd124072d60 con 0x7fd12413b040 2024-09-16T05:39:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd12409fba0 con 0x7fd124072270 2024-09-16T05:39:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd12409eed0 con 0x7fd12409f150 2024-09-16T05:39:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient(hunting): authenticate will time out at 1207.081010s 2024-09-16T05:39:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd122d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd124072270 legacy=0x7fd124072650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd12259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd12409f150 0x7fd124139920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44230/0 (socket says 172.21.15.63:44230) 2024-09-16T05:39:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd12259c640 1 -- 172.21.15.63:0/1685730204 learned_addr learned my addr 172.21.15.63:0/1685730204 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:10.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd12359e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd12413b040 0x7fd12413d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33500/0 (socket says 172.21.15.63:33500) 2024-09-16T05:39:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3002803432 0 0) 0x7fd12409eed0 con 0x7fd12409f150 2024-09-16T05:39:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:39:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:10.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient(hunting): my global_id is 5807 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd10c003530 con 0x7fd12409f150 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3026474245 0 0) 0x7fd124072d60 con 0x7fd12413b040 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient(hunting): my global_id is 5829 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd12409eed0 con 0x7fd12413b040 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4229235174 0 0) 0x7fd10c003530 con 0x7fd12409f150 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 >> v1:172.21.15.63:6789/0 conn(0x7fd124072270 legacy=0x7fd124072650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 >> v1:172.21.15.6:6789/0 conn(0x7fd12413b040 legacy=0x7fd12413d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:39:10.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 monclient: found mon.noname-b 2024-09-16T05:39:10.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12409f580 con 0x7fd12409f150 2024-09-16T05:39:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: _finish_auth 0 2024-09-16T05:39:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:39:10.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd118001620 con 0x7fd12409f150 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: dump: 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:10.317 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 5 monclient: authenticate success, global_id 5807 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd118001940 con 0x7fd12409f150 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:10.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 1 -- 172.21.15.63:0/1685730204 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd118001d10 con 0x7fd12409f150 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd121d9b640 10 monclient: dump: 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:10.319 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 4 set_mon_vals no callback set 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 10 monclient: shutdown 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- 172.21.15.63:0/1685730204 >> v1:172.21.15.6:6790/0 conn(0x7fd12409f150 legacy=0x7fd124139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- 172.21.15.63:0/1685730204 shutdown_connections 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- 172.21.15.63:0/1685730204 >> 172.21.15.63:0/1685730204 conn(0x7fd1240a4180 msgr2=0x7fd124075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:10.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- 172.21.15.63:0/1685730204 shutdown_connections 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.302+0000 7fd1290a4640 1 -- 172.21.15.63:0/1685730204 wait complete. 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: build_initial_monmap 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: monmap: 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:10.321 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 1 Processor -- start 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 1 -- start start 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: init 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient(hunting): picked mon.noname-a con 0x7fd124072270 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient(hunting): picked mon.noname-b con 0x7fd12409f150 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient(hunting): picked mon.noname-c con 0x7fd12413b040 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1240a59c0 con 0x7fd124072270 2024-09-16T05:39:10.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd124120760 con 0x7fd12409f150 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd12259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd12409f150 legacy=0x7fd12411eb50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd124122170 con 0x7fd12413b040 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient(hunting): authenticate will time out at 1207.085522s 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd122d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd124072270 0x7fd12411e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33510/0 (socket says 172.21.15.63:33510) 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd122d9d640 1 -- 172.21.15.63:0/3421455266 learned_addr learned my addr 172.21.15.63:0/3421455266 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 868494718 0 0) 0x7fd1240a59c0 con 0x7fd124072270 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): my global_id is 5832 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd110003530 con 0x7fd124072270 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 595659462 0 0) 0x7fd124122170 con 0x7fd12413b040 2024-09-16T05:39:10.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): my global_id is 5810 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1240a59c0 con 0x7fd12413b040 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4270001966 0 0) 0x7fd1240a59c0 con 0x7fd12413b040 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd124122170 con 0x7fd12413b040 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd11c001c20 con 0x7fd12413b040 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): dump: 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:10.324 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3934874693 0 0) 0x7fd110003530 con 0x7fd124072270 2024-09-16T05:39:10.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1240a59c0 con 0x7fd124072270 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd114004160 con 0x7fd124072270 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient(hunting): dump: 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:10.326 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4020390017 0 0) 0x7fd124122170 con 0x7fd12413b040 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 >> v1:172.21.15.63:6789/0 conn(0x7fd12409f150 legacy=0x7fd12411eb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 >> v1:172.21.15.6:6789/0 conn(0x7fd124072270 legacy=0x7fd12411e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 monclient: found mon.c 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:10.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd124070f80 con 0x7fd12413b040 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 5 monclient: authenticate success, global_id 5810 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: _renew_subs 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1240709c0 con 0x7fd12413b040 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: _renew_subs 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:10.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd124121bb0 con 0x7fd12413b040 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd11c004b70 con 0x7fd12413b040 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd11c004ec0 con 0x7fd12413b040 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd12159a640 4 set_mon_vals no callback set 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.306+0000 7fd10b7fe640 10 monclient: dump: 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:10.329 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.310+0000 7fd1290a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.310+0000 7fd1290a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.310+0000 7fd1290a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.310+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd104005090 con 0x7fd12413b040 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.310+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd11c00fc60 con 0x7fd12413b040 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.310+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(160..160 src has 1..160) ==== 8396+0+0 (unknown 4096962467 0 0) 0x7fd11c045630 con 0x7fd12413b040 2024-09-16T05:39:10.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.310+0000 7fd122d9d640 10 monclient: get_auth_request con 0x7fd110036a70 auth_method 0 2024-09-16T05:39:10.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.322+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd11c015630 con 0x7fd12413b040 2024-09-16T05:39:10.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.322+0000 7fd10b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:10.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.322+0000 7fd10b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:10.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.490+0000 7fd1290a4640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T05:39:10.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.490+0000 7fd1290a4640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T05:39:10.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.490+0000 7fd1290a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:10.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:10.490+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) -- 0x7fd104005380 con 0x7fd12413b040 2024-09-16T05:39:11.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:11.030+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'globster' v161) ==== 145+0+0 (unknown 1348230089 0 0) 0x7fd11c014f30 con 0x7fd12413b040 2024-09-16T05:39:11.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:11.030+0000 7fd10b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T05:39:11.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:11.030+0000 7fd10b7fe640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'globster' 2024-09-16T05:39:11.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:11.134+0000 7fd1290a4640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T05:39:11.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:11.134+0000 7fd1290a4640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T05:39:11.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:11.134+0000 7fd1290a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:11.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:11.134+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) -- 0x7fd1040010a0 con 0x7fd12413b040 2024-09-16T05:39:12.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.042+0000 7fd10b7fe640 1 -- 172.21.15.63:0/3421455266 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' v162) ==== 257+0+0 (unknown 4152588892 0 0) 0x7fd11c01dae0 con 0x7fd12413b040 2024-09-16T05:39:12.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.042+0000 7fd10b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T05:39:12.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.042+0000 7fd10b7fe640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-16T05:39:12.045 INFO:tasks.workunit.client.0.smithi063.stderr:osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-16T05:39:12.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.046+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd110036a70 legacy=0x7fd110038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.046+0000 7fd1290a4640 10 monclient: shutdown 2024-09-16T05:39:12.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.046+0000 7fd1290a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:12.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.046+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 >> v1:172.21.15.6:6790/0 conn(0x7fd12413b040 legacy=0x7fd1241d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.046+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 shutdown_connections 2024-09-16T05:39:12.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.046+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 >> 172.21.15.63:0/3421455266 conn(0x7fd1240a4180 msgr2=0x7fd1240a5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:12.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.046+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 shutdown_connections 2024-09-16T05:39:12.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.046+0000 7fd1290a4640 1 -- 172.21.15.63:0/3421455266 wait complete. 2024-09-16T05:39:12.073 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:12.073 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:12.073 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep globster 2024-09-16T05:39:12.073 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '10\.' 2024-09-16T05:39:12.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient: get_monmap_and_config 2024-09-16T05:39:12.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient: build_initial_monmap 2024-09-16T05:39:12.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient: monmap: 2024-09-16T05:39:12.198 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:12.198 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.198 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:12.198153+0000 2024-09-16T05:39:12.198 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:12.198153+0000 2024-09-16T05:39:12.198 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:12.199 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.199 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:12.199 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:12.199 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:12.199 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 1 Processor -- start 2024-09-16T05:39:12.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 1 -- start start 2024-09-16T05:39:12.199 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient: init 2024-09-16T05:39:12.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:12.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:12.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4409aa60 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:12.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient(hunting): picked mon.noname-a con 0x7f3f440a1ff0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:12.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4409b550 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:12.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f4409cc60 con 0x7f3f440a1ff0 2024-09-16T05:39:12.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f4409bda0 con 0x7f3f4409b550 2024-09-16T05:39:12.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f4409c3c0 con 0x7f3f4409aa60 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f4ab60640 10 monclient(hunting): authenticate will time out at 1208.975936s 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.194+0000 7f3f490fd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3f4409b550 legacy=0x7f3f4413c2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f43fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f440a1ff0 0x7f3f44138bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33516/0 (socket says 172.21.15.63:33516) 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f43fff640 1 -- 172.21.15.63:0/946543235 learned_addr learned my addr 172.21.15.63:0/946543235 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041434753 0 0) 0x7f3f4409c3c0 con 0x7f3f4409aa60 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient(hunting): my global_id is 5813 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f34003530 con 0x7f3f4409aa60 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 965129797 0 0) 0x7f3f4409cc60 con 0x7f3f440a1ff0 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:12.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:12.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:12.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient(hunting): my global_id is 5835 2024-09-16T05:39:12.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f4409c3c0 con 0x7f3f440a1ff0 2024-09-16T05:39:12.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1441255388 0 0) 0x7f3f4409c3c0 con 0x7f3f440a1ff0 2024-09-16T05:39:12.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 >> v1:172.21.15.6:6790/0 conn(0x7f3f4409aa60 legacy=0x7f3f4409ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 >> v1:172.21.15.63:6789/0 conn(0x7f3f4409b550 legacy=0x7f3f4413c2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 monclient: found mon.noname-a 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4409d5d0 con 0x7f3f440a1ff0 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3f30001600 con 0x7f3f440a1ff0 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:12.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: dump: 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 5 monclient: authenticate success, global_id 5835 2024-09-16T05:39:12.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f30001860 con 0x7f3f440a1ff0 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 1 -- 172.21.15.63:0/946543235 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3f30001c70 con 0x7f3f440a1ff0 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f437fe640 10 monclient: dump: 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.210 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 4 set_mon_vals no callback set 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient: shutdown 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- 172.21.15.63:0/946543235 >> v1:172.21.15.6:6789/0 conn(0x7f3f440a1ff0 legacy=0x7f3f44138bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- 172.21.15.63:0/946543235 shutdown_connections 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- 172.21.15.63:0/946543235 >> 172.21.15.63:0/946543235 conn(0x7f3f440a2da0 msgr2=0x7f3f440a31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- 172.21.15.63:0/946543235 shutdown_connections 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- 172.21.15.63:0/946543235 wait complete. 2024-09-16T05:39:12.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient: build_initial_monmap 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient: monmap: 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 Processor -- start 2024-09-16T05:39:12.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- start start 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient: init 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4409aa60 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4409b550 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient(hunting): picked mon.noname-a con 0x7f3f440a1ff0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f44134710 con 0x7f3f440a1ff0 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f4409e790 con 0x7f3f4409aa60 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f4409d2c0 con 0x7f3f4409b550 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f488fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3f4409aa60 legacy=0x7f3f44134000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.198+0000 7f3f4ab60640 10 monclient(hunting): authenticate will time out at 1208.979779s 2024-09-16T05:39:12.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f490fd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f440a1ff0 0x7f3f441d3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33524/0 (socket says 172.21.15.63:33524) 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f490fd640 1 -- 172.21.15.63:0/3216180669 learned_addr learned my addr 172.21.15.63:0/3216180669 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293545901 0 0) 0x7f3f4409d2c0 con 0x7f3f4409b550 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): my global_id is 5816 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f28003530 con 0x7f3f4409b550 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680463462 0 0) 0x7f3f44134710 con 0x7f3f440a1ff0 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): my global_id is 5838 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f4409d2c0 con 0x7f3f440a1ff0 2024-09-16T05:39:12.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1174473906 0 0) 0x7f3f4409d2c0 con 0x7f3f440a1ff0 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f44134710 con 0x7f3f440a1ff0 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3f3c004340 con 0x7f3f440a1ff0 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): dump: 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.215 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4121405555 0 0) 0x7f3f28003530 con 0x7f3f4409b550 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f4409d2c0 con 0x7f3f4409b550 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3f30001bb0 con 0x7f3f4409b550 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient(hunting): dump: 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.216 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3704121069 0 0) 0x7f3f44134710 con 0x7f3f440a1ff0 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 >> v1:172.21.15.6:6790/0 conn(0x7f3f4409b550 legacy=0x7f3f441d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 >> v1:172.21.15.63:6789/0 conn(0x7f3f4409aa60 legacy=0x7f3f44134000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 monclient: found mon.a 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f44071150 con 0x7f3f440a1ff0 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 5 monclient: authenticate success, global_id 5838 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 10 monclient: _renew_subs 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f44127a20 con 0x7f3f440a1ff0 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 10 monclient: _renew_subs 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f44127740 con 0x7f3f440a1ff0 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f3c004850 con 0x7f3f440a1ff0 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:12.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3f3c004ba0 con 0x7f3f440a1ff0 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f42ffd640 4 set_mon_vals no callback set 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 10 monclient: dump: 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.219 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f24005090 con 0x7f3f440a1ff0 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3f3c004d40 con 0x7f3f440a1ff0 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(162..162 src has 1..162) ==== 9085+0+0 (unknown 2742275622 0 0) 0x7f3f3c045840 con 0x7f3f440a1ff0 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.202+0000 7f3f488fc640 10 monclient: get_auth_request con 0x7f3f28036a70 auth_method 0 2024-09-16T05:39:12.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.214+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3f3c0154f0 con 0x7f3f440a1ff0 2024-09-16T05:39:12.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.218+0000 7f3f417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:12.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.218+0000 7f3f417fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:12.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.386+0000 7f3f4ab60640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.386+0000 7f3f4ab60640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.386+0000 7f3f4ab60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.386+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f3f24005380 con 0x7f3f440a1ff0 2024-09-16T05:39:12.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.386+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) ==== 83+0+2378 (unknown 1241242665 0 2507362742) 0x7f3f3c0191a0 con 0x7f3f440a1ff0 2024-09-16T05:39:12.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.386+0000 7f3f417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.386+0000 7f3f417fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:12.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.486+0000 7f3f4ab60640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.486+0000 7f3f4ab60640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.486+0000 7f3f4ab60640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.486+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f3f24005770 con 0x7f3f440a1ff0 2024-09-16T05:39:12.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f417fa640 1 -- 172.21.15.63:0/3216180669 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) ==== 83+0+2378 (unknown 1241242665 0 2507362742) 0x7f3f3c01da60 con 0x7f3f440a1ff0 2024-09-16T05:39:12.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f417fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f417fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:12.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3f28036a70 legacy=0x7f3f28038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f4ab60640 10 monclient: shutdown 2024-09-16T05:39:12.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f4ab60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:12.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 >> v1:172.21.15.6:6789/0 conn(0x7f3f440a1ff0 legacy=0x7f3f441d3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 shutdown_connections 2024-09-16T05:39:12.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 >> 172.21.15.63:0/3216180669 conn(0x7f3f440a2da0 msgr2=0x7f3f4409ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:12.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 shutdown_connections 2024-09-16T05:39:12.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.490+0000 7f3f4ab60640 1 -- 172.21.15.63:0/3216180669 wait complete. 2024-09-16T05:39:12.509 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:12.509 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:12.510 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:12.510 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:12.510 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep globster 2024-09-16T05:39:12.510 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '11\.' 2024-09-16T05:39:12.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient: get_monmap_and_config 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient: build_initial_monmap 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient: monmap: 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:12.629790+0000 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:12.629790+0000 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:12.630 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 1 Processor -- start 2024-09-16T05:39:12.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 1 -- start start 2024-09-16T05:39:12.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient: init 2024-09-16T05:39:12.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:12.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:12.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient(hunting): picked mon.noname-a con 0x7f2de40a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:12.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient(hunting): picked mon.noname-b con 0x7f2de40a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:12.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient(hunting): picked mon.noname-c con 0x7f2de40a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:12.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40a4fd0 con 0x7f2de40a3ca0 2024-09-16T05:39:12.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40a5b40 con 0x7f2de40a4790 2024-09-16T05:39:12.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40a1a70 con 0x7f2de40a08a0 2024-09-16T05:39:12.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:12.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2dea622640 10 monclient(hunting): authenticate will time out at 1209.407314s 2024-09-16T05:39:12.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2de8bbf640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2de40a4790 legacy=0x7f2de413f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:12.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2de3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2de40a3ca0 0x7f2de40a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33536/0 (socket says 172.21.15.63:33536) 2024-09-16T05:39:12.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.626+0000 7f2de3fff640 1 -- 172.21.15.63:0/3491834588 learned_addr learned my addr 172.21.15.63:0/3491834588 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:12.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092252540 0 0) 0x7f2de40a4fd0 con 0x7f2de40a3ca0 2024-09-16T05:39:12.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:12.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:12.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:12.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient(hunting): my global_id is 5841 2024-09-16T05:39:12.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dcc003530 con 0x7f2de40a3ca0 2024-09-16T05:39:12.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107339369 0 0) 0x7f2de40a1a70 con 0x7f2de40a08a0 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient(hunting): my global_id is 5819 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de40a4fd0 con 0x7f2de40a08a0 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 348868680 0 0) 0x7f2dcc003530 con 0x7f2de40a3ca0 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 >> v1:172.21.15.6:6790/0 conn(0x7f2de40a08a0 legacy=0x7f2de413beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 >> v1:172.21.15.63:6789/0 conn(0x7f2de40a4790 legacy=0x7f2de413f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 monclient: found mon.noname-a 2024-09-16T05:39:12.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de40a53e0 con 0x7f2de40a3ca0 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2dd40019a0 con 0x7f2de40a3ca0 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: dump: 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.641 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 5 monclient: authenticate success, global_id 5841 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2dd4001c00 con 0x7f2de40a3ca0 2024-09-16T05:39:12.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 1 -- 172.21.15.63:0/3491834588 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2dd4001f50 con 0x7f2de40a3ca0 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de2ffd640 10 monclient: dump: 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.643 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 4 set_mon_vals no callback set 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient: shutdown 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- 172.21.15.63:0/3491834588 >> v1:172.21.15.6:6789/0 conn(0x7f2de40a3ca0 legacy=0x7f2de40a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- 172.21.15.63:0/3491834588 shutdown_connections 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- 172.21.15.63:0/3491834588 >> 172.21.15.63:0/3491834588 conn(0x7f2de40a67c0 msgr2=0x7f2de4075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- 172.21.15.63:0/3491834588 shutdown_connections 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- 172.21.15.63:0/3491834588 wait complete. 2024-09-16T05:39:12.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient: build_initial_monmap 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient: monmap: 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:12.645 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 Processor -- start 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- start start 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient: init 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient(hunting): picked mon.noname-b con 0x7f2de40a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient(hunting): picked mon.noname-c con 0x7f2de40a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient(hunting): picked mon.noname-a con 0x7f2de40a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de4123630 con 0x7f2de40a4790 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de4121ea0 con 0x7f2de40a08a0 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2de40a08a0 legacy=0x7f2de4135c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:12.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40a0500 con 0x7f2de40a3ca0 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2dea622640 10 monclient(hunting): authenticate will time out at 1209.411034s 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de8bbf640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2de40a4790 0x7f2de41d3ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33542/0 (socket says 172.21.15.63:33542) 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de8bbf640 1 -- 172.21.15.63:0/1252887654 learned_addr learned my addr 172.21.15.63:0/1252887654 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702971097 0 0) 0x7f2de4123630 con 0x7f2de40a4790 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 10 monclient(hunting): my global_id is 5844 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd0003530 con 0x7f2de40a4790 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1750135825 0 0) 0x7f2de40a0500 con 0x7f2de40a3ca0 2024-09-16T05:39:12.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 10 monclient(hunting): my global_id is 5822 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.630+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de4123630 con 0x7f2de40a3ca0 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 282990969 0 0) 0x7f2dd0003530 con 0x7f2de40a4790 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2de40a0500 con 0x7f2de40a4790 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ddc004040 con 0x7f2de40a4790 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.648 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3207882251 0 0) 0x7f2de4123630 con 0x7f2de40a3ca0 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dd0003530 con 0x7f2de40a3ca0 2024-09-16T05:39:12.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2dd80017e0 con 0x7f2de40a3ca0 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.650 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3028394174 0 0) 0x7f2de40a0500 con 0x7f2de40a4790 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 >> v1:172.21.15.6:6790/0 conn(0x7f2de40a3ca0 legacy=0x7f2de41d0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 >> v1:172.21.15.63:6789/0 conn(0x7f2de40a08a0 legacy=0x7f2de4135c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 monclient: found mon.a 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de4121c20 con 0x7f2de40a4790 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 5 monclient: authenticate success, global_id 5844 2024-09-16T05:39:12.651 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 10 monclient: _renew_subs 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2de41369f0 con 0x7f2de40a4790 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 10 monclient: _renew_subs 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2de4136550 con 0x7f2de40a4790 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ddc0042a0 con 0x7f2de40a4790 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ddc0045f0 con 0x7f2de40a4790 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 10 monclient: dump: 2024-09-16T05:39:12.652 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de27fc640 4 set_mon_vals no callback set 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:12.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:12.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc4005090 con 0x7f2de40a4790 2024-09-16T05:39:12.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2ddc004810 con 0x7f2de40a4790 2024-09-16T05:39:12.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(162..162 src has 1..162) ==== 9085+0+0 (unknown 2742275622 0 0) 0x7f2ddc045580 con 0x7f2de40a4790 2024-09-16T05:39:12.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.634+0000 7f2de3fff640 10 monclient: get_auth_request con 0x7f2dd0036a00 auth_method 0 2024-09-16T05:39:12.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.646+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2ddc015210 con 0x7f2de40a4790 2024-09-16T05:39:12.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.646+0000 7f2de0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:12.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.646+0000 7f2de0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:12.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.818+0000 7f2dea622640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.818+0000 7f2dea622640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.818+0000 7f2dea622640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.818+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f2dc4005380 con 0x7f2de40a4790 2024-09-16T05:39:12.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.818+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) ==== 83+0+2378 (unknown 1241242665 0 2507362742) 0x7f2ddc018ec0 con 0x7f2de40a4790 2024-09-16T05:39:12.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.818+0000 7f2de0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.818+0000 7f2de0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:12.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.918+0000 7f2dea622640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.918+0000 7f2dea622640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.918+0000 7f2dea622640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:12.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.918+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f2dc4005770 con 0x7f2de40a4790 2024-09-16T05:39:12.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2de0ff9640 1 -- 172.21.15.63:0/1252887654 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) ==== 83+0+2378 (unknown 1241242665 0 2507362742) 0x7f2ddc01d780 con 0x7f2de40a4790 2024-09-16T05:39:12.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2de0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:12.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2de0ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:12.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2dd0036a00 legacy=0x7f2dd0038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2dea622640 10 monclient: shutdown 2024-09-16T05:39:12.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2dea622640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:12.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 >> v1:172.21.15.6:6789/0 conn(0x7f2de40a4790 legacy=0x7f2de41d3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:12.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 shutdown_connections 2024-09-16T05:39:12.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 >> 172.21.15.63:0/1252887654 conn(0x7f2de40a67c0 msgr2=0x7f2de41bebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:12.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 shutdown_connections 2024-09-16T05:39:12.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:12.922+0000 7f2dea622640 1 -- 172.21.15.63:0/1252887654 wait complete. 2024-09-16T05:39:12.940 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:12.940 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:12.940 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:12.941 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:12.941 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep globster 2024-09-16T05:39:12.941 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '12\.' 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient: get_monmap_and_config 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient: build_initial_monmap 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient: monmap: 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:13.058033+0000 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:13.058033+0000 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:13.058 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.059 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:13.059 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:13.059 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:13.059 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 1 Processor -- start 2024-09-16T05:39:13.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 1 -- start start 2024-09-16T05:39:13.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient: init 2024-09-16T05:39:13.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient(hunting): picked mon.noname-c con 0x7f23c00a12b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient(hunting): picked mon.noname-a con 0x7f23c0072fa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient(hunting): picked mon.noname-b con 0x7f23c01395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c00a05b0 con 0x7f23c0072fa0 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c009c510 con 0x7f23c00a12b0 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c009ce10 con 0x7f23c01395a0 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23c56d7640 10 monclient(hunting): authenticate will time out at 1209.835551s 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23beffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f23c00a12b0 legacy=0x7f23c009f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23bf7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f23c01395a0 0x7f23c013b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44284/0 (socket says 172.21.15.63:44284) 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.054+0000 7f23be7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f23c0072fa0 0x7f23c009fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33546/0 (socket says 172.21.15.63:33546) 2024-09-16T05:39:13.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bf7fe640 1 -- 172.21.15.63:0/358583333 learned_addr learned my addr 172.21.15.63:0/358583333 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:13.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2529159324 0 0) 0x7f23c009ce10 con 0x7f23c01395a0 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient(hunting): my global_id is 5825 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23a8003530 con 0x7f23c01395a0 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 319397678 0 0) 0x7f23c00a05b0 con 0x7f23c0072fa0 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient(hunting): my global_id is 5847 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23c009ce10 con 0x7f23c0072fa0 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1360395555 0 0) 0x7f23a8003530 con 0x7f23c01395a0 2024-09-16T05:39:13.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 >> v1:172.21.15.63:6789/0 conn(0x7f23c00a12b0 legacy=0x7f23c009f790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 >> v1:172.21.15.6:6789/0 conn(0x7f23c0072fa0 legacy=0x7f23c009fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 monclient: found mon.noname-b 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c009d0f0 con 0x7f23c01395a0 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23b80019a0 con 0x7f23c01395a0 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: dump: 2024-09-16T05:39:13.068 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 5 monclient: authenticate success, global_id 5825 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:13.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23b8001c00 con 0x7f23c01395a0 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 1 -- 172.21.15.63:0/358583333 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23b8001f90 con 0x7f23c01395a0 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 4 set_mon_vals no callback set 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bdffb640 10 monclient: dump: 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.070 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient: shutdown 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- 172.21.15.63:0/358583333 >> v1:172.21.15.6:6790/0 conn(0x7f23c01395a0 legacy=0x7f23c013b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- 172.21.15.63:0/358583333 shutdown_connections 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- 172.21.15.63:0/358583333 >> 172.21.15.63:0/358583333 conn(0x7f23c00a1c90 msgr2=0x7f23c00a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- 172.21.15.63:0/358583333 shutdown_connections 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- 172.21.15.63:0/358583333 wait complete. 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient: build_initial_monmap 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient: monmap: 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:13.071 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 Processor -- start 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- start start 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient: init 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:13.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient(hunting): picked mon.noname-a con 0x7f23c0072fa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient(hunting): picked mon.noname-c con 0x7f23c00a12b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient(hunting): picked mon.noname-b con 0x7f23c01395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c0122dc0 con 0x7f23c0072fa0 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c011c970 con 0x7f23c01395a0 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c013dc00 con 0x7f23c00a12b0 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23bf7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f23c01395a0 legacy=0x7f23c01d3430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23c56d7640 10 monclient(hunting): authenticate will time out at 1209.839379s 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23beffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f23c0072fa0 0x7f23c011bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33552/0 (socket says 172.21.15.63:33552) 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.058+0000 7f23beffd640 1 -- 172.21.15.63:0/527672404 learned_addr learned my addr 172.21.15.63:0/527672404 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1540277063 0 0) 0x7f23c013dc00 con 0x7f23c00a12b0 2024-09-16T05:39:13.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): my global_id is 5828 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23ac003560 con 0x7f23c00a12b0 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102451297 0 0) 0x7f23c0122dc0 con 0x7f23c0072fa0 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): my global_id is 5850 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23c013dc00 con 0x7f23c0072fa0 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2664876316 0 0) 0x7f23ac003560 con 0x7f23c00a12b0 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23c0122dc0 con 0x7f23c00a12b0 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23b40019a0 con 0x7f23c00a12b0 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:13.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): dump: 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2636301899 0 0) 0x7f23c013dc00 con 0x7f23c0072fa0 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23ac003560 con 0x7f23c0072fa0 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23b00040e0 con 0x7f23c0072fa0 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient(hunting): dump: 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.076 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1025227004 0 0) 0x7f23c0122dc0 con 0x7f23c00a12b0 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 >> v1:172.21.15.63:6789/0 conn(0x7f23c01395a0 legacy=0x7f23c01d3430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 >> v1:172.21.15.6:6789/0 conn(0x7f23c0072fa0 legacy=0x7f23c011bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 monclient: found mon.c 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c0122ae0 con 0x7f23c00a12b0 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:13.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 5 monclient: authenticate success, global_id 5828 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 10 monclient: _renew_subs 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f23c011d1f0 con 0x7f23c00a12b0 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 10 monclient: _renew_subs 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f23c01d4b50 con 0x7f23c00a12b0 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23b40047f0 con 0x7f23c00a12b0 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23b4004b40 con 0x7f23c00a12b0 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:13.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23bd7fa640 4 set_mon_vals no callback set 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 10 monclient: dump: 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:13.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:13.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23a0005090 con 0x7f23c00a12b0 2024-09-16T05:39:13.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f23b400f8e0 con 0x7f23c00a12b0 2024-09-16T05:39:13.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(162..162 src has 1..162) ==== 9085+0+0 (unknown 2742275622 0 0) 0x7f23b4045710 con 0x7f23c00a12b0 2024-09-16T05:39:13.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.062+0000 7f23beffd640 10 monclient: get_auth_request con 0x7f23ac036a50 auth_method 0 2024-09-16T05:39:13.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.074+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f23b40153a0 con 0x7f23c00a12b0 2024-09-16T05:39:13.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.074+0000 7f23a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:13.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.074+0000 7f23a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:13.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.246+0000 7f23c56d7640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:13.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.246+0000 7f23c56d7640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:13.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.246+0000 7f23c56d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:13.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.246+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f23a0005380 con 0x7f23c00a12b0 2024-09-16T05:39:13.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.246+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) ==== 83+0+2378 (unknown 1241242665 0 2507362742) 0x7f23b4019050 con 0x7f23c00a12b0 2024-09-16T05:39:13.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.250+0000 7f23a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:13.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.250+0000 7f23a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:13.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.350+0000 7f23c56d7640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:13.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.350+0000 7f23c56d7640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:13.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.350+0000 7f23c56d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:13.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.350+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f23a00010a0 con 0x7f23c00a12b0 2024-09-16T05:39:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.350+0000 7f23a77fe640 1 -- 172.21.15.63:0/527672404 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v162) ==== 83+0+2378 (unknown 1241242665 0 2507362742) 0x7f23b401d910 con 0x7f23c00a12b0 2024-09-16T05:39:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.350+0000 7f23a77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:13.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.350+0000 7f23a77fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.354+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 >> v1:172.21.15.63:6812/3643207211 conn(0x7f23ac036a50 legacy=0x7f23ac038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.354+0000 7f23c56d7640 10 monclient: shutdown 2024-09-16T05:39:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.354+0000 7f23c56d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:13.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.354+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 >> v1:172.21.15.6:6790/0 conn(0x7f23c00a12b0 legacy=0x7f23c011c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.354+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 shutdown_connections 2024-09-16T05:39:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.354+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 >> 172.21.15.63:0/527672404 conn(0x7f23c00a1c90 msgr2=0x7f23c013be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.354+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 shutdown_connections 2024-09-16T05:39:13.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.354+0000 7f23c56d7640 1 -- 172.21.15.63:0/527672404 wait complete. 2024-09-16T05:39:13.372 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:13.373 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T05:39:13.373 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight-compat osd.0 7.5 2024-09-16T05:39:13.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient: get_monmap_and_config 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient: build_initial_monmap 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient: monmap: 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:13.496735+0000 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:13.496735+0000 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:13.497 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:13.498 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:13.498 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 1 Processor -- start 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 1 -- start start 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient: init 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient(hunting): picked mon.noname-a con 0x7fc02c134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient(hunting): picked mon.noname-c con 0x7fc02c0a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient(hunting): picked mon.noname-b con 0x7fc02c06dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc02c135420 con 0x7fc02c134930 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc02c09e830 con 0x7fc02c0a0810 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc02c0a0dc0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:13.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient(hunting): authenticate will time out at 1210.273652s 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc030853640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc02c0a0810 legacy=0x7fc02c139820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc031054640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc02c134930 0x7fc02c134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33556/0 (socket says 172.21.15.63:33556) 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc031054640 1 -- 172.21.15.63:0/445072808 learned_addr learned my addr 172.21.15.63:0/445072808 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2909497472 0 0) 0x7fc02c135420 con 0x7fc02c134930 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient(hunting): my global_id is 5853 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc018003530 con 0x7fc02c134930 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3358433189 0 0) 0x7fc02c0a0dc0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:13.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient(hunting): my global_id is 5831 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc02c135420 con 0x7fc02c06dfa0 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3070672131 0 0) 0x7fc02c135420 con 0x7fc02c06dfa0 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 >> v1:172.21.15.63:6789/0 conn(0x7fc02c0a0810 legacy=0x7fc02c139820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 >> v1:172.21.15.6:6789/0 conn(0x7fc02c134930 legacy=0x7fc02c134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 monclient: found mon.noname-b 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02c0a2500 con 0x7fc02c06dfa0 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: _finish_auth 0 2024-09-16T05:39:13.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc024001980 con 0x7fc02c06dfa0 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: dump: 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.510 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 5 monclient: authenticate success, global_id 5831 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc024001ce0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc0332b8640 4 set_mon_vals no callback set 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 1 -- 172.21.15.63:0/445072808 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc0240020b0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:13.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.494+0000 7fc023fff640 10 monclient: dump: 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: shutdown 2024-09-16T05:39:13.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- 172.21.15.63:0/445072808 >> v1:172.21.15.6:6790/0 conn(0x7fc02c06dfa0 legacy=0x7fc02c13cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- 172.21.15.63:0/445072808 shutdown_connections 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- 172.21.15.63:0/445072808 >> 172.21.15.63:0/445072808 conn(0x7fc02c0a1e90 msgr2=0x7fc02c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- 172.21.15.63:0/445072808 shutdown_connections 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- 172.21.15.63:0/445072808 wait complete. 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: build_initial_monmap 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: monmap: 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:13.513 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 Processor -- start 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- start start 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: init 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:13.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient(hunting): picked mon.noname-c con 0x7fc02c06dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient(hunting): picked mon.noname-a con 0x7fc02c0a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient(hunting): picked mon.noname-b con 0x7fc02c134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc02c071dc0 con 0x7fc02c0a0810 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc02c13e370 con 0x7fc02c134930 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc02c09c0f0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc031855640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc02c134930 legacy=0x7fc02c1d16f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient(hunting): authenticate will time out at 1210.278165s 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc030853640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc02c0a0810 0x7fc02c1cdfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:33564/0 (socket says 172.21.15.63:33564) 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc030853640 1 -- 172.21.15.63:0/3674476703 learned_addr learned my addr 172.21.15.63:0/3674476703 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:13.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726811775 0 0) 0x7fc02c071dc0 con 0x7fc02c0a0810 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): my global_id is 5856 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc010003530 con 0x7fc02c0a0810 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2208167899 0 0) 0x7fc02c09c0f0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): my global_id is 5834 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc02c071dc0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4125596650 0 0) 0x7fc010003530 con 0x7fc02c0a0810 2024-09-16T05:39:13.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc02c09c0f0 con 0x7fc02c0a0810 2024-09-16T05:39:13.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 212732592 0 0) 0x7fc02c071dc0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc010003530 con 0x7fc02c06dfa0 2024-09-16T05:39:13.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc028004160 con 0x7fc02c0a0810 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): dump: 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.518 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc01c0018a0 con 0x7fc02c06dfa0 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient(hunting): dump: 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.519 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 414653725 0 0) 0x7fc02c09c0f0 con 0x7fc02c0a0810 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 >> v1:172.21.15.6:6790/0 conn(0x7fc02c06dfa0 legacy=0x7fc02c13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 >> v1:172.21.15.63:6789/0 conn(0x7fc02c134930 legacy=0x7fc02c1d16f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 monclient: found mon.a 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02c137710 con 0x7fc02c0a0810 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:13.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc021ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 5 monclient: authenticate success, global_id 5856 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: _renew_subs 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc02c13dc30 con 0x7fc02c0a0810 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: _renew_subs 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.498+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc02c136820 con 0x7fc02c0a0810 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc0280043c0 con 0x7fc02c0a0810 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc021ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc028004710 con 0x7fc02c0a0810 2024-09-16T05:39:13.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc021ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:13.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc021ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:13.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc0237fe640 4 set_mon_vals no callback set 2024-09-16T05:39:13.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc021ffb640 10 monclient: dump: 2024-09-16T05:39:13.522 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:13.522 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:13.522 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.522 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:13.522 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc0332b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc0332b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc0332b8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00c005090 con 0x7fc02c0a0810 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc028004930 con 0x7fc02c0a0810 2024-09-16T05:39:13.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(162..162 src has 1..162) ==== 9085+0+0 (unknown 2742275622 0 0) 0x7fc0280456b0 con 0x7fc02c0a0810 2024-09-16T05:39:13.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.502+0000 7fc031054640 10 monclient: get_auth_request con 0x7fc010036a10 auth_method 0 2024-09-16T05:39:13.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.514+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc0280153a0 con 0x7fc02c0a0810 2024-09-16T05:39:13.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.514+0000 7fc021ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:13.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.514+0000 7fc021ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.678+0000 7fc0332b8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T05:39:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.678+0000 7fc0332b8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T05:39:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.678+0000 7fc0332b8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:13.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:13.678+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) -- 0x7fc00c005380 con 0x7fc02c0a0810 2024-09-16T05:39:14.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:14.046+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v163) ==== 118+0+0 (unknown 2846499926 0 0) 0x7fc028014ca0 con 0x7fc02c0a0810 2024-09-16T05:39:14.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:14.046+0000 7fc021ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T05:39:14.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:14.046+0000 7fc021ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:14.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:14.154+0000 7fc0332b8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T05:39:14.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:14.154+0000 7fc0332b8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T05:39:14.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:14.154+0000 7fc0332b8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:14.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:14.154+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) -- 0x7fc00c005ac0 con 0x7fc02c0a0810 2024-09-16T05:39:15.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.050+0000 7fc021ffb640 1 -- 172.21.15.63:0/3674476703 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v164) ==== 118+0+0 (unknown 1625482069 0 0) 0x7fc028018ff0 con 0x7fc02c0a0810 2024-09-16T05:39:15.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.050+0000 7fc021ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T05:39:15.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.050+0000 7fc021ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:15.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.054+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc010036a10 legacy=0x7fc010038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:15.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.054+0000 7fc0332b8640 10 monclient: shutdown 2024-09-16T05:39:15.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.054+0000 7fc0332b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:15.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.054+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 >> v1:172.21.15.6:6789/0 conn(0x7fc02c0a0810 legacy=0x7fc02c1cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:15.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.054+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 shutdown_connections 2024-09-16T05:39:15.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.054+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 >> 172.21.15.63:0/3674476703 conn(0x7fc02c0a1e90 msgr2=0x7fc02c0a2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:15.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.054+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 shutdown_connections 2024-09-16T05:39:15.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.054+0000 7fc0332b8640 1 -- 172.21.15.63:0/3674476703 wait complete. 2024-09-16T05:39:15.080 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight cool osd.0 8.5 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient: get_monmap_and_config 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient: build_initial_monmap 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient: monmap: 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:15.206118+0000 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:15.206118+0000 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:15.206 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:15.207 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:15.207 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:15.207 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:15.207 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:15.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 1 Processor -- start 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 1 -- start start 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient: init 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e8134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e80a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e806dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e8135420 con 0x7fd1e80a0810 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e809e830 con 0x7fd1e806dfa0 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e80a0dc0 con 0x7fd1e8134930 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1ecf8a640 10 monclient(hunting): authenticate will time out at 1211.983161s 2024-09-16T05:39:15.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1e6d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd1e806dfa0 legacy=0x7fd1e813cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1dffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd1e80a0810 0x7fd1e8139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60064/0 (socket says 172.21.15.63:60064) 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1dffff640 1 -- 172.21.15.63:0/2538510686 learned_addr learned my addr 172.21.15.63:0/2538510686 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.202+0000 7fd1e659c640 1 --1- 172.21.15.63:0/2538510686 >> v1:172.21.15.6:6790/0 conn(0x7fd1e8134930 0x7fd1e8134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41158/0 (socket says 172.21.15.63:41158) 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1262405975 0 0) 0x7fd1e8135420 con 0x7fd1e80a0810 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient(hunting): my global_id is 5859 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1d4003530 con 0x7fd1e80a0810 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109792179 0 0) 0x7fd1e80a0dc0 con 0x7fd1e8134930 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:15.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient(hunting): my global_id is 5837 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1e8135420 con 0x7fd1e8134930 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266953529 0 0) 0x7fd1d4003530 con 0x7fd1e80a0810 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 >> v1:172.21.15.6:6790/0 conn(0x7fd1e8134930 legacy=0x7fd1e8134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 >> v1:172.21.15.63:6789/0 conn(0x7fd1e806dfa0 legacy=0x7fd1e813cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 monclient: found mon.noname-a 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e80a2500 con 0x7fd1e80a0810 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: _finish_auth 0 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:39:15.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd1d0001620 con 0x7fd1e80a0810 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: dump: 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 5 monclient: authenticate success, global_id 5859 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1d0001880 con 0x7fd1e80a0810 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/2538510686 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd1d0001c90 con 0x7fd1e80a0810 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e5d9b640 10 monclient: dump: 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.214 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 4 set_mon_vals no callback set 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient: shutdown 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/2538510686 >> v1:172.21.15.6:6789/0 conn(0x7fd1e80a0810 legacy=0x7fd1e8139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/2538510686 shutdown_connections 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/2538510686 >> 172.21.15.63:0/2538510686 conn(0x7fd1e80a1e90 msgr2=0x7fd1e8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/2538510686 shutdown_connections 2024-09-16T05:39:15.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/2538510686 wait complete. 2024-09-16T05:39:15.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient: build_initial_monmap 2024-09-16T05:39:15.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient: monmap: 2024-09-16T05:39:15.219 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:15.219 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:15.219 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:15.219 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:15.219 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:15.220 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:15.220 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:15.220 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:15.220 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:15.220 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:15.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 Processor -- start 2024-09-16T05:39:15.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- start start 2024-09-16T05:39:15.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient: init 2024-09-16T05:39:15.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:15.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:15.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e806dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:15.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e80a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:15.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e8134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:15.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e813a0b0 con 0x7fd1e8134930 2024-09-16T05:39:15.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e813a390 con 0x7fd1e806dfa0 2024-09-16T05:39:15.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e8071dd0 con 0x7fd1e80a0810 2024-09-16T05:39:15.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:15.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1ecf8a640 10 monclient(hunting): authenticate will time out at 1211.987083s 2024-09-16T05:39:15.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1e659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd1e806dfa0 legacy=0x7fd1e809c0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:15.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1dffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd1e80a0810 0x7fd1e809c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41166/0 (socket says 172.21.15.63:41166) 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1dffff640 1 -- 172.21.15.63:0/549661498 learned_addr learned my addr 172.21.15.63:0/549661498 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533759178 0 0) 0x7fd1e8071dd0 con 0x7fd1e80a0810 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 10 monclient(hunting): my global_id is 5840 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1c8003530 con 0x7fd1e80a0810 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768285437 0 0) 0x7fd1e813a0b0 con 0x7fd1e8134930 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 10 monclient(hunting): my global_id is 5862 2024-09-16T05:39:15.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.206+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1e8071dd0 con 0x7fd1e8134930 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2921406235 0 0) 0x7fd1c8003530 con 0x7fd1e80a0810 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1e813a0b0 con 0x7fd1e80a0810 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3764830178 0 0) 0x7fd1e8071dd0 con 0x7fd1e8134930 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1c8003530 con 0x7fd1e8134930 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd1d0001c10 con 0x7fd1e80a0810 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient(hunting): dump: 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.224 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd1e0004340 con 0x7fd1e8134930 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient(hunting): dump: 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:15.225 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 606886014 0 0) 0x7fd1e813a0b0 con 0x7fd1e80a0810 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 >> v1:172.21.15.63:6789/0 conn(0x7fd1e806dfa0 legacy=0x7fd1e809c0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 >> v1:172.21.15.6:6789/0 conn(0x7fd1e8134930 legacy=0x7fd1e81d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:15.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 monclient: found mon.c 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e813e2a0 con 0x7fd1e80a0810 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 5 monclient: authenticate success, global_id 5840 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 10 monclient: _renew_subs 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e8139dd0 con 0x7fd1e80a0810 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 10 monclient: _renew_subs 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:15.227 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1e809cf10 con 0x7fd1e80a0810 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1d0001880 con 0x7fd1e80a0810 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd1d0004da0 con 0x7fd1e80a0810 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1e559a640 4 set_mon_vals no callback set 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 10 monclient: dump: 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:15.228 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1cc005090 con 0x7fd1e80a0810 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd1d0004fc0 con 0x7fd1e80a0810 2024-09-16T05:39:15.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(164..164 src has 1..164) ==== 9023+0+0 (unknown 1052003863 0 0) 0x7fd1d0045760 con 0x7fd1e80a0810 2024-09-16T05:39:15.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.210+0000 7fd1e659c640 10 monclient: get_auth_request con 0x7fd1c80369c0 auth_method 0 2024-09-16T05:39:15.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.222+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd1d00155b0 con 0x7fd1e80a0810 2024-09-16T05:39:15.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.222+0000 7fd1deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:15.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.222+0000 7fd1deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:15.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.398+0000 7fd1ecf8a640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T05:39:15.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.398+0000 7fd1ecf8a640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T05:39:15.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.398+0000 7fd1ecf8a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:15.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:15.398+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) -- 0x7fd1cc005380 con 0x7fd1e80a0810 2024-09-16T05:39:16.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:16.034+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v165) ==== 127+0+0 (unknown 4204214558 0 0) 0x7fd1d0014eb0 con 0x7fd1e80a0810 2024-09-16T05:39:16.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:16.034+0000 7fd1deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T05:39:16.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:16.034+0000 7fd1deffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:16.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:16.142+0000 7fd1ecf8a640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T05:39:16.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:16.142+0000 7fd1ecf8a640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T05:39:16.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:16.142+0000 7fd1ecf8a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:16.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:16.142+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) -- 0x7fd1cc001150 con 0x7fd1e80a0810 2024-09-16T05:39:17.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.034+0000 7fd1deffd640 1 -- 172.21.15.63:0/549661498 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v166) ==== 127+0+0 (unknown 2010489126 0 0) 0x7fd1d0040760 con 0x7fd1e80a0810 2024-09-16T05:39:17.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.034+0000 7fd1deffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T05:39:17.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.034+0000 7fd1deffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:17.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.038+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd1c80369c0 legacy=0x7fd1c8038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:17.044 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.042+0000 7fd1ecf8a640 10 monclient: shutdown 2024-09-16T05:39:17.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.042+0000 7fd1ecf8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:17.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.042+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 >> v1:172.21.15.6:6790/0 conn(0x7fd1e80a0810 legacy=0x7fd1e809c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:17.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.042+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 shutdown_connections 2024-09-16T05:39:17.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.042+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 >> 172.21.15.63:0/549661498 conn(0x7fd1e80a1e90 msgr2=0x7fd1e80a34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:17.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.042+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 shutdown_connections 2024-09-16T05:39:17.045 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.042+0000 7fd1ecf8a640 1 -- 172.21.15.63:0/549661498 wait complete. 2024-09-16T05:39:17.065 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set reweight cold osd.0 6.5 6.6 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient: get_monmap_and_config 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient: build_initial_monmap 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient: monmap: 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:17.197893+0000 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:17.197893+0000 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:17.198 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:17.199 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:17.199 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:17.199 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:17.199 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:17.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 1 Processor -- start 2024-09-16T05:39:17.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 1 -- start start 2024-09-16T05:39:17.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient: init 2024-09-16T05:39:17.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:17.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:17.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient(hunting): picked mon.noname-b con 0x7febbc09c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient(hunting): picked mon.noname-c con 0x7febbc09e970 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient(hunting): picked mon.noname-a con 0x7febbc139a30 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc134380 con 0x7febbc139a30 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc0a1310 con 0x7febbc09e970 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc0a1570 con 0x7febbc09c300 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc3d7c640 10 monclient(hunting): authenticate will time out at 1213.975478s 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc1317640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7febbc09e970 legacy=0x7febbc133c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc1b18640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7febbc09c300 0x7febbc133560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41182/0 (socket says 172.21.15.63:41182) 2024-09-16T05:39:17.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc2319640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7febbc139a30 0x7febbc13be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60086/0 (socket says 172.21.15.63:60086) 2024-09-16T05:39:17.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.194+0000 7febc1b18640 1 -- 172.21.15.63:0/1511518129 learned_addr learned my addr 172.21.15.63:0/1511518129 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:17.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3475189468 0 0) 0x7febbc134380 con 0x7febbc139a30 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient(hunting): _negotiate 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient(hunting): my global_id is 5865 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feba8003530 con 0x7febbc139a30 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957352486 0 0) 0x7febbc0a1570 con 0x7febbc09c300 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient(hunting): _negotiate 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:17.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient(hunting): my global_id is 5843 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febbc134380 con 0x7febbc09c300 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2616204155 0 0) 0x7feba8003530 con 0x7febbc139a30 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 >> v1:172.21.15.6:6790/0 conn(0x7febbc09c300 legacy=0x7febbc133560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 >> v1:172.21.15.63:6789/0 conn(0x7febbc09e970 legacy=0x7febbc133c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: _finish_hunting 0 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 monclient: found mon.noname-a 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc09dcb0 con 0x7febbc139a30 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: _finish_auth 0 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: _check_auth_tickets 2024-09-16T05:39:17.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febb40018d0 con 0x7febbc139a30 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: dump: 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:17.206 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 5 monclient: authenticate success, global_id 5865 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febb4001b80 con 0x7febbc139a30 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 1 -- 172.21.15.63:0/1511518129 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febb4001f50 con 0x7febbc139a30 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc0b16640 10 monclient: dump: 2024-09-16T05:39:17.207 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 4 set_mon_vals no callback set 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 10 monclient: shutdown 2024-09-16T05:39:17.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 1 -- 172.21.15.63:0/1511518129 >> v1:172.21.15.6:6789/0 conn(0x7febbc139a30 legacy=0x7febbc13be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 1 -- 172.21.15.63:0/1511518129 shutdown_connections 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 1 -- 172.21.15.63:0/1511518129 >> 172.21.15.63:0/1511518129 conn(0x7febbc075470 msgr2=0x7febbc075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 1 -- 172.21.15.63:0/1511518129 shutdown_connections 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 1 -- 172.21.15.63:0/1511518129 wait complete. 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 10 monclient: build_initial_monmap 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 10 monclient: monmap: 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:17.209 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:17.210 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:17.210 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:17.210 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:17.210 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:17.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 1 Processor -- start 2024-09-16T05:39:17.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 1 -- start start 2024-09-16T05:39:17.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 10 monclient: init 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.198+0000 7febc3d7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient(hunting): picked mon.noname-a con 0x7febbc09c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient(hunting): picked mon.noname-b con 0x7febbc09e970 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient(hunting): picked mon.noname-c con 0x7febbc139a30 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc121ed0 con 0x7febbc09c300 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc1317640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7febbc09e970 legacy=0x7febbc1d0360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc0a3610 con 0x7febbc09e970 2024-09-16T05:39:17.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc120750 con 0x7febbc139a30 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient(hunting): authenticate will time out at 1213.980180s 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc2319640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7febbc139a30 0x7febbc1d3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41188/0 (socket says 172.21.15.63:41188) 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc1b18640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7febbc09c300 0x7febbc1337e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60102/0 (socket says 172.21.15.63:60102) 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc2319640 1 -- 172.21.15.63:0/1240046269 learned_addr learned my addr 172.21.15.63:0/1240046269 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046588560 0 0) 0x7febbc121ed0 con 0x7febbc09c300 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): my global_id is 5868 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feba0003530 con 0x7febbc09c300 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3115082683 0 0) 0x7febbc120750 con 0x7febbc139a30 2024-09-16T05:39:17.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): my global_id is 5846 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febbc121ed0 con 0x7febbc139a30 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1645590413 0 0) 0x7feba0003530 con 0x7febbc09c300 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febbc120750 con 0x7febbc09c300 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febb8001900 con 0x7febbc09c300 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): dump: 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:17.216 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 451223666 0 0) 0x7febbc121ed0 con 0x7febbc139a30 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feba0003530 con 0x7febbc139a30 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febb4001890 con 0x7febbc139a30 2024-09-16T05:39:17.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient(hunting): dump: 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:17.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3421165621 0 0) 0x7febbc120750 con 0x7febbc09c300 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 >> v1:172.21.15.6:6790/0 conn(0x7febbc139a30 legacy=0x7febbc1d3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 >> v1:172.21.15.63:6789/0 conn(0x7febbc09e970 legacy=0x7febbc1d0360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 monclient: found mon.a 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc121bf0 con 0x7febbc09c300 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 5 monclient: authenticate success, global_id 5868 2024-09-16T05:39:17.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient: _renew_subs 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7febbc072670 con 0x7febbc09c300 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient: _renew_subs 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7febbc120a00 con 0x7febbc09c300 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febb80047f0 con 0x7febbc09c300 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febb8004b40 con 0x7febbc09c300 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb3fff640 4 set_mon_vals no callback set 2024-09-16T05:39:17.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 10 monclient: dump: 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:17.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:17.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba4005090 con 0x7febbc09c300 2024-09-16T05:39:17.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7febb8004ce0 con 0x7febbc09c300 2024-09-16T05:39:17.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(166..166 src has 1..166) ==== 9023+0+0 (unknown 2020057737 0 0) 0x7febb80455d0 con 0x7febbc09c300 2024-09-16T05:39:17.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.202+0000 7febc1317640 10 monclient: get_auth_request con 0x7feba0036b10 auth_method 0 2024-09-16T05:39:17.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.218+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7febb80152a0 con 0x7febbc09c300 2024-09-16T05:39:17.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.218+0000 7febb27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:17.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.218+0000 7febb27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:17.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.394+0000 7febc3d7c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T05:39:17.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.394+0000 7febc3d7c640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T05:39:17.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.394+0000 7febc3d7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:17.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:17.394+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) -- 0x7feba4005380 con 0x7febbc09c300 2024-09-16T05:39:18.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:18.082+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v167) ==== 132+0+0 (unknown 1372943697 0 0) 0x7febb8018f50 con 0x7febbc09c300 2024-09-16T05:39:18.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:18.082+0000 7febb27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T05:39:18.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:18.082+0000 7febb27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:18.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:18.186+0000 7febc3d7c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T05:39:18.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:18.186+0000 7febc3d7c640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T05:39:18.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:18.186+0000 7febc3d7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:18.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:18.186+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) -- 0x7feba4001180 con 0x7febbc09c300 2024-09-16T05:39:19.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febb27fc640 1 -- 172.21.15.63:0/1240046269 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v168) ==== 132+0+0 (unknown 1274418496 0 0) 0x7feba4001180 con 0x7febbc09c300 2024-09-16T05:39:19.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febb27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T05:39:19.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febb27fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:19.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 >> v1:172.21.15.63:6812/3643207211 conn(0x7feba0036b10 legacy=0x7feba0038fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febc3d7c640 10 monclient: shutdown 2024-09-16T05:39:19.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febc3d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:19.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 >> v1:172.21.15.6:6789/0 conn(0x7febbc09c300 legacy=0x7febbc1337e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 shutdown_connections 2024-09-16T05:39:19.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 >> 172.21.15.63:0/1240046269 conn(0x7febbc075470 msgr2=0x7febbc0a03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:19.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 shutdown_connections 2024-09-16T05:39:19.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.086+0000 7febc3d7c640 1 -- 172.21.15.63:0/1240046269 wait complete. 2024-09-16T05:39:19.110 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:19.111 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:19.111 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep globster 2024-09-16T05:39:19.111 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '7\.' 2024-09-16T05:39:19.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient: get_monmap_and_config 2024-09-16T05:39:19.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient: build_initial_monmap 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient: monmap: 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:19.234571+0000 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:19.234571+0000 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:19.235 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 1 Processor -- start 2024-09-16T05:39:19.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 1 -- start start 2024-09-16T05:39:19.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient: init 2024-09-16T05:39:19.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:19.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:19.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient(hunting): picked mon.noname-c con 0x7fc65009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:19.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient(hunting): picked mon.noname-b con 0x7fc65006dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:19.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient(hunting): picked mon.noname-a con 0x7fc6500a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc65009f800 con 0x7fc6500a0210 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6500a0ab0 con 0x7fc65009ac90 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6500a0d90 con 0x7fc65006dd30 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc655bda640 10 monclient(hunting): authenticate will time out at 1216.011545s 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc64f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc65009ac90 legacy=0x7fc65009b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc647fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc65006dd30 0x7fc6501399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41190/0 (socket says 172.21.15.63:41190) 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc64ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6500a0210 0x7fc65013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60116/0 (socket says 172.21.15.63:60116) 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.230+0000 7fc647fff640 1 -- 172.21.15.63:0/831600194 learned_addr learned my addr 172.21.15.63:0/831600194 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3850039493 0 0) 0x7fc65009f800 con 0x7fc6500a0210 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:19.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient(hunting): my global_id is 5871 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc63c003530 con 0x7fc6500a0210 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3268928706 0 0) 0x7fc6500a0d90 con 0x7fc65006dd30 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient(hunting): my global_id is 5849 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc65009f800 con 0x7fc65006dd30 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2865382079 0 0) 0x7fc63c003530 con 0x7fc6500a0210 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 >> v1:172.21.15.6:6790/0 conn(0x7fc65006dd30 legacy=0x7fc6501399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 >> v1:172.21.15.63:6789/0 conn(0x7fc65009ac90 legacy=0x7fc65009b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:19.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 monclient: found mon.noname-a 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65009e770 con 0x7fc6500a0210 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: _finish_auth 0 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc6480019a0 con 0x7fc6500a0210 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: dump: 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.242 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 5 monclient: authenticate success, global_id 5871 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc648001c00 con 0x7fc6500a0210 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 1 -- 172.21.15.63:0/831600194 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc648002010 con 0x7fc6500a0210 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:19.243 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 4 set_mon_vals no callback set 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64effd640 10 monclient: dump: 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.244 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient: shutdown 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- 172.21.15.63:0/831600194 >> v1:172.21.15.6:6789/0 conn(0x7fc6500a0210 legacy=0x7fc65013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- 172.21.15.63:0/831600194 shutdown_connections 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- 172.21.15.63:0/831600194 >> 172.21.15.63:0/831600194 conn(0x7fc6500a1c90 msgr2=0x7fc6500a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- 172.21.15.63:0/831600194 shutdown_connections 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- 172.21.15.63:0/831600194 wait complete. 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient: build_initial_monmap 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient: monmap: 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:19.245 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 Processor -- start 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- start start 2024-09-16T05:39:19.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient: init 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient(hunting): picked mon.noname-c con 0x7fc65006dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient(hunting): picked mon.noname-b con 0x7fc65009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient(hunting): picked mon.noname-a con 0x7fc6500a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6501241b0 con 0x7fc6500a0210 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc650122640 con 0x7fc65009ac90 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc650124490 con 0x7fc65006dd30 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc647fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc65009ac90 legacy=0x7fc650133aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:19.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc655bda640 10 monclient(hunting): authenticate will time out at 1216.015529s 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64f7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc65006dd30 0x7fc6501c8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41202/0 (socket says 172.21.15.63:41202) 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6500a0210 0x7fc6501d0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60126/0 (socket says 172.21.15.63:60126) 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.234+0000 7fc64f7fe640 1 -- 172.21.15.63:0/2080415418 learned_addr learned my addr 172.21.15.63:0/2080415418 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 719188100 0 0) 0x7fc6501241b0 con 0x7fc6500a0210 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): my global_id is 5874 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc630003530 con 0x7fc6500a0210 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1109236565 0 0) 0x7fc650124490 con 0x7fc65006dd30 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:19.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): my global_id is 5852 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6501241b0 con 0x7fc65006dd30 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4170202168 0 0) 0x7fc630003530 con 0x7fc6500a0210 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc650124490 con 0x7fc6500a0210 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc648001f50 con 0x7fc6500a0210 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): dump: 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.252 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3459169935 0 0) 0x7fc6501241b0 con 0x7fc65006dd30 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc630003530 con 0x7fc65006dd30 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc6400040e0 con 0x7fc65006dd30 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient(hunting): dump: 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.253 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3931506504 0 0) 0x7fc650124490 con 0x7fc6500a0210 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 >> v1:172.21.15.6:6790/0 conn(0x7fc65006dd30 legacy=0x7fc6501c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 >> v1:172.21.15.63:6789/0 conn(0x7fc65009ac90 legacy=0x7fc650133aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 monclient: found mon.a 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65006e160 con 0x7fc6500a0210 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient: _finish_auth 0 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 5 monclient: authenticate success, global_id 5874 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 10 monclient: _renew_subs 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.255 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc650123ed0 con 0x7fc6500a0210 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 10 monclient: _renew_subs 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc650134380 con 0x7fc6500a0210 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc648001c00 con 0x7fc6500a0210 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc648005060 con 0x7fc6500a0210 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64e7fc640 4 set_mon_vals no callback set 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 10 monclient: dump: 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.256 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc634005090 con 0x7fc6500a0210 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc64800fe00 con 0x7fc6500a0210 2024-09-16T05:39:19.257 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(168..168 src has 1..168) ==== 9023+0+0 (unknown 2914014114 0 0) 0x7fc6480457f0 con 0x7fc6500a0210 2024-09-16T05:39:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.238+0000 7fc64f7fe640 10 monclient: get_auth_request con 0x7fc630036a70 auth_method 0 2024-09-16T05:39:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.250+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc648015600 con 0x7fc6500a0210 2024-09-16T05:39:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.250+0000 7fc64cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:19.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.250+0000 7fc64cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:19.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.418+0000 7fc655bda640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.418+0000 7fc655bda640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.418+0000 7fc655bda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.418+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fc634005380 con 0x7fc6500a0210 2024-09-16T05:39:19.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.422+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v168) ==== 83+0+2320 (unknown 603138176 0 2610836559) 0x7fc64801da50 con 0x7fc6500a0210 2024-09-16T05:39:19.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.422+0000 7fc64cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.422+0000 7fc64cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:19.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.522+0000 7fc655bda640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.522+0000 7fc655bda640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.522+0000 7fc655bda640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.522+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fc634005770 con 0x7fc6500a0210 2024-09-16T05:39:19.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.522+0000 7fc64cff9640 1 -- 172.21.15.63:0/2080415418 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v168) ==== 83+0+2320 (unknown 603138176 0 2610836559) 0x7fc6480405d0 con 0x7fc6500a0210 2024-09-16T05:39:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.522+0000 7fc64cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.522+0000 7fc64cff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.526+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc630036a70 legacy=0x7fc630038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.526+0000 7fc655bda640 10 monclient: shutdown 2024-09-16T05:39:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.526+0000 7fc655bda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.526+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 >> v1:172.21.15.6:6789/0 conn(0x7fc6500a0210 legacy=0x7fc6501d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.526+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 shutdown_connections 2024-09-16T05:39:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.526+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 >> 172.21.15.63:0/2080415418 conn(0x7fc6500a1c90 msgr2=0x7fc6500a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.526+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 shutdown_connections 2024-09-16T05:39:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.526+0000 7fc655bda640 1 -- 172.21.15.63:0/2080415418 wait complete. 2024-09-16T05:39:19.543 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T05:39:19.543 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T05:39:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep globster 2024-09-16T05:39:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '8\.' 2024-09-16T05:39:19.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient: get_monmap_and_config 2024-09-16T05:39:19.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient: build_initial_monmap 2024-09-16T05:39:19.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient: monmap: 2024-09-16T05:39:19.662 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:19.662 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.662 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:19.662219+0000 2024-09-16T05:39:19.663 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:19.662219+0000 2024-09-16T05:39:19.663 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:19.663 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.663 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:19.663 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:19.663 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:19.663 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 1 Processor -- start 2024-09-16T05:39:19.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 1 -- start start 2024-09-16T05:39:19.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient: init 2024-09-16T05:39:19.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient(hunting): picked mon.noname-a con 0x7f05bc0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient(hunting): picked mon.noname-b con 0x7f05bc09b450 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient(hunting): picked mon.noname-c con 0x7f05bc09b8a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05bc09e970 con 0x7f05bc0a08a0 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05bc09c370 con 0x7f05bc09b8a0 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05bc09d620 con 0x7f05bc09b450 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c2aa1640 10 monclient(hunting): authenticate will time out at 1216.439794s 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.658+0000 7f05c103e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f05bc09b8a0 legacy=0x7f05bc13c0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c083d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f05bc0a08a0 0x7f05bc09ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60138/0 (socket says 172.21.15.63:60138) 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c083d640 1 -- 172.21.15.63:0/2330997105 learned_addr learned my addr 172.21.15.63:0/2330997105 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:19.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bbfff640 1 --1- 172.21.15.63:0/2330997105 >> v1:172.21.15.6:6790/0 conn(0x7f05bc09b450 0x7f05bc1389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41210/0 (socket says 172.21.15.63:41210) 2024-09-16T05:39:19.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762644260 0 0) 0x7f05bc09e970 con 0x7f05bc0a08a0 2024-09-16T05:39:19.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:19.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient(hunting): my global_id is 5877 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05ac005740 con 0x7f05bc0a08a0 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887315682 0 0) 0x7f05bc09d620 con 0x7f05bc09b450 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient(hunting): my global_id is 5855 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05bc09e970 con 0x7f05bc09b450 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4254183073 0 0) 0x7f05ac005740 con 0x7f05bc0a08a0 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 >> v1:172.21.15.6:6790/0 conn(0x7f05bc09b450 legacy=0x7f05bc1389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 >> v1:172.21.15.63:6789/0 conn(0x7f05bc09b8a0 legacy=0x7f05bc13c0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05bc09d900 con 0x7f05bc0a08a0 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f05b0001550 con 0x7f05bc0a08a0 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: dump: 2024-09-16T05:39:19.673 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 5 monclient: authenticate success, global_id 5877 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f05b00017b0 con 0x7f05bc0a08a0 2024-09-16T05:39:19.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 1 -- 172.21.15.63:0/2330997105 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f05b0001b40 con 0x7f05bc0a08a0 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05bb7fe640 10 monclient: dump: 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.675 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 4 set_mon_vals no callback set 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient: shutdown 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2330997105 >> v1:172.21.15.6:6789/0 conn(0x7f05bc0a08a0 legacy=0x7f05bc09ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2330997105 shutdown_connections 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2330997105 >> 172.21.15.63:0/2330997105 conn(0x7f05bc0a1c90 msgr2=0x7f05bc0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2330997105 shutdown_connections 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2330997105 wait complete. 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient: build_initial_monmap 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient: monmap: 2024-09-16T05:39:19.676 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 Processor -- start 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- start start 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient: init 2024-09-16T05:39:19.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient(hunting): picked mon.noname-b con 0x7f05bc09b450 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient(hunting): picked mon.noname-c con 0x7f05bc09b8a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient(hunting): picked mon.noname-a con 0x7f05bc0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05bc0a3ed0 con 0x7f05bc0a08a0 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05bc126350 con 0x7f05bc09b450 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05bc126910 con 0x7f05bc09b8a0 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c083d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f05bc09b450 legacy=0x7f05bc122980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.662+0000 7f05c2aa1640 10 monclient(hunting): authenticate will time out at 1216.443782s 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05bbfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f05bc09b8a0 0x7f05bc123090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41218/0 (socket says 172.21.15.63:41218) 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05bbfff640 1 -- 172.21.15.63:0/2106852562 learned_addr learned my addr 172.21.15.63:0/2106852562 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:19.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 336687410 0 0) 0x7f05bc126910 con 0x7f05bc09b8a0 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): my global_id is 5858 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05a0003560 con 0x7f05bc09b8a0 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477913385 0 0) 0x7f05b4003d40 con 0x7f05bc0a08a0 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): my global_id is 5880 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05bc126910 con 0x7f05bc0a08a0 2024-09-16T05:39:19.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 902195813 0 0) 0x7f05bc126910 con 0x7f05bc0a08a0 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05b4003d40 con 0x7f05bc0a08a0 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f05b4004450 con 0x7f05bc0a08a0 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): dump: 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.680 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3746115242 0 0) 0x7f05a0003560 con 0x7f05bc09b8a0 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05bc126910 con 0x7f05bc09b8a0 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f05a8001850 con 0x7f05bc09b8a0 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient(hunting): dump: 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.681 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2036487503 0 0) 0x7f05b4003d40 con 0x7f05bc0a08a0 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 >> v1:172.21.15.6:6790/0 conn(0x7f05bc09b8a0 legacy=0x7f05bc123090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 >> v1:172.21.15.63:6789/0 conn(0x7f05bc09b450 legacy=0x7f05bc122980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 monclient: found mon.a 2024-09-16T05:39:19.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05bc0a43e0 con 0x7f05bc0a08a0 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 5 monclient: authenticate success, global_id 5880 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 10 monclient: _renew_subs 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05bc126070 con 0x7f05bc0a08a0 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 10 monclient: _renew_subs 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f05bc124f00 con 0x7f05bc0a08a0 2024-09-16T05:39:19.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f05b4004960 con 0x7f05bc0a08a0 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f05b4004cb0 con 0x7f05bc0a08a0 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 10 monclient: dump: 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:19.684 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05baffd640 4 set_mon_vals no callback set 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f059c005090 con 0x7f05bc0a08a0 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f05b4004e50 con 0x7f05bc0a08a0 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(168..168 src has 1..168) ==== 9023+0+0 (unknown 2914014114 0 0) 0x7f05b4045820 con 0x7f05bc0a08a0 2024-09-16T05:39:19.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.666+0000 7f05c083d640 10 monclient: get_auth_request con 0x7f05a0036b40 auth_method 0 2024-09-16T05:39:19.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.678+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f05bc0a3ed0 con 0x7f05bc0a08a0 2024-09-16T05:39:19.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.678+0000 7f05b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:19.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.678+0000 7f05b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:19.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.854+0000 7f05c2aa1640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.854+0000 7f05c2aa1640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.854+0000 7f05c2aa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.857 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.854+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f059c005380 con 0x7f05bc0a08a0 2024-09-16T05:39:19.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.854+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v168) ==== 83+0+2320 (unknown 603138176 0 2610836559) 0x7f05b40155d0 con 0x7f05bc0a08a0 2024-09-16T05:39:19.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.854+0000 7f05b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.858 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.854+0000 7f05b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:19.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.958+0000 7f05c2aa1640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.958+0000 7f05c2aa1640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.958+0000 7f05c2aa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:19.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.958+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f059c005770 con 0x7f05bc0a08a0 2024-09-16T05:39:19.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.958+0000 7f05b97fa640 1 -- 172.21.15.63:0/2106852562 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v168) ==== 83+0+2320 (unknown 603138176 0 2610836559) 0x7f05b4014ed0 con 0x7f05bc0a08a0 2024-09-16T05:39:19.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.958+0000 7f05b97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:19.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.958+0000 7f05b97fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:19.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.962+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 >> v1:172.21.15.63:6812/3643207211 conn(0x7f05a0036b40 legacy=0x7f05a0039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.962+0000 7f05c2aa1640 10 monclient: shutdown 2024-09-16T05:39:19.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.962+0000 7f05c2aa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:19.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.962+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 >> v1:172.21.15.6:6789/0 conn(0x7f05bc0a08a0 legacy=0x7f05bc1237a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:19.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.962+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 shutdown_connections 2024-09-16T05:39:19.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.962+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 >> 172.21.15.63:0/2106852562 conn(0x7f05bc0a1c90 msgr2=0x7f05bc1b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:19.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.962+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 shutdown_connections 2024-09-16T05:39:19.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:19.962+0000 7f05c2aa1640 1 -- 172.21.15.63:0/2106852562 wait complete. 2024-09-16T05:39:19.979 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T05:39:19.980 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T05:39:19.980 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T05:39:19.981 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep 'osd\.0' 2024-09-16T05:39:19.981 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep globster 2024-09-16T05:39:19.981 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep '6\.' 2024-09-16T05:39:20.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.098+0000 7f7da4d3f640 10 monclient: get_monmap_and_config 2024-09-16T05:39:20.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.098+0000 7f7da4d3f640 10 monclient: build_initial_monmap 2024-09-16T05:39:20.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.098+0000 7f7da4d3f640 10 monclient: monmap: 2024-09-16T05:39:20.103 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:20.103 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.103 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:20.103195+0000 2024-09-16T05:39:20.103 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:20.103195+0000 2024-09-16T05:39:20.104 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:20.104 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.104 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:20.104 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:20.104 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:20.104 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.098+0000 7f7da4d3f640 1 Processor -- start 2024-09-16T05:39:20.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.098+0000 7f7da4d3f640 1 -- start start 2024-09-16T05:39:20.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.098+0000 7f7da4d3f640 10 monclient: init 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 10 monclient(hunting): picked mon.noname-a con 0x7f7da00a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 10 monclient(hunting): picked mon.noname-b con 0x7f7da00a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 10 monclient(hunting): picked mon.noname-c con 0x7f7da00a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da00a4fd0 con 0x7f7da00a3ca0 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da00a5b40 con 0x7f7da00a4790 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da00a1a70 con 0x7f7da00a08a0 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7da4d3f640 10 monclient(hunting): authenticate will time out at 1216.880146s 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9ed9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7da00a4790 legacy=0x7f7da013f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9dd9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7da00a08a0 0x7f7da013beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41224/0 (socket says 172.21.15.63:41224) 2024-09-16T05:39:20.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9e59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7da00a3ca0 0x7f7da00a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60158/0 (socket says 172.21.15.63:60158) 2024-09-16T05:39:20.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9dd9b640 1 -- 172.21.15.63:0/1760517195 learned_addr learned my addr 172.21.15.63:0/1760517195 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:20.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006757261 0 0) 0x7f7da00a1a70 con 0x7f7da00a08a0 2024-09-16T05:39:20.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:20.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:20.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:20.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 10 monclient(hunting): my global_id is 5861 2024-09-16T05:39:20.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d88003530 con 0x7f7da00a08a0 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979429982 0 0) 0x7f7da00a4fd0 con 0x7f7da00a3ca0 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 10 monclient(hunting): my global_id is 5883 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.102+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7da00a1a70 con 0x7f7da00a3ca0 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2246810719 0 0) 0x7f7da00a1a70 con 0x7f7da00a3ca0 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 >> v1:172.21.15.6:6790/0 conn(0x7f7da00a08a0 legacy=0x7f7da013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 >> v1:172.21.15.63:6789/0 conn(0x7f7da00a4790 legacy=0x7f7da013f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 1 monclient: found mon.noname-a 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da00a53e0 con 0x7f7da00a3ca0 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: _finish_auth 0 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d90001600 con 0x7f7da00a3ca0 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: dump: 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.110 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.111 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.111 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.111 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.111 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.111 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.111 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7da4d3f640 5 monclient: authenticate success, global_id 5883 2024-09-16T05:39:20.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7da4d3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:20.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d90001860 con 0x7f7da00a3ca0 2024-09-16T05:39:20.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:20.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 1 -- 172.21.15.63:0/1760517195 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d90001bf0 con 0x7f7da00a3ca0 2024-09-16T05:39:20.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:20.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:20.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7d9d59a640 10 monclient: dump: 2024-09-16T05:39:20.112 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.112 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7da4d3f640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:20.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.106+0000 7f7da4d3f640 4 set_mon_vals no callback set 2024-09-16T05:39:20.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: shutdown 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- 172.21.15.63:0/1760517195 >> v1:172.21.15.6:6789/0 conn(0x7f7da00a3ca0 legacy=0x7f7da00a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- 172.21.15.63:0/1760517195 shutdown_connections 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- 172.21.15.63:0/1760517195 >> 172.21.15.63:0/1760517195 conn(0x7f7da00a67c0 msgr2=0x7f7da0075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- 172.21.15.63:0/1760517195 shutdown_connections 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- 172.21.15.63:0/1760517195 wait complete. 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: build_initial_monmap 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: monmap: 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:20.118 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 Processor -- start 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- start start 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: init 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient(hunting): picked mon.noname-b con 0x7f7da00a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:20.119 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient(hunting): picked mon.noname-c con 0x7f7da00a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient(hunting): picked mon.noname-a con 0x7f7da00a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d9e59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7da00a08a0 legacy=0x7f7da0121d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da01256d0 con 0x7f7da00a4790 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da0122b70 con 0x7f7da00a08a0 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da0124500 con 0x7f7da00a3ca0 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient(hunting): authenticate will time out at 1216.890137s 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d9dd9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7da00a3ca0 0x7f7da0122460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41236/0 (socket says 172.21.15.63:41236) 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d9dd9b640 1 -- 172.21.15.63:0/147390146 learned_addr learned my addr 172.21.15.63:0/147390146 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 70622317 0 0) 0x7f7da01256d0 con 0x7f7da00a4790 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:20.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): my global_id is 5886 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d80003560 con 0x7f7da00a4790 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3140605575 0 0) 0x7f7da0124500 con 0x7f7da00a3ca0 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): my global_id is 5864 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7da01256d0 con 0x7f7da00a3ca0 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1567681917 0 0) 0x7f7d80003560 con 0x7f7da00a4790 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7da0124500 con 0x7f7da00a4790 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d98004340 con 0x7f7da00a4790 2024-09-16T05:39:20.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): dump: 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.122 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2392517262 0 0) 0x7f7da01256d0 con 0x7f7da00a3ca0 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d80003560 con 0x7f7da00a3ca0 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d940017a0 con 0x7f7da00a3ca0 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient(hunting): dump: 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.123 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1205594519 0 0) 0x7f7da0124500 con 0x7f7da00a4790 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 >> v1:172.21.15.6:6790/0 conn(0x7f7da00a3ca0 legacy=0x7f7da0122460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 >> v1:172.21.15.63:6789/0 conn(0x7f7da00a08a0 legacy=0x7f7da0121d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 monclient: found mon.a 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da0124280 con 0x7f7da00a4790 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient: _finish_auth 0 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7d8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 5 monclient: authenticate success, global_id 5886 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: _renew_subs 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7da00a8720 con 0x7f7da00a4790 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: _renew_subs 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.110+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7da01253f0 con 0x7f7da00a4790 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d980045a0 con 0x7f7da00a4790 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d8effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:20.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7d98004970 con 0x7f7da00a4790 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d9cd99640 4 set_mon_vals no callback set 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d8effd640 10 monclient: dump: 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.126 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7da4d3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7da4d3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7da4d3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d84005090 con 0x7f7da00a4790 2024-09-16T05:39:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7d98004b90 con 0x7f7da00a4790 2024-09-16T05:39:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(168..168 src has 1..168) ==== 9023+0+0 (unknown 2914014114 0 0) 0x7f7d98045840 con 0x7f7da00a4790 2024-09-16T05:39:20.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.114+0000 7f7d9e59c640 10 monclient: get_auth_request con 0x7f7d80036a50 auth_method 0 2024-09-16T05:39:20.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.126+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7d980155d0 con 0x7f7da00a4790 2024-09-16T05:39:20.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.126+0000 7f7d8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:20.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.126+0000 7f7d8effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:20.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.294+0000 7f7da4d3f640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:20.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.294+0000 7f7da4d3f640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:20.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.294+0000 7f7da4d3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.294+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f7d84005380 con 0x7f7da00a4790 2024-09-16T05:39:20.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.298+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v168) ==== 83+0+2320 (unknown 603138176 0 2610836559) 0x7f7d98014ed0 con 0x7f7da00a4790 2024-09-16T05:39:20.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.298+0000 7f7d8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:20.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.298+0000 7f7d8effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:20.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.402+0000 7f7da4d3f640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:20.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.402+0000 7f7da4d3f640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:20.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.402+0000 7f7da4d3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.402+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f7d84005770 con 0x7f7da00a4790 2024-09-16T05:39:20.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.402+0000 7f7d8effd640 1 -- 172.21.15.63:0/147390146 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v168) ==== 83+0+2320 (unknown 603138176 0 2610836559) 0x7f7d9801da80 con 0x7f7da00a4790 2024-09-16T05:39:20.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.402+0000 7f7d8effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T05:39:20.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.402+0000 7f7d8effd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:20.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.406+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7d80036a50 legacy=0x7f7d80038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:20.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.406+0000 7f7da4d3f640 10 monclient: shutdown 2024-09-16T05:39:20.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.406+0000 7f7da4d3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:20.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.406+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 >> v1:172.21.15.6:6789/0 conn(0x7f7da00a4790 legacy=0x7f7da01d38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:20.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.406+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 shutdown_connections 2024-09-16T05:39:20.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.406+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 >> 172.21.15.63:0/147390146 conn(0x7f7da00a67c0 msgr2=0x7f7da01bcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:20.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.406+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 shutdown_connections 2024-09-16T05:39:20.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.406+0000 7f7da4d3f640 1 -- 172.21.15.63:0/147390146 wait complete. 2024-09-16T05:39:20.423 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T05:39:20.423 INFO:tasks.workunit.client.0.smithi063.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T05:39:20.424 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T05:39:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient: get_monmap_and_config 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient: build_initial_monmap 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient: monmap: 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:20.545751+0000 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:20.545751+0000 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:20.546 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 1 Processor -- start 2024-09-16T05:39:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 1 -- start start 2024-09-16T05:39:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient: init 2024-09-16T05:39:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient(hunting): picked mon.noname-b con 0x7fea7c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient(hunting): picked mon.noname-a con 0x7fea7c06dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient(hunting): picked mon.noname-c con 0x7fea7c0a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c09f800 con 0x7fea7c06dd30 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c0a0ab0 con 0x7fea7c0a0210 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c0a0d90 con 0x7fea7c09ac90 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea83881640 10 monclient(hunting): authenticate will time out at 1217.322715s 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea81e1e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fea7c0a0210 legacy=0x7fea7c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea80e1c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fea7c06dd30 0x7fea7c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60172/0 (socket says 172.21.15.63:60172) 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea8161d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fea7c09ac90 0x7fea7c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41238/0 (socket says 172.21.15.63:41238) 2024-09-16T05:39:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea80e1c640 1 -- 172.21.15.63:0/2351960501 learned_addr learned my addr 172.21.15.63:0/2351960501 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3238850000 0 0) 0x7fea7c0a0d90 con 0x7fea7c09ac90 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 10 monclient(hunting): my global_id is 5867 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea68003530 con 0x7fea7c09ac90 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443243394 0 0) 0x7fea7c09f800 con 0x7fea7c06dd30 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 10 monclient(hunting): my global_id is 5889 2024-09-16T05:39:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.542+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea7c0a0d90 con 0x7fea7c06dd30 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3958425932 0 0) 0x7fea7c0a0d90 con 0x7fea7c06dd30 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 >> v1:172.21.15.6:6790/0 conn(0x7fea7c09ac90 legacy=0x7fea7c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 >> v1:172.21.15.63:6789/0 conn(0x7fea7c0a0210 legacy=0x7fea7c13d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 1 monclient: found mon.noname-a 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c09e770 con 0x7fea7c06dd30 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: _finish_auth 0 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea6c001600 con 0x7fea7c06dd30 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: dump: 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.553 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 5 monclient: authenticate success, global_id 5889 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea6c001860 con 0x7fea7c06dd30 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 1 -- 172.21.15.63:0/2351960501 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea6c001cb0 con 0x7fea7c06dd30 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea73fff640 10 monclient: dump: 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.555 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 4 set_mon_vals no callback set 2024-09-16T05:39:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient: shutdown 2024-09-16T05:39:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- 172.21.15.63:0/2351960501 >> v1:172.21.15.6:6789/0 conn(0x7fea7c06dd30 legacy=0x7fea7c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- 172.21.15.63:0/2351960501 shutdown_connections 2024-09-16T05:39:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- 172.21.15.63:0/2351960501 >> 172.21.15.63:0/2351960501 conn(0x7fea7c0a1c90 msgr2=0x7fea7c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- 172.21.15.63:0/2351960501 shutdown_connections 2024-09-16T05:39:20.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- 172.21.15.63:0/2351960501 wait complete. 2024-09-16T05:39:20.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient: build_initial_monmap 2024-09-16T05:39:20.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient: monmap: 2024-09-16T05:39:20.560 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:20.560 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:20.560 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:20.560 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:20.560 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:20.560 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 Processor -- start 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- start start 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient: init 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient(hunting): picked mon.noname-b con 0x7fea7c06dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient(hunting): picked mon.noname-a con 0x7fea7c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:20.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient(hunting): picked mon.noname-c con 0x7fea7c0a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea8161d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fea7c06dd30 legacy=0x7fea7c133b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c09c0f0 con 0x7fea7c09ac90 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c123f50 con 0x7fea7c06dd30 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c1226c0 con 0x7fea7c0a0210 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea83881640 10 monclient(hunting): authenticate will time out at 1217.327107s 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea80e1c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fea7c09ac90 0x7fea7c1cdf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60176/0 (socket says 172.21.15.63:60176) 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea80e1c640 1 -- 172.21.15.63:0/1319553201 learned_addr learned my addr 172.21.15.63:0/1319553201 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282083864 0 0) 0x7fea7c09e770 con 0x7fea7c09ac90 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 10 monclient(hunting): my global_id is 5892 2024-09-16T05:39:20.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea60003530 con 0x7fea7c09ac90 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469137667 0 0) 0x7fea7c1226c0 con 0x7fea7c0a0210 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 10 monclient(hunting): my global_id is 5870 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.546+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea7c09e770 con 0x7fea7c0a0210 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1072969046 0 0) 0x7fea60003530 con 0x7fea7c09ac90 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea7c1226c0 con 0x7fea7c09ac90 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea6c001bf0 con 0x7fea7c09ac90 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient(hunting): dump: 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.563 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2779409198 0 0) 0x7fea7c09e770 con 0x7fea7c0a0210 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea60003530 con 0x7fea7c0a0210 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea74004340 con 0x7fea7c0a0210 2024-09-16T05:39:20.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient(hunting): dump: 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.565 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4010763335 0 0) 0x7fea7c1226c0 con 0x7fea7c09ac90 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 >> v1:172.21.15.6:6790/0 conn(0x7fea7c0a0210 legacy=0x7fea7c1d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 >> v1:172.21.15.63:6789/0 conn(0x7fea7c06dd30 legacy=0x7fea7c133b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 monclient: found mon.a 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c06e640 con 0x7fea7c09ac90 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:20.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 5 monclient: authenticate success, global_id 5892 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 10 monclient: _renew_subs 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fea7c124230 con 0x7fea7c09ac90 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 10 monclient: _renew_subs 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fea7c134400 con 0x7fea7c09ac90 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea6c001860 con 0x7fea7c09ac90 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fea6c004c40 con 0x7fea7c09ac90 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:20.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 10 monclient: dump: 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea72ffd640 4 set_mon_vals no callback set 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:20.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:20.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea5c005090 con 0x7fea7c09ac90 2024-09-16T05:39:20.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fea6c004e60 con 0x7fea7c09ac90 2024-09-16T05:39:20.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(168..168 src has 1..168) ==== 9023+0+0 (unknown 2914014114 0 0) 0x7fea6c045610 con 0x7fea7c09ac90 2024-09-16T05:39:20.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.550+0000 7fea8161d640 10 monclient: get_auth_request con 0x7fea600369d0 auth_method 0 2024-09-16T05:39:20.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.562+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fea7c09c0f0 con 0x7fea7c09ac90 2024-09-16T05:39:20.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.562+0000 7fea71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:20.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.562+0000 7fea71ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:20.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.730+0000 7fea83881640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:20.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.730+0000 7fea83881640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:20.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.730+0000 7fea83881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:20.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:20.730+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7fea5c005b90 con 0x7fea7c09ac90 2024-09-16T05:39:21.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:21.030+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v169) ==== 123+0+0 (unknown 2305552966 0 0) 0x7fea6c015400 con 0x7fea7c09ac90 2024-09-16T05:39:21.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:21.030+0000 7fea71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:21.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:21.030+0000 7fea71ffb640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T05:39:21.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:21.130+0000 7fea83881640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:21.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:21.130+0000 7fea83881640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:21.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:21.130+0000 7fea83881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:21.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:21.130+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7fea5c005610 con 0x7fea7c09ac90 2024-09-16T05:39:22.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.118+0000 7fea71ffb640 1 -- 172.21.15.63:0/1319553201 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v170) ==== 149+0+0 (unknown 3617173280 0 0) 0x7fea6c014d00 con 0x7fea7c09ac90 2024-09-16T05:39:22.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.118+0000 7fea71ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T05:39:22.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.118+0000 7fea71ffb640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T05:39:22.122 INFO:tasks.workunit.client.0.smithi063.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T05:39:22.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.122+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 >> v1:172.21.15.63:6812/3643207211 conn(0x7fea600369d0 legacy=0x7fea60038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:22.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.122+0000 7fea83881640 10 monclient: shutdown 2024-09-16T05:39:22.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.122+0000 7fea83881640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:22.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.122+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 >> v1:172.21.15.6:6789/0 conn(0x7fea7c09ac90 legacy=0x7fea7c1cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:22.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.122+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 shutdown_connections 2024-09-16T05:39:22.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.122+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 >> 172.21.15.63:0/1319553201 conn(0x7fea7c0a1c90 msgr2=0x7fea7c134f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:22.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.122+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 shutdown_connections 2024-09-16T05:39:22.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.122+0000 7fea83881640 1 -- 172.21.15.63:0/1319553201 wait complete. 2024-09-16T05:39:22.144 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool rm cool cool --yes-i-really-really-mean-it 2024-09-16T05:39:22.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient: get_monmap_and_config 2024-09-16T05:39:22.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient: build_initial_monmap 2024-09-16T05:39:22.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient: monmap: 2024-09-16T05:39:22.270 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:22.270 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:22.270 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:22.270013+0000 2024-09-16T05:39:22.270 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:22.270013+0000 2024-09-16T05:39:22.271 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:22.271 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:22.271 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:22.271 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:22.271 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:22.271 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:22.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 1 Processor -- start 2024-09-16T05:39:22.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 1 -- start start 2024-09-16T05:39:22.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient: init 2024-09-16T05:39:22.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient(hunting): picked mon.noname-a con 0x7f239c134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient(hunting): picked mon.noname-c con 0x7f239c0a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient(hunting): picked mon.noname-b con 0x7f239c06dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c135420 con 0x7f239c134930 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c09e830 con 0x7f239c0a0810 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c0a0dc0 con 0x7f239c06dfa0 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f23a1f9b640 10 monclient(hunting): authenticate will time out at 1219.047409s 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f239affd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f239c0a0810 legacy=0x7f239c139820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:22.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f239b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f239c134930 0x7f239c134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60182/0 (socket says 172.21.15.63:60182) 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.266+0000 7f239b7fe640 1 -- 172.21.15.63:0/360778672 learned_addr learned my addr 172.21.15.63:0/360778672 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153447283 0 0) 0x7f239c135420 con 0x7f239c134930 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient(hunting): my global_id is 5895 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2384003530 con 0x7f239c134930 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3360291159 0 0) 0x7f239c0a0dc0 con 0x7f239c06dfa0 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:22.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient(hunting): my global_id is 5873 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f239c135420 con 0x7f239c06dfa0 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1880602041 0 0) 0x7f2384003530 con 0x7f239c134930 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 >> v1:172.21.15.6:6790/0 conn(0x7f239c06dfa0 legacy=0x7f239c13cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 >> v1:172.21.15.63:6789/0 conn(0x7f239c0a0810 legacy=0x7f239c139820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 monclient: found mon.noname-a 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c0a2500 con 0x7f239c134930 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:22.277 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f238c001620 con 0x7f239c134930 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: dump: 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:22.278 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 5 monclient: authenticate success, global_id 5895 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f238c001880 con 0x7f239c134930 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 1 -- 172.21.15.63:0/360778672 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f238c001c90 con 0x7f239c134930 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:22.279 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239a7fc640 10 monclient: dump: 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:22.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 4 set_mon_vals no callback set 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient: shutdown 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- 172.21.15.63:0/360778672 >> v1:172.21.15.6:6789/0 conn(0x7f239c134930 legacy=0x7f239c134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- 172.21.15.63:0/360778672 shutdown_connections 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- 172.21.15.63:0/360778672 >> 172.21.15.63:0/360778672 conn(0x7f239c0a1e90 msgr2=0x7f239c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- 172.21.15.63:0/360778672 shutdown_connections 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- 172.21.15.63:0/360778672 wait complete. 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient: build_initial_monmap 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient: monmap: 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:22.281 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:22.282 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:22.282 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:22.282 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:22.282 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:22.282 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:22.282 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 Processor -- start 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- start start 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient: init 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient(hunting): picked mon.noname-b con 0x7f239c06dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient(hunting): picked mon.noname-a con 0x7f239c0a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient(hunting): picked mon.noname-c con 0x7f239c134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:22.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c1c6630 con 0x7f239c0a0810 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c13e3c0 con 0x7f239c06dfa0 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f239c06dfa0 legacy=0x7f239c13d560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c09c0f0 con 0x7f239c134930 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f23a1f9b640 10 monclient(hunting): authenticate will time out at 1219.051254s 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f239c0a0810 0x7f239c1cdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60186/0 (socket says 172.21.15.63:60186) 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.270+0000 7f239affd640 1 -- 172.21.15.63:0/255940771 learned_addr learned my addr 172.21.15.63:0/255940771 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675070554 0 0) 0x7f239c09c0f0 con 0x7f239c134930 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:22.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): my global_id is 5876 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2388003560 con 0x7f239c134930 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1068771647 0 0) 0x7f239c1c6630 con 0x7f239c0a0810 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): my global_id is 5898 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f239c09c0f0 con 0x7f239c0a0810 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2511993109 0 0) 0x7f2388003560 con 0x7f239c134930 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f239c1c6630 con 0x7f239c134930 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2394001c00 con 0x7f239c134930 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:22.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): dump: 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:22.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2028828518 0 0) 0x7f239c09c0f0 con 0x7f239c0a0810 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2388003560 con 0x7f239c0a0810 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23900040e0 con 0x7f239c0a0810 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient(hunting): dump: 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:22.290 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 165935340 0 0) 0x7f239c1c6630 con 0x7f239c134930 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 >> v1:172.21.15.63:6789/0 conn(0x7f239c06dfa0 legacy=0x7f239c13d560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 >> v1:172.21.15.6:6789/0 conn(0x7f239c0a0810 legacy=0x7f239c1cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 monclient: found mon.c 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c137750 con 0x7f239c134930 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient: _finish_auth 0 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:22.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 5 monclient: authenticate success, global_id 5876 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 10 monclient: _renew_subs 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f239c13dc70 con 0x7f239c134930 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 10 monclient: _renew_subs 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f239c136860 con 0x7f239c134930 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2394003980 con 0x7f239c134930 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2394003cd0 con 0x7f239c134930 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:22.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 10 monclient: dump: 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23997fa640 4 set_mon_vals no callback set 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:22.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f237c005090 con 0x7f239c134930 2024-09-16T05:39:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f239400ea70 con 0x7f239c134930 2024-09-16T05:39:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(170..170 src has 1..170) ==== 8334+0+0 (unknown 4140450557 0 0) 0x7f23940444e0 con 0x7f239c134930 2024-09-16T05:39:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.274+0000 7f239b7fe640 10 monclient: get_auth_request con 0x7f2388036aa0 auth_method 0 2024-09-16T05:39:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.286+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f23940144c0 con 0x7f239c134930 2024-09-16T05:39:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.286+0000 7f2383fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:22.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.286+0000 7f2383fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:22.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.454+0000 7f23a1f9b640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:22.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.454+0000 7f23a1f9b640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:22.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.454+0000 7f23a1f9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:22.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:22.454+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) -- 0x7f237c005380 con 0x7f239c134930 2024-09-16T05:39:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.130+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' removed v171) ==== 148+0+0 (unknown 3004671035 0 0) 0x7f2394013dc0 con 0x7f239c134930 2024-09-16T05:39:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.130+0000 7f2383fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:23.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.130+0000 7f2383fff640 10 monclient: _finish_command 2 = system:0 pool 'cool' removed 2024-09-16T05:39:23.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.226+0000 7f23a1f9b640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:23.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.226+0000 7f23a1f9b640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:23.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.226+0000 7f23a1f9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:23.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.226+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) -- 0x7f237c001150 con 0x7f239c134930 2024-09-16T05:39:23.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.230+0000 7f2383fff640 1 -- 172.21.15.63:0/255940771 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' does not exist v171) ==== 155+0+0 (unknown 2069572265 0 0) 0x7f2394018110 con 0x7f239c134930 2024-09-16T05:39:23.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.230+0000 7f2383fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:23.234 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.230+0000 7f2383fff640 10 monclient: _finish_command 3 = system:0 pool 'cool' does not exist 2024-09-16T05:39:23.234 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'cool' does not exist 2024-09-16T05:39:23.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.234+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2388036aa0 legacy=0x7f2388038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:23.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.234+0000 7f23a1f9b640 10 monclient: shutdown 2024-09-16T05:39:23.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.234+0000 7f23a1f9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:23.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.234+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 >> v1:172.21.15.6:6790/0 conn(0x7f239c134930 legacy=0x7f239c1d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:23.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.234+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 shutdown_connections 2024-09-16T05:39:23.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.234+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 >> 172.21.15.63:0/255940771 conn(0x7f239c0a1e90 msgr2=0x7f239c09f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:23.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.234+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 shutdown_connections 2024-09-16T05:39:23.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.234+0000 7f23a1f9b640 1 -- 172.21.15.63:0/255940771 wait complete. 2024-09-16T05:39:23.252 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd pool rm cold cold --yes-i-really-really-mean-it 2024-09-16T05:39:23.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient: get_monmap_and_config 2024-09-16T05:39:23.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient: build_initial_monmap 2024-09-16T05:39:23.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient: monmap: 2024-09-16T05:39:23.377 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:23.377 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:23.377 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:23.377271+0000 2024-09-16T05:39:23.378 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:23.377271+0000 2024-09-16T05:39:23.378 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:23.378 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:23.378 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:23.378 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:23.378 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:23.378 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:23.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 1 Processor -- start 2024-09-16T05:39:23.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 1 -- start start 2024-09-16T05:39:23.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient: init 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient(hunting): picked mon.noname-a con 0x7f0b80134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient(hunting): picked mon.noname-b con 0x7f0b800a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient(hunting): picked mon.noname-c con 0x7f0b8006dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b80135420 con 0x7f0b80134930 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b8009e830 con 0x7f0b8006dfa0 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b800a0dc0 con 0x7f0b800a0810 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 10 monclient(hunting): authenticate will time out at 1220.154381s 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0b8006dfa0 legacy=0x7f0b8013cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:23.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0b800a0810 0x7f0b80139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41266/0 (socket says 172.21.15.63:41266) 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b80134930 0x7f0b80134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60200/0 (socket says 172.21.15.63:60200) 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7effd640 1 -- 172.21.15.63:0/2585399571 learned_addr learned my addr 172.21.15.63:0/2585399571 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3014914679 0 0) 0x7f0b800a0dc0 con 0x7f0b800a0810 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient(hunting): my global_id is 5879 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b68003530 con 0x7f0b800a0810 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613249469 0 0) 0x7f0b80135420 con 0x7f0b80134930 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:23.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient(hunting): my global_id is 5901 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b800a0dc0 con 0x7f0b80134930 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2930147634 0 0) 0x7f0b68003530 con 0x7f0b800a0810 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 >> v1:172.21.15.63:6789/0 conn(0x7f0b8006dfa0 legacy=0x7f0b8013cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 >> v1:172.21.15.6:6789/0 conn(0x7f0b80134930 legacy=0x7f0b80134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 monclient: found mon.noname-b 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b800a2500 con 0x7f0b800a0810 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:23.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:23.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b74001620 con 0x7f0b800a0810 2024-09-16T05:39:23.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:23.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:23.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b7e7fc640 10 monclient: dump: 2024-09-16T05:39:23.384 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:23.384 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:23.384 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 5 monclient: authenticate success, global_id 5879 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.374+0000 7f0b85d8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b74001940 con 0x7f0b800a0810 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7e7fc640 1 -- 172.21.15.63:0/2585399571 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b74001d10 con 0x7f0b800a0810 2024-09-16T05:39:23.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7e7fc640 10 monclient: dump: 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:23.386 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:23.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:23.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 4 set_mon_vals no callback set 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: shutdown 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2585399571 >> v1:172.21.15.6:6790/0 conn(0x7f0b800a0810 legacy=0x7f0b80139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2585399571 shutdown_connections 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2585399571 >> 172.21.15.63:0/2585399571 conn(0x7f0b800a1e90 msgr2=0x7f0b80075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2585399571 shutdown_connections 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2585399571 wait complete. 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: build_initial_monmap 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: monmap: 2024-09-16T05:39:23.392 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 Processor -- start 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- start start 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: init 2024-09-16T05:39:23.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient(hunting): picked mon.noname-c con 0x7f0b8006dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient(hunting): picked mon.noname-a con 0x7f0b800a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient(hunting): picked mon.noname-b con 0x7f0b80134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b80134110 con 0x7f0b800a0810 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b801350e0 con 0x7f0b80134930 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b80132fa0 con 0x7f0b8006dfa0 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient(hunting): authenticate will time out at 1220.158198s 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0b80134930 legacy=0x7f0b80131890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7f7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0b8006dfa0 0x7f0b80130a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:41282/0 (socket says 172.21.15.63:41282) 2024-09-16T05:39:23.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b800a0810 0x7f0b80131180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:60208/0 (socket says 172.21.15.63:60208) 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b7f7fe640 1 -- 172.21.15.63:0/2727049068 learned_addr learned my addr 172.21.15.63:0/2727049068 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265106342 0 0) 0x7f0b80132fa0 con 0x7f0b8006dfa0 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): my global_id is 5882 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b6c003530 con 0x7f0b8006dfa0 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223895846 0 0) 0x7f0b80134110 con 0x7f0b800a0810 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): my global_id is 5904 2024-09-16T05:39:23.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b80132fa0 con 0x7f0b800a0810 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 211719802 0 0) 0x7f0b6c003530 con 0x7f0b8006dfa0 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b80134110 con 0x7f0b8006dfa0 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b70001920 con 0x7f0b8006dfa0 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): dump: 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:23.396 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 899975665 0 0) 0x7f0b80132fa0 con 0x7f0b800a0810 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b6c003530 con 0x7f0b800a0810 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b74001440 con 0x7f0b800a0810 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient(hunting): dump: 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:23.397 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3351596509 0 0) 0x7f0b80134110 con 0x7f0b8006dfa0 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 >> v1:172.21.15.63:6789/0 conn(0x7f0b80134930 legacy=0x7f0b80131890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 >> v1:172.21.15.6:6789/0 conn(0x7f0b800a0810 legacy=0x7f0b80131180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:23.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 monclient: found mon.c 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b80132b60 con 0x7f0b8006dfa0 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient: _finish_auth 0 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 5 monclient: authenticate success, global_id 5882 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: _renew_subs 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.378+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b801326c0 con 0x7f0b8006dfa0 2024-09-16T05:39:23.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b85d8e640 10 monclient: _renew_subs 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b85d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b801d29a0 con 0x7f0b8006dfa0 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b70001c80 con 0x7f0b8006dfa0 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b67fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0b70004c40 con 0x7f0b8006dfa0 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b7dffb640 4 set_mon_vals no callback set 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b67fff640 10 monclient: dump: 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:23.400 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b85d8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b85d8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:23.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b85d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:23.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b60005090 con 0x7f0b8006dfa0 2024-09-16T05:39:23.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0b7000f9e0 con 0x7f0b8006dfa0 2024-09-16T05:39:23.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(171..171 src has 1..171) ==== 7663+0+0 (unknown 3748817033 0 0) 0x7f0b70045290 con 0x7f0b8006dfa0 2024-09-16T05:39:23.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.382+0000 7f0b7effd640 10 monclient: get_auth_request con 0x7f0b6c036a90 auth_method 0 2024-09-16T05:39:23.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.394+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0b70015520 con 0x7f0b8006dfa0 2024-09-16T05:39:23.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.394+0000 7f0b67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:23.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.394+0000 7f0b67fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.558+0000 7f0b85d8e640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.558+0000 7f0b85d8e640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.558+0000 7f0b85d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:23.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:23.558+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0b60005380 con 0x7f0b8006dfa0 2024-09-16T05:39:24.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.150+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' removed v172) ==== 148+0+0 (unknown 1371674694 0 0) 0x7f0b70014e20 con 0x7f0b8006dfa0 2024-09-16T05:39:24.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.150+0000 7f0b67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:24.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.150+0000 7f0b67fff640 10 monclient: _finish_command 2 = system:0 pool 'cold' removed 2024-09-16T05:39:24.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.250+0000 7f0b85d8e640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:24.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.250+0000 7f0b85d8e640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:24.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.250+0000 7f0b85d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:24.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.250+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0b60001150 con 0x7f0b8006dfa0 2024-09-16T05:39:24.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.254+0000 7f0b67fff640 1 -- 172.21.15.63:0/2727049068 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' does not exist v172) ==== 155+0+0 (unknown 3369014765 0 0) 0x7f0b70019170 con 0x7f0b8006dfa0 2024-09-16T05:39:24.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.254+0000 7f0b67fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T05:39:24.258 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.254+0000 7f0b67fff640 10 monclient: _finish_command 3 = system:0 pool 'cold' does not exist 2024-09-16T05:39:24.258 INFO:tasks.workunit.client.0.smithi063.stderr:pool 'cold' does not exist 2024-09-16T05:39:24.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.254+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0b6c036a90 legacy=0x7f0b6c038f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:24.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.254+0000 7f0b85d8e640 10 monclient: shutdown 2024-09-16T05:39:24.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.254+0000 7f0b85d8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:24.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.254+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 >> v1:172.21.15.6:6790/0 conn(0x7f0b8006dfa0 legacy=0x7f0b80130a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:24.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.258+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 shutdown_connections 2024-09-16T05:39:24.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.258+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 >> 172.21.15.63:0/2727049068 conn(0x7f0b800a1e90 msgr2=0x7f0b800a2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:24.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.258+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 shutdown_connections 2024-09-16T05:39:24.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.258+0000 7f0b85d8e640 1 -- 172.21.15.63:0/2727049068 wait complete. 2024-09-16T05:39:24.276 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T05:39:24.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient: get_monmap_and_config 2024-09-16T05:39:24.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient: build_initial_monmap 2024-09-16T05:39:24.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient: monmap: 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:24.398589+0000 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:24.398589+0000 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:24.399 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 1 Processor -- start 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 1 -- start start 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient: init 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient(hunting): picked mon.noname-a con 0x7fc04c0a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient(hunting): picked mon.noname-c con 0x7fc04c0a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient(hunting): picked mon.noname-b con 0x7fc04c0a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:24.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c0a4fd0 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c0a5b40 con 0x7fc04c0a08a0 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c0a1a70 con 0x7fc04c0a4790 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc053d48640 10 monclient(hunting): authenticate will time out at 1221.175573s 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.394+0000 7fc0512e3640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc04c0a08a0 legacy=0x7fc04c13beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc051ae4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc04c0a3ca0 0x7fc04c0a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57538/0 (socket says 172.21.15.63:57538) 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc051ae4640 1 -- 172.21.15.63:0/3279338328 learned_addr learned my addr 172.21.15.63:0/3279338328 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc0522e5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc04c0a4790 0x7fc04c13f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60596/0 (socket says 172.21.15.63:60596) 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815354228 0 0) 0x7fc04c0a4fd0 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient(hunting): _negotiate 2024-09-16T05:39:24.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient(hunting): my global_id is 5907 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc034003530 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1559383183 0 0) 0x7fc04c0a1a70 con 0x7fc04c0a4790 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient(hunting): _negotiate 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient(hunting): my global_id is 5885 2024-09-16T05:39:24.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc04c0a4fd0 con 0x7fc04c0a4790 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 101831473 0 0) 0x7fc034003530 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 >> v1:172.21.15.6:6790/0 conn(0x7fc04c0a4790 legacy=0x7fc04c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 >> v1:172.21.15.63:6789/0 conn(0x7fc04c0a08a0 legacy=0x7fc04c13beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: _finish_hunting 0 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 monclient: found mon.noname-a 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04c0a53e0 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: _finish_auth 0 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: _check_auth_tickets 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc0440019a0 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: dump: 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:24.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 5 monclient: authenticate success, global_id 5907 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc044001c00 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 1 -- 172.21.15.63:0/3279338328 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc044001f50 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 4 set_mon_vals no callback set 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc050ae2640 10 monclient: dump: 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.408 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient: shutdown 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- 172.21.15.63:0/3279338328 >> v1:172.21.15.6:6789/0 conn(0x7fc04c0a3ca0 legacy=0x7fc04c0a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- 172.21.15.63:0/3279338328 shutdown_connections 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- 172.21.15.63:0/3279338328 >> 172.21.15.63:0/3279338328 conn(0x7fc04c0a67c0 msgr2=0x7fc04c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:24.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- 172.21.15.63:0/3279338328 shutdown_connections 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- 172.21.15.63:0/3279338328 wait complete. 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient: build_initial_monmap 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient: monmap: 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:24.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 Processor -- start 2024-09-16T05:39:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- start start 2024-09-16T05:39:24.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient: init 2024-09-16T05:39:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient(hunting): picked mon.noname-a con 0x7fc04c0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient(hunting): picked mon.noname-b con 0x7fc04c0a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient(hunting): picked mon.noname-c con 0x7fc04c0a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c0a0500 con 0x7fc04c0a08a0 2024-09-16T05:39:24.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c0a84b0 con 0x7fc04c0a3ca0 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc0512e3640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc04c0a3ca0 legacy=0x7fc04c1d0470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c121450 con 0x7fc04c0a4790 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc053d48640 10 monclient(hunting): authenticate will time out at 1221.179568s 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc051ae4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc04c0a08a0 0x7fc04c135d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57542/0 (socket says 172.21.15.63:57542) 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc051ae4640 1 -- 172.21.15.63:0/1772267571 learned_addr learned my addr 172.21.15.63:0/1772267571 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.398+0000 7fc0522e5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc04c0a4790 0x7fc04c1d3ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60598/0 (socket says 172.21.15.63:60598) 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242287718 0 0) 0x7fc04c121450 con 0x7fc04c0a4790 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): my global_id is 5888 2024-09-16T05:39:24.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc038003530 con 0x7fc04c0a4790 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452005909 0 0) 0x7fc04c0a0500 con 0x7fc04c0a08a0 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): my global_id is 5910 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc04c121450 con 0x7fc04c0a08a0 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1620938258 0 0) 0x7fc038003530 con 0x7fc04c0a4790 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc04c0a0500 con 0x7fc04c0a4790 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3768694191 0 0) 0x7fc04c121450 con 0x7fc04c0a08a0 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc038003530 con 0x7fc04c0a08a0 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc03c0018c0 con 0x7fc04c0a4790 2024-09-16T05:39:24.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): dump: 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:24.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc0440018a0 con 0x7fc04c0a08a0 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient(hunting): dump: 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:24.418 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 914329454 0 0) 0x7fc04c0a0500 con 0x7fc04c0a4790 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 >> v1:172.21.15.63:6789/0 conn(0x7fc04c0a3ca0 legacy=0x7fc04c1d0470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 >> v1:172.21.15.6:6789/0 conn(0x7fc04c0a08a0 legacy=0x7fc04c135d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 monclient: found mon.c 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04c123740 con 0x7fc04c0a4790 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:24.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 5 monclient: authenticate success, global_id 5888 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 10 monclient: _renew_subs 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc04c121d30 con 0x7fc04c0a4790 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 10 monclient: _renew_subs 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc04c136490 con 0x7fc04c0a4790 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc03c0047f0 con 0x7fc04c0a4790 2024-09-16T05:39:24.420 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc03c004b40 con 0x7fc04c0a4790 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 10 monclient: dump: 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:24.421 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04bfff640 4 set_mon_vals no callback set 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc02c005090 con 0x7fc04c0a4790 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc03c00f8e0 con 0x7fc04c0a4790 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(172..172 src has 1..172) ==== 6856+0+0 (unknown 3435643238 0 0) 0x7fc03c044de0 con 0x7fc04c0a4790 2024-09-16T05:39:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.402+0000 7fc051ae4640 10 monclient: get_auth_request con 0x7fc038036970 auth_method 0 2024-09-16T05:39:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.414+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc03c0153b0 con 0x7fc04c0a4790 2024-09-16T05:39:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.414+0000 7fc04a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:24.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.414+0000 7fc04a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:24.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.586+0000 7fc053d48640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:24.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.586+0000 7fc053d48640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:24.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.586+0000 7fc053d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:24.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:24.586+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7fc02c005380 con 0x7fc04c0a4790 2024-09-16T05:39:25.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:25.154+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v173) ==== 78+0+0 (unknown 2639836633 0 0) 0x7fc03c019060 con 0x7fc04c0a4790 2024-09-16T05:39:25.159 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:25.154+0000 7fc04a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:25.160 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:25.154+0000 7fc04a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:25.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:25.278+0000 7fc053d48640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:25.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:25.278+0000 7fc053d48640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:25.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:25.278+0000 7fc053d48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:25.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:25.278+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7fc02c002220 con 0x7fc04c0a4790 2024-09-16T05:39:26.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.038+0000 7fc04a7fc640 1 -- 172.21.15.63:0/1772267571 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v174) ==== 78+0+0 (unknown 1595632566 0 0) 0x7fc03c01d920 con 0x7fc04c0a4790 2024-09-16T05:39:26.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.038+0000 7fc04a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:26.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.038+0000 7fc04a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:26.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.046+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc038036970 legacy=0x7fc038038e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:26.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.046+0000 7fc053d48640 10 monclient: shutdown 2024-09-16T05:39:26.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.046+0000 7fc053d48640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:26.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.046+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 >> v1:172.21.15.6:6790/0 conn(0x7fc04c0a4790 legacy=0x7fc04c1d3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:26.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.046+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 shutdown_connections 2024-09-16T05:39:26.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.046+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 >> 172.21.15.63:0/1772267571 conn(0x7fc04c0a67c0 msgr2=0x7fc04c136e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:26.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.046+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 shutdown_connections 2024-09-16T05:39:26.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.046+0000 7fc053d48640 1 -- 172.21.15.63:0/1772267571 wait complete. 2024-09-16T05:39:26.075 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T05:39:26.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient: get_monmap_and_config 2024-09-16T05:39:26.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient: build_initial_monmap 2024-09-16T05:39:26.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient: monmap: 2024-09-16T05:39:26.198 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:26.198 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:26.198 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:26.197997+0000 2024-09-16T05:39:26.198 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:26.197997+0000 2024-09-16T05:39:26.199 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:26.199 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:26.199 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:26.199 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:26.199 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:26.199 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:26.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 1 Processor -- start 2024-09-16T05:39:26.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 1 -- start start 2024-09-16T05:39:26.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient: init 2024-09-16T05:39:26.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:26.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:26.202 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient(hunting): picked mon.noname-a con 0x7fc84c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient(hunting): picked mon.noname-b con 0x7fc84c06dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient(hunting): picked mon.noname-c con 0x7fc84c0a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84c09f800 con 0x7fc84c09ac90 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84c0a0ab0 con 0x7fc84c0a0210 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84c0a0d90 con 0x7fc84c06dd30 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc8519d2640 10 monclient(hunting): authenticate will time out at 1222.974898s 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc84b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc84c0a0210 legacy=0x7fc84c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc84a7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc84c06dd30 0x7fc84c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60632/0 (socket says 172.21.15.63:60632) 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc84affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc84c09ac90 0x7fc84c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57568/0 (socket says 172.21.15.63:57568) 2024-09-16T05:39:26.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc84a7fc640 1 -- 172.21.15.63:0/3158990614 learned_addr learned my addr 172.21.15.63:0/3158990614 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2131395217 0 0) 0x7fc84c09f800 con 0x7fc84c09ac90 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 10 monclient(hunting): my global_id is 5913 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc834003530 con 0x7fc84c09ac90 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3904763108 0 0) 0x7fc84c0a0d90 con 0x7fc84c06dd30 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 10 monclient(hunting): my global_id is 5891 2024-09-16T05:39:26.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.194+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc84c09f800 con 0x7fc84c06dd30 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3221201240 0 0) 0x7fc834003530 con 0x7fc84c09ac90 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 >> v1:172.21.15.6:6790/0 conn(0x7fc84c06dd30 legacy=0x7fc84c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 >> v1:172.21.15.63:6789/0 conn(0x7fc84c0a0210 legacy=0x7fc84c13d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 1 monclient: found mon.noname-a 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c09e770 con 0x7fc84c09ac90 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc83c0018d0 con 0x7fc84c09ac90 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:26.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: dump: 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 5 monclient: authenticate success, global_id 5913 2024-09-16T05:39:26.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc83c001bf0 con 0x7fc84c09ac90 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 1 -- 172.21.15.63:0/3158990614 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc83c001fc0 con 0x7fc84c09ac90 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc849ffb640 10 monclient: dump: 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:26.207 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 4 set_mon_vals no callback set 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient: shutdown 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- 172.21.15.63:0/3158990614 >> v1:172.21.15.6:6789/0 conn(0x7fc84c09ac90 legacy=0x7fc84c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- 172.21.15.63:0/3158990614 shutdown_connections 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- 172.21.15.63:0/3158990614 >> 172.21.15.63:0/3158990614 conn(0x7fc84c0a1c90 msgr2=0x7fc84c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- 172.21.15.63:0/3158990614 shutdown_connections 2024-09-16T05:39:26.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- 172.21.15.63:0/3158990614 wait complete. 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient: build_initial_monmap 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient: monmap: 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:26.209 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:26.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 Processor -- start 2024-09-16T05:39:26.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- start start 2024-09-16T05:39:26.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient: init 2024-09-16T05:39:26.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient(hunting): picked mon.noname-b con 0x7fc84c06dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient(hunting): picked mon.noname-a con 0x7fc84c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient(hunting): picked mon.noname-c con 0x7fc84c0a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84c09ca70 con 0x7fc84c09ac90 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc84affd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc84c06dd30 legacy=0x7fc84c133a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84c09c520 con 0x7fc84c06dd30 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84c120160 con 0x7fc84c0a0210 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc8519d2640 10 monclient(hunting): authenticate will time out at 1222.978802s 2024-09-16T05:39:26.214 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc84a7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc84c09ac90 0x7fc84c1cbd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57582/0 (socket says 172.21.15.63:57582) 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc84a7fc640 1 -- 172.21.15.63:0/3083227918 learned_addr learned my addr 172.21.15.63:0/3083227918 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419223537 0 0) 0x7fc84c09f800 con 0x7fc84c09ac90 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 10 monclient(hunting): my global_id is 5916 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc838003530 con 0x7fc84c09ac90 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3102151284 0 0) 0x7fc84c120160 con 0x7fc84c0a0210 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:26.215 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 10 monclient(hunting): my global_id is 5894 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.198+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc84c09f800 con 0x7fc84c0a0210 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 725522022 0 0) 0x7fc838003530 con 0x7fc84c09ac90 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc84c120160 con 0x7fc84c09ac90 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc840001780 con 0x7fc84c09ac90 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient(hunting): dump: 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.216 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1574990791 0 0) 0x7fc84c09f800 con 0x7fc84c0a0210 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc838003530 con 0x7fc84c0a0210 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc844003f40 con 0x7fc84c0a0210 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient(hunting): dump: 2024-09-16T05:39:26.217 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 583846306 0 0) 0x7fc84c120160 con 0x7fc84c09ac90 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 >> v1:172.21.15.6:6790/0 conn(0x7fc84c0a0210 legacy=0x7fc84c1cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:26.218 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 >> v1:172.21.15.63:6789/0 conn(0x7fc84c06dd30 legacy=0x7fc84c133a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 monclient: found mon.a 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c122170 con 0x7fc84c09ac90 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient: _finish_auth 0 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 5 monclient: authenticate success, global_id 5916 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 10 monclient: _renew_subs 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c122450 con 0x7fc84c09ac90 2024-09-16T05:39:26.219 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 10 monclient: _renew_subs 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc84c134190 con 0x7fc84c09ac90 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc840001cb0 con 0x7fc84c09ac90 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc840004990 con 0x7fc84c09ac90 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8497fa640 4 set_mon_vals no callback set 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 10 monclient: dump: 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:26.220 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:26.221 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82c005090 con 0x7fc84c09ac90 2024-09-16T05:39:26.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc84000f730 con 0x7fc84c09ac90 2024-09-16T05:39:26.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(174..174 src has 1..174) ==== 6433+0+0 (unknown 2902098349 0 0) 0x7fc840044c30 con 0x7fc84c09ac90 2024-09-16T05:39:26.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.202+0000 7fc84affd640 10 monclient: get_auth_request con 0x7fc8380369d0 auth_method 0 2024-09-16T05:39:26.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.214+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc84c09ca70 con 0x7fc84c09ac90 2024-09-16T05:39:26.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.214+0000 7fc833fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:26.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.214+0000 7fc833fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:26.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.386+0000 7fc8519d2640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:26.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.386+0000 7fc8519d2640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:26.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.386+0000 7fc8519d2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:26.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:26.386+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7fc82c005380 con 0x7fc84c09ac90 2024-09-16T05:39:27.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.034+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v175) ==== 82+0+0 (unknown 414686427 0 0) 0x7fc840015320 con 0x7fc84c09ac90 2024-09-16T05:39:27.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.034+0000 7fc833fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:27.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.034+0000 7fc833fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:27.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.146+0000 7fc8519d2640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.146+0000 7fc8519d2640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.146+0000 7fc8519d2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.146+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7fc82c002220 con 0x7fc84c09ac90 2024-09-16T05:39:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc833fff640 1 -- 172.21.15.63:0/3083227918 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v175) ==== 115+0+0 (unknown 3086697838 0 0) 0x7fc840018fd0 con 0x7fc84c09ac90 2024-09-16T05:39:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc833fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:27.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc833fff640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T05:39:27.153 INFO:tasks.workunit.client.0.smithi063.stderr:compat weight-set already created 2024-09-16T05:39:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc8380369d0 legacy=0x7fc838038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc8519d2640 10 monclient: shutdown 2024-09-16T05:39:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc8519d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 >> v1:172.21.15.6:6789/0 conn(0x7fc84c09ac90 legacy=0x7fc84c1cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:27.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 shutdown_connections 2024-09-16T05:39:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 >> 172.21.15.63:0/3083227918 conn(0x7fc84c0a1c90 msgr2=0x7fc84c134870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 shutdown_connections 2024-09-16T05:39:27.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.150+0000 7fc8519d2640 1 -- 172.21.15.63:0/3083227918 wait complete. 2024-09-16T05:39:27.170 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket fooo host 2024-09-16T05:39:27.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.290+0000 7f64a3dd8640 10 monclient: get_monmap_and_config 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.290+0000 7f64a3dd8640 10 monclient: build_initial_monmap 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.290+0000 7f64a3dd8640 10 monclient: monmap: 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:27.295634+0000 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:27.295634+0000 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:27.296 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:27.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 Processor -- start 2024-09-16T05:39:27.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- start start 2024-09-16T05:39:27.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient: init 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient(hunting): picked mon.noname-c con 0x7f649c136960 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient(hunting): picked mon.noname-b con 0x7f649c0a1c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient(hunting): picked mon.noname-a con 0x7f649c137450 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f649c0a09c0 con 0x7f649c137450 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f649c0a0f80 con 0x7f649c136960 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f649c0a4bb0 con 0x7f649c0a1c40 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient(hunting): authenticate will time out at 1224.072937s 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a1b74640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f649c136960 legacy=0x7f649c136d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:27.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a1373640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f649c0a1c40 0x7f649c13bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60654/0 (socket says 172.21.15.63:60654) 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a2375640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f649c137450 0x7f649c13f4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57592/0 (socket says 172.21.15.63:57592) 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a1373640 1 -- 172.21.15.63:0/4072904190 learned_addr learned my addr 172.21.15.63:0/4072904190 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821093798 0 0) 0x7f649c0a09c0 con 0x7f649c137450 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient(hunting): _negotiate 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient(hunting): my global_id is 5919 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6488003530 con 0x7f649c137450 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 15295790 0 0) 0x7f649c0a4bb0 con 0x7f649c0a1c40 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient(hunting): _negotiate 2024-09-16T05:39:27.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient(hunting): my global_id is 5897 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f649c0a09c0 con 0x7f649c0a1c40 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1651507283 0 0) 0x7f6488003530 con 0x7f649c137450 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 >> v1:172.21.15.6:6790/0 conn(0x7f649c0a1c40 legacy=0x7f649c13bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 >> v1:172.21.15.63:6789/0 conn(0x7f649c136960 legacy=0x7f649c136d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: _finish_hunting 0 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 monclient: found mon.noname-a 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649c0a4e60 con 0x7f649c137450 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: _finish_auth 0 2024-09-16T05:39:27.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: _check_auth_tickets 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f64940019a0 con 0x7f649c137450 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: dump: 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:27.303 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 5 monclient: authenticate success, global_id 5919 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6494001bd0 con 0x7f649c137450 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 4 set_mon_vals no callback set 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 1 -- 172.21.15.63:0/4072904190 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6494001fe0 con 0x7f649c137450 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a0b72640 10 monclient: dump: 2024-09-16T05:39:27.304 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 10 monclient: shutdown 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- 172.21.15.63:0/4072904190 >> v1:172.21.15.6:6789/0 conn(0x7f649c137450 legacy=0x7f649c13f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- 172.21.15.63:0/4072904190 shutdown_connections 2024-09-16T05:39:27.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- 172.21.15.63:0/4072904190 >> 172.21.15.63:0/4072904190 conn(0x7f649c0a60a0 msgr2=0x7f649c0a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:27.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- 172.21.15.63:0/4072904190 shutdown_connections 2024-09-16T05:39:27.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.294+0000 7f64a3dd8640 1 -- 172.21.15.63:0/4072904190 wait complete. 2024-09-16T05:39:27.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: build_initial_monmap 2024-09-16T05:39:27.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: monmap: 2024-09-16T05:39:27.309 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 1 Processor -- start 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 1 -- start start 2024-09-16T05:39:27.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: init 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient(hunting): picked mon.noname-c con 0x7f649c0a1c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient(hunting): picked mon.noname-a con 0x7f649c136960 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient(hunting): picked mon.noname-b con 0x7f649c137450 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f649c0a0500 con 0x7f649c136960 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f649c124210 con 0x7f649c137450 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f649c1244f0 con 0x7f649c0a1c40 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a2375640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f649c137450 legacy=0x7f649c1d37d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient(hunting): authenticate will time out at 1224.077329s 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a1373640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f649c136960 0x7f649c1d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57608/0 (socket says 172.21.15.63:57608) 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a1373640 1 -- 172.21.15.63:0/2045364494 learned_addr learned my addr 172.21.15.63:0/2045364494 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:27.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 638696192 0 0) 0x7f6490001100 con 0x7f649c136960 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): my global_id is 5922 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6480003530 con 0x7f649c136960 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380334292 0 0) 0x7f649c1244f0 con 0x7f649c0a1c40 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): my global_id is 5900 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6490001100 con 0x7f649c0a1c40 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2301702073 0 0) 0x7f6480003530 con 0x7f649c136960 2024-09-16T05:39:27.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f649c1244f0 con 0x7f649c136960 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6490001950 con 0x7f649c136960 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): dump: 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:27.313 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 54514759 0 0) 0x7f6490001100 con 0x7f649c0a1c40 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6480003530 con 0x7f649c0a1c40 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6498004090 con 0x7f649c0a1c40 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient(hunting): dump: 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.314 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 879948746 0 0) 0x7f649c1244f0 con 0x7f649c136960 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 >> v1:172.21.15.6:6790/0 conn(0x7f649c0a1c40 legacy=0x7f649c135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 >> v1:172.21.15.63:6789/0 conn(0x7f649c137450 legacy=0x7f649c1d37d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:27.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 monclient: found mon.a 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649c0a73a0 con 0x7f649c136960 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 5 monclient: authenticate success, global_id 5922 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: _renew_subs 2024-09-16T05:39:27.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f649c122dc0 con 0x7f649c136960 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: _renew_subs 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f649c1366e0 con 0x7f649c136960 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6490001c50 con 0x7f649c136960 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648ffff640 4 set_mon_vals no callback set 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6490004c40 con 0x7f649c136960 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.298+0000 7f648e7fc640 10 monclient: dump: 2024-09-16T05:39:27.317 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.302+0000 7f64a3dd8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.302+0000 7f64a3dd8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.302+0000 7f64a3dd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:27.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.302+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6484005090 con 0x7f649c136960 2024-09-16T05:39:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.302+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f649000f9e0 con 0x7f649c136960 2024-09-16T05:39:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.302+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(175..175 src has 1..175) ==== 6761+0+0 (unknown 3421497477 0 0) 0x7f6490044e10 con 0x7f649c136960 2024-09-16T05:39:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.302+0000 7f64a1b74640 10 monclient: get_auth_request con 0x7f6480036ac0 auth_method 0 2024-09-16T05:39:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.314+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f649c0a0500 con 0x7f649c136960 2024-09-16T05:39:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.314+0000 7f648e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:27.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.314+0000 7f648e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:27.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.482+0000 7f64a3dd8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T05:39:27.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.482+0000 7f64a3dd8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T05:39:27.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.482+0000 7f64a3dd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:27.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:27.482+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) -- 0x7f6484005b90 con 0x7f649c136960 2024-09-16T05:39:28.200 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.198+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 added bucket fooo type host to crush map v176) ==== 140+0+0 (unknown 2311839872 0 0) 0x7f6490015420 con 0x7f649c136960 2024-09-16T05:39:28.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.198+0000 7f648e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T05:39:28.201 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.198+0000 7f648e7fc640 10 monclient: _finish_command 2 = system:0 added bucket fooo type host to crush map 2024-09-16T05:39:28.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.298+0000 7f64a3dd8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T05:39:28.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.298+0000 7f64a3dd8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T05:39:28.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.298+0000 7f64a3dd8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:28.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.298+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) -- 0x7f64840010a0 con 0x7f649c136960 2024-09-16T05:39:28.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f648e7fc640 1 -- 172.21.15.63:0/2045364494 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 bucket 'fooo' already exists v176) ==== 128+0+0 (unknown 421667752 0 0) 0x7f64900146e0 con 0x7f649c136960 2024-09-16T05:39:28.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f648e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T05:39:28.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f648e7fc640 10 monclient: _finish_command 3 = system:0 bucket 'fooo' already exists 2024-09-16T05:39:28.305 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'fooo' already exists 2024-09-16T05:39:28.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6480036ac0 legacy=0x7f6480038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:28.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f64a3dd8640 10 monclient: shutdown 2024-09-16T05:39:28.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f64a3dd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:28.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 >> v1:172.21.15.6:6789/0 conn(0x7f649c136960 legacy=0x7f649c1d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:28.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 shutdown_connections 2024-09-16T05:39:28.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 >> 172.21.15.63:0/2045364494 conn(0x7f649c0a60a0 msgr2=0x7f649c0a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:28.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 shutdown_connections 2024-09-16T05:39:28.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.302+0000 7f64a3dd8640 1 -- 172.21.15.63:0/2045364494 wait complete. 2024-09-16T05:39:28.321 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush move fooo root=default 2024-09-16T05:39:28.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.438+0000 7f98ca97b640 10 monclient: get_monmap_and_config 2024-09-16T05:39:28.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.438+0000 7f98ca97b640 10 monclient: build_initial_monmap 2024-09-16T05:39:28.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.438+0000 7f98ca97b640 10 monclient: monmap: 2024-09-16T05:39:28.443 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:28.443 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:28.443 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:28.443212+0000 2024-09-16T05:39:28.444 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:28.443212+0000 2024-09-16T05:39:28.444 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:28.444 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:28.444 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:28.444 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:28.444 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:28.444 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:28.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.438+0000 7f98ca97b640 1 Processor -- start 2024-09-16T05:39:28.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.438+0000 7f98ca97b640 1 -- start start 2024-09-16T05:39:28.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.438+0000 7f98ca97b640 10 monclient: init 2024-09-16T05:39:28.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:28.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:28.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient(hunting): picked mon.noname-b con 0x7f98c4136960 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient(hunting): picked mon.noname-a con 0x7f98c40a1c40 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient(hunting): picked mon.noname-c con 0x7f98c4137450 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c40a09c0 con 0x7f98c40a1c40 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c40a0f80 con 0x7f98c4137450 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c40a4bb0 con 0x7f98c4136960 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient(hunting): authenticate will time out at 1225.220220s 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c8f18640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f98c4137450 legacy=0x7f98c413f4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f98c40a1c40 0x7f98c413bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57618/0 (socket says 172.21.15.63:57618) 2024-09-16T05:39:28.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c37fe640 1 -- 172.21.15.63:0/2942149141 learned_addr learned my addr 172.21.15.63:0/2942149141 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 756712090 0 0) 0x7f98c40a09c0 con 0x7f98c40a1c40 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient(hunting): my global_id is 5925 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98ac003530 con 0x7f98c40a1c40 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565753310 0 0) 0x7f98c40a4bb0 con 0x7f98c4136960 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:28.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient(hunting): my global_id is 5903 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98c40a09c0 con 0x7f98c4136960 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 551752862 0 0) 0x7f98ac003530 con 0x7f98c40a1c40 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 >> v1:172.21.15.6:6790/0 conn(0x7f98c4136960 legacy=0x7f98c4136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 >> v1:172.21.15.63:6789/0 conn(0x7f98c4137450 legacy=0x7f98c413f4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 monclient: found mon.noname-a 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c40a4e60 con 0x7f98c40a1c40 2024-09-16T05:39:28.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98b8001550 con 0x7f98c40a1c40 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: dump: 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:28.451 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 5 monclient: authenticate success, global_id 5925 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98b8001800 con 0x7f98c40a1c40 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:28.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 1 -- 172.21.15.63:0/2942149141 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98b8001c10 con 0x7f98c40a1c40 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 4 set_mon_vals no callback set 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98c2ffd640 10 monclient: dump: 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:28.453 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient: shutdown 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- 172.21.15.63:0/2942149141 >> v1:172.21.15.6:6789/0 conn(0x7f98c40a1c40 legacy=0x7f98c413bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- 172.21.15.63:0/2942149141 shutdown_connections 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- 172.21.15.63:0/2942149141 >> 172.21.15.63:0/2942149141 conn(0x7f98c40a60a0 msgr2=0x7f98c40a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- 172.21.15.63:0/2942149141 shutdown_connections 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- 172.21.15.63:0/2942149141 wait complete. 2024-09-16T05:39:28.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient: build_initial_monmap 2024-09-16T05:39:28.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient: monmap: 2024-09-16T05:39:28.459 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:28.459 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:28.459 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:28.459 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:28.459 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:28.459 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 Processor -- start 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 1 -- start start 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.442+0000 7f98ca97b640 10 monclient: init 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient(hunting): picked mon.noname-b con 0x7f98c40a1c40 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:28.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient(hunting): picked mon.noname-a con 0x7f98c4136960 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient(hunting): picked mon.noname-c con 0x7f98c4137450 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c4122470 con 0x7f98c4136960 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c40a6fa0 con 0x7f98c40a1c40 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c3fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f98c40a1c40 legacy=0x7f98c4135d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c40a0500 con 0x7f98c4137450 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient(hunting): authenticate will time out at 1225.224486s 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f98c4136960 0x7f98c41d01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57622/0 (socket says 172.21.15.63:57622) 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c37fe640 1 -- 172.21.15.63:0/2405041248 learned_addr learned my addr 172.21.15.63:0/2405041248 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c8f18640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f98c4137450 0x7f98c41d3920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60700/0 (socket says 172.21.15.63:60700) 2024-09-16T05:39:28.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 143819849 0 0) 0x7f98c4122470 con 0x7f98c4136960 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): my global_id is 5928 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98b0003530 con 0x7f98c4136960 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2238621982 0 0) 0x7f98c40a0500 con 0x7f98c4137450 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): my global_id is 5906 2024-09-16T05:39:28.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98c4122470 con 0x7f98c4137450 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2578245597 0 0) 0x7f98b0003530 con 0x7f98c4136960 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98c40a0500 con 0x7f98c4136960 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98b8001c10 con 0x7f98c4136960 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.463 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1266791432 0 0) 0x7f98c4122470 con 0x7f98c4137450 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98b0003530 con 0x7f98c4137450 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98bc0042c0 con 0x7f98c4137450 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:28.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:28.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 836586577 0 0) 0x7f98c40a0500 con 0x7f98c4136960 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 >> v1:172.21.15.6:6790/0 conn(0x7f98c4137450 legacy=0x7f98c41d3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 >> v1:172.21.15.63:6789/0 conn(0x7f98c40a1c40 legacy=0x7f98c4135d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 monclient: found mon.a 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c4124760 con 0x7f98c4136960 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:39:28.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 5 monclient: authenticate success, global_id 5928 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient: _renew_subs 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f98c4123ec0 con 0x7f98c4136960 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient: _renew_subs 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f98c41364a0 con 0x7f98c4136960 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98b8001800 con 0x7f98c4136960 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:28.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c27fc640 4 set_mon_vals no callback set 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98b8004cd0 con 0x7f98c4136960 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 10 monclient: dump: 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:28.468 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98a4005090 con 0x7f98c4136960 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.446+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f98b8004ef0 con 0x7f98c4136960 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.450+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(176..176 src has 1..176) ==== 6789+0+0 (unknown 3694055913 0 0) 0x7f98b8044f60 con 0x7f98c4136960 2024-09-16T05:39:28.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.450+0000 7f98c3fff640 10 monclient: get_auth_request con 0x7f98b0036940 auth_method 0 2024-09-16T05:39:28.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.462+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f98b8015550 con 0x7f98c4136960 2024-09-16T05:39:28.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.462+0000 7f98c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:28.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.462+0000 7f98c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:28.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.630+0000 7f98ca97b640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T05:39:28.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.630+0000 7f98ca97b640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T05:39:28.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.630+0000 7f98ca97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:28.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:28.630+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) -- 0x7f98a4005b90 con 0x7f98c4136960 2024-09-16T05:39:29.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.202+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 moved item id -16 name 'fooo' to location {root=default} in crush map v177) ==== 173+0+0 (unknown 2678420201 0 0) 0x7f98b80191a0 con 0x7f98c4136960 2024-09-16T05:39:29.204 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.202+0000 7f98c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T05:39:29.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.202+0000 7f98c0ff9640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T05:39:29.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.302+0000 7f98ca97b640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T05:39:29.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.302+0000 7f98ca97b640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T05:39:29.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.302+0000 7f98ca97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:29.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.302+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) -- 0x7f98a4005610 con 0x7f98c4136960 2024-09-16T05:39:29.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98c0ff9640 1 -- 172.21.15.63:0/2405041248 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 no need to move item id -16 name 'fooo' to location {root=default} in crush map v177) ==== 183+0+0 (unknown 1397628914 0 0) 0x7f98b801da60 con 0x7f98c4136960 2024-09-16T05:39:29.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98c0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T05:39:29.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98c0ff9640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T05:39:29.309 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T05:39:29.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 >> v1:172.21.15.63:6812/3643207211 conn(0x7f98b0036940 legacy=0x7f98b0038e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:29.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98ca97b640 10 monclient: shutdown 2024-09-16T05:39:29.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98ca97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:29.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 >> v1:172.21.15.6:6789/0 conn(0x7f98c4136960 legacy=0x7f98c41d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:29.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 shutdown_connections 2024-09-16T05:39:29.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 >> 172.21.15.63:0/2405041248 conn(0x7f98c40a60a0 msgr2=0x7f98c40a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:29.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 shutdown_connections 2024-09-16T05:39:29.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.306+0000 7f98ca97b640 1 -- 172.21.15.63:0/2405041248 wait complete. 2024-09-16T05:39:29.326 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket barr rack 2024-09-16T05:39:29.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient: get_monmap_and_config 2024-09-16T05:39:29.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient: build_initial_monmap 2024-09-16T05:39:29.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient: monmap: 2024-09-16T05:39:29.449 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:29.449 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:29.449 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:29.449136+0000 2024-09-16T05:39:29.449 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:29.449136+0000 2024-09-16T05:39:29.450 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:29.450 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:29.450 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:29.450 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:29.450 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:29.450 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 1 Processor -- start 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 1 -- start start 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient: init 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient(hunting): picked mon.noname-b con 0x7f27ac09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient(hunting): picked mon.noname-c con 0x7f27ac0a0280 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient(hunting): picked mon.noname-a con 0x7f27ac09b780 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27ac074200 con 0x7f27ac09b780 2024-09-16T05:39:29.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27ac09f900 con 0x7f27ac0a0280 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27ac09e870 con 0x7f27ac09ac90 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 10 monclient(hunting): authenticate will time out at 1226.226401s 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1b35640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f27ac0a0280 legacy=0x7f27ac1398b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b2336640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f27ac09ac90 0x7f27ac09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60708/0 (socket says 172.21.15.63:60708) 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b2b37640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f27ac09b780 0x7f27ac13cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57636/0 (socket says 172.21.15.63:57636) 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b2336640 1 -- 172.21.15.63:0/665678405 learned_addr learned my addr 172.21.15.63:0/665678405 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3389894459 0 0) 0x7f27ac09e870 con 0x7f27ac09ac90 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient(hunting): _negotiate 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient(hunting): my global_id is 5909 2024-09-16T05:39:29.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2794003530 con 0x7f27ac09ac90 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2826983003 0 0) 0x7f27ac074200 con 0x7f27ac09b780 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient(hunting): _negotiate 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient(hunting): my global_id is 5931 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27ac09e870 con 0x7f27ac09b780 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2097486437 0 0) 0x7f2794003530 con 0x7f27ac09ac90 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 >> v1:172.21.15.63:6789/0 conn(0x7f27ac0a0280 legacy=0x7f27ac1398b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 >> v1:172.21.15.6:6789/0 conn(0x7f27ac09b780 legacy=0x7f27ac13cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient: _finish_hunting 0 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 monclient: found mon.noname-b 2024-09-16T05:39:29.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27ac141730 con 0x7f27ac09ac90 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient: _finish_auth 0 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient: _check_auth_tickets 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f27a4001620 con 0x7f27ac09ac90 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b1334640 10 monclient: dump: 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:29.456 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 5 monclient: authenticate success, global_id 5909 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.446+0000 7f27b459a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27a4001880 con 0x7f27ac09ac90 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b1334640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b1334640 1 -- 172.21.15.63:0/665678405 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f27a4001bd0 con 0x7f27ac09ac90 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b1334640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:29.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b1334640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b1334640 10 monclient: dump: 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:29.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 4 set_mon_vals no callback set 2024-09-16T05:39:29.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: shutdown 2024-09-16T05:39:29.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:29.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- 172.21.15.63:0/665678405 >> v1:172.21.15.6:6790/0 conn(0x7f27ac09ac90 legacy=0x7f27ac09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:29.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- 172.21.15.63:0/665678405 shutdown_connections 2024-09-16T05:39:29.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- 172.21.15.63:0/665678405 >> 172.21.15.63:0/665678405 conn(0x7f27ac0a5290 msgr2=0x7f27ac0a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:29.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- 172.21.15.63:0/665678405 shutdown_connections 2024-09-16T05:39:29.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- 172.21.15.63:0/665678405 wait complete. 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: build_initial_monmap 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: monmap: 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:29.463 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 Processor -- start 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- start start 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: init 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient(hunting): picked mon.noname-c con 0x7f27ac09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient(hunting): picked mon.noname-b con 0x7f27ac09b780 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient(hunting): picked mon.noname-a con 0x7f27ac0a0280 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27ac09d250 con 0x7f27ac0a0280 2024-09-16T05:39:29.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27ac09e0d0 con 0x7f27ac09b780 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27ac140c70 con 0x7f27ac09ac90 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient(hunting): authenticate will time out at 1226.230081s 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b1b35640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f27ac09b780 legacy=0x7f27ac09cb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b2336640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f27ac09ac90 0x7f27ac09c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60710/0 (socket says 172.21.15.63:60710) 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b2336640 1 -- 172.21.15.63:0/1093319927 learned_addr learned my addr 172.21.15.63:0/1093319927 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b2336640 1 -- 172.21.15.63:0/1093319927 >> v1:172.21.15.6:6790/0 conn(0x7f27ac09ac90 legacy=0x7f27ac09c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b2336640 1 -- 172.21.15.63:0/1093319927 >> v1:172.21.15.6:6790/0 conn(0x7f27ac09ac90 legacy=0x7f27ac09c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b2336640 1 --1- 172.21.15.63:0/1093319927 >> v1:172.21.15.6:6790/0 conn(0x7f27ac09ac90 0x7f27ac09c430 :-1 s=OPENED pgs=702 cs=1 l=1).handle_message read tag failed 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b2336640 1 --1- 172.21.15.63:0/1093319927 >> v1:172.21.15.6:6790/0 conn(0x7f27ac09ac90 0x7f27ac09c430 :-1 s=OPENED pgs=702 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6790/0 2024-09-16T05:39:29.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3840157008 0 0) 0x7f27ac09d250 con 0x7f27ac0a0280 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): my global_id is 5934 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2798003530 con 0x7f27ac0a0280 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 485450130 0 0) 0x7f2798003530 con 0x7f27ac0a0280 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27ac09d250 con 0x7f27ac0a0280 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f279c0018c0 con 0x7f27ac0a0280 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): dump: 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:29.466 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 782113464 0 0) 0x7f27ac09d250 con 0x7f27ac0a0280 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 >> v1:172.21.15.6:6790/0 conn(0x7f27ac09ac90 legacy=0x7f27ac09c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 >> v1:172.21.15.63:6789/0 conn(0x7f27ac09b780 legacy=0x7f27ac09cb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:29.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 monclient: found mon.a 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27ac13f820 con 0x7f27ac0a0280 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 5 monclient: authenticate success, global_id 5934 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: _renew_subs 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:29.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27ac13f4e0 con 0x7f27ac0a0280 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: _renew_subs 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.450+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f27ac1d4dd0 con 0x7f27ac0a0280 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f279c0047f0 con 0x7f27ac0a0280 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f279c004b40 con 0x7f27ac0a0280 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27aaffd640 10 monclient: dump: 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:29.469 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27b0b33640 4 set_mon_vals no callback set 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27b459a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27b459a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27b459a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278c005090 con 0x7f27ac0a0280 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f279c00f8e0 con 0x7f27ac0a0280 2024-09-16T05:39:29.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(177..177 src has 1..177) ==== 6977+0+0 (unknown 3675683155 0 0) 0x7f279c044e60 con 0x7f27ac0a0280 2024-09-16T05:39:29.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.454+0000 7f27b2336640 10 monclient: get_auth_request con 0x7f2798036a00 auth_method 0 2024-09-16T05:39:29.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.466+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f279c015330 con 0x7f27ac0a0280 2024-09-16T05:39:29.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.466+0000 7f27aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:29.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.466+0000 7f27aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.630+0000 7f27b459a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T05:39:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.630+0000 7f27b459a640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T05:39:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.630+0000 7f27b459a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:29.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:29.630+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) -- 0x7f278c005b90 con 0x7f27ac0a0280 2024-09-16T05:39:30.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.206+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 added bucket barr type rack to crush map v178) ==== 140+0+0 (unknown 285250865 0 0) 0x7f279c018fe0 con 0x7f27ac0a0280 2024-09-16T05:39:30.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.210+0000 7f27aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T05:39:30.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.210+0000 7f27aaffd640 10 monclient: _finish_command 2 = system:0 added bucket barr type rack to crush map 2024-09-16T05:39:30.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.318+0000 7f27b459a640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T05:39:30.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.318+0000 7f27b459a640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T05:39:30.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.318+0000 7f27b459a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:30.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.318+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) -- 0x7f278c0010a0 con 0x7f27ac0a0280 2024-09-16T05:39:30.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.318+0000 7f27aaffd640 1 -- 172.21.15.63:0/1093319927 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 bucket 'barr' already exists v178) ==== 128+0+0 (unknown 1110192437 0 0) 0x7f279c01d8a0 con 0x7f27ac0a0280 2024-09-16T05:39:30.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.318+0000 7f27aaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T05:39:30.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.318+0000 7f27aaffd640 10 monclient: _finish_command 3 = system:0 bucket 'barr' already exists 2024-09-16T05:39:30.323 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'barr' already exists 2024-09-16T05:39:30.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.322+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2798036a00 legacy=0x7f2798038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:30.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.322+0000 7f27b459a640 10 monclient: shutdown 2024-09-16T05:39:30.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.322+0000 7f27b459a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:30.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.322+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 >> v1:172.21.15.6:6789/0 conn(0x7f27ac0a0280 legacy=0x7f27ac1d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:30.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.322+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 shutdown_connections 2024-09-16T05:39:30.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.322+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 >> 172.21.15.63:0/1093319927 conn(0x7f27ac0a5290 msgr2=0x7f27ac137180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:30.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.322+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 shutdown_connections 2024-09-16T05:39:30.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.322+0000 7f27b459a640 1 -- 172.21.15.63:0/1093319927 wait complete. 2024-09-16T05:39:30.339 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush move barr root=default 2024-09-16T05:39:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 10 monclient: get_monmap_and_config 2024-09-16T05:39:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 10 monclient: build_initial_monmap 2024-09-16T05:39:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 10 monclient: monmap: 2024-09-16T05:39:30.463 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:30.462964+0000 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:30.462964+0000 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:30.464 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:30.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 1 Processor -- start 2024-09-16T05:39:30.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 1 -- start start 2024-09-16T05:39:30.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 10 monclient: init 2024-09-16T05:39:30.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:30.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:30.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 10 monclient(hunting): picked mon.noname-b con 0x7f018409b540 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:30.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.458+0000 7f018a5b9640 10 monclient(hunting): picked mon.noname-c con 0x7f01840a1520 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 10 monclient(hunting): picked mon.noname-a con 0x7f0184138d70 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018409f890 con 0x7f0184138d70 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0184072e80 con 0x7f01840a1520 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f01837fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f01840a1520 legacy=0x7f01840a0240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018409ce30 con 0x7f018409b540 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 10 monclient(hunting): authenticate will time out at 1227.240095s 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0188b56640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0184138d70 0x7f018413b160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57660/0 (socket says 172.21.15.63:57660) 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0183fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f018409b540 0x7f018409fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60724/0 (socket says 172.21.15.63:60724) 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0188b56640 1 -- 172.21.15.63:0/44331066 learned_addr learned my addr 172.21.15.63:0/44331066 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1873178847 0 0) 0x7f018409ce30 con 0x7f018409b540 2024-09-16T05:39:30.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient(hunting): my global_id is 5912 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f016c003450 con 0x7f018409b540 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148016857 0 0) 0x7f018409f890 con 0x7f0184138d70 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient(hunting): my global_id is 5937 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f018409ce30 con 0x7f0184138d70 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1018721006 0 0) 0x7f016c003450 con 0x7f018409b540 2024-09-16T05:39:30.469 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 >> v1:172.21.15.63:6789/0 conn(0x7f01840a1520 legacy=0x7f01840a0240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 >> v1:172.21.15.6:6789/0 conn(0x7f0184138d70 legacy=0x7f018413b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 monclient: found mon.noname-b 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018409d110 con 0x7f018409b540 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0174001550 con 0x7f018409b540 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:30.470 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: dump: 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 5 monclient: authenticate success, global_id 5912 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:30.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01740017b0 con 0x7f018409b540 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 4 set_mon_vals no callback set 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 1 -- 172.21.15.63:0/44331066 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0174001b00 con 0x7f018409b540 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f0182ffd640 10 monclient: dump: 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:30.472 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 10 monclient: shutdown 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- 172.21.15.63:0/44331066 >> v1:172.21.15.6:6790/0 conn(0x7f018409b540 legacy=0x7f018409fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- 172.21.15.63:0/44331066 shutdown_connections 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- 172.21.15.63:0/44331066 >> 172.21.15.63:0/44331066 conn(0x7f01840a5290 msgr2=0x7f01840a56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- 172.21.15.63:0/44331066 shutdown_connections 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- 172.21.15.63:0/44331066 wait complete. 2024-09-16T05:39:30.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 10 monclient: build_initial_monmap 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 10 monclient: monmap: 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:30.474 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:30.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 Processor -- start 2024-09-16T05:39:30.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 1 -- start start 2024-09-16T05:39:30.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.462+0000 7f018a5b9640 10 monclient: init 2024-09-16T05:39:30.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:30.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:30.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient(hunting): picked mon.noname-a con 0x7f018409b540 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:30.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient(hunting): picked mon.noname-c con 0x7f01840a1520 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:30.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient(hunting): picked mon.noname-b con 0x7f0184138d70 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0184124240 con 0x7f018409b540 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0184123070 con 0x7f0184138d70 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0188b56640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0184138d70 legacy=0x7f01841d38a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0184124520 con 0x7f01840a1520 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient(hunting): authenticate will time out at 1227.244215s 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0183fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f018409b540 0x7f0184134ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57668/0 (socket says 172.21.15.63:57668) 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0183fff640 1 -- 172.21.15.63:0/4025559794 learned_addr learned my addr 172.21.15.63:0/4025559794 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3238126071 0 0) 0x7f0184124520 con 0x7f01840a1520 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:30.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): my global_id is 5915 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0170003560 con 0x7f01840a1520 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659802236 0 0) 0x7f0184124240 con 0x7f018409b540 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): my global_id is 5940 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0184124520 con 0x7f018409b540 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2084869946 0 0) 0x7f0170003560 con 0x7f01840a1520 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0184124240 con 0x7f01840a1520 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01780040d0 con 0x7f01840a1520 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:30.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:30.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1549954711 0 0) 0x7f0184124520 con 0x7f018409b540 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0170003560 con 0x7f018409b540 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 59896399 0 0) 0x7f0184124240 con 0x7f01840a1520 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 >> v1:172.21.15.63:6789/0 conn(0x7f0184138d70 legacy=0x7f01841d38a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 >> v1:172.21.15.6:6789/0 conn(0x7f018409b540 legacy=0x7f0184134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 monclient: found mon.c 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01840a1b50 con 0x7f01840a1520 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient: _finish_auth 0 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:39:30.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 5 monclient: authenticate success, global_id 5915 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient: _renew_subs 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0184123f60 con 0x7f01840a1520 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient: _renew_subs 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f01841355e0 con 0x7f01840a1520 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0178004380 con 0x7f01840a1520 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f01827fc640 4 set_mon_vals no callback set 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0178004750 con 0x7f01840a1520 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:30.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 10 monclient: dump: 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:30.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0164005090 con 0x7f01840a1520 2024-09-16T05:39:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0178004970 con 0x7f01840a1520 2024-09-16T05:39:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(178..178 src has 1..178) ==== 7005+0+0 (unknown 2958250567 0 0) 0x7f0178044d80 con 0x7f01840a1520 2024-09-16T05:39:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.466+0000 7f0183fff640 10 monclient: get_auth_request con 0x7f0170036cd0 auth_method 0 2024-09-16T05:39:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.482+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0178015380 con 0x7f01840a1520 2024-09-16T05:39:30.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.482+0000 7f0180ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:30.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.482+0000 7f0180ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:30.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.654+0000 7f018a5b9640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T05:39:30.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.654+0000 7f018a5b9640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T05:39:30.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.654+0000 7f018a5b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:30.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:30.654+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) -- 0x7f0164005b90 con 0x7f01840a1520 2024-09-16T05:39:31.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.038+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 moved item id -19 name 'barr' to location {root=default} in crush map v179) ==== 173+0+0 (unknown 2609127767 0 0) 0x7f0178019030 con 0x7f01840a1520 2024-09-16T05:39:31.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.038+0000 7f0180ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T05:39:31.043 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.038+0000 7f0180ff9640 10 monclient: _finish_command 2 = system:0 moved item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T05:39:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.142+0000 7f018a5b9640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T05:39:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.142+0000 7f018a5b9640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T05:39:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.142+0000 7f018a5b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:31.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.142+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) -- 0x7f0164005610 con 0x7f01840a1520 2024-09-16T05:39:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f0180ff9640 1 -- 172.21.15.63:0/4025559794 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 no need to move item id -19 name 'barr' to location {root=default} in crush map v179) ==== 183+0+0 (unknown 2552598464 0 0) 0x7f01780406b0 con 0x7f01840a1520 2024-09-16T05:39:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f0180ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T05:39:31.149 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f0180ff9640 10 monclient: _finish_command 3 = system:0 no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T05:39:31.150 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T05:39:31.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0170036cd0 legacy=0x7f0170039190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f018a5b9640 10 monclient: shutdown 2024-09-16T05:39:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f018a5b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 >> v1:172.21.15.6:6790/0 conn(0x7f01840a1520 legacy=0x7f01841d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 shutdown_connections 2024-09-16T05:39:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 >> 172.21.15.63:0/4025559794 conn(0x7f01840a5290 msgr2=0x7f0184071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 shutdown_connections 2024-09-16T05:39:31.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.146+0000 7f018a5b9640 1 -- 172.21.15.63:0/4025559794 wait complete. 2024-09-16T05:39:31.168 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush move fooo rack=barr 2024-09-16T05:39:31.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.286+0000 7f07dcebc640 10 monclient: get_monmap_and_config 2024-09-16T05:39:31.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.286+0000 7f07dcebc640 10 monclient: build_initial_monmap 2024-09-16T05:39:31.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.286+0000 7f07dcebc640 10 monclient: monmap: 2024-09-16T05:39:31.291 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:31.291 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:31.291 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:31.291316+0000 2024-09-16T05:39:31.292 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:31.291316+0000 2024-09-16T05:39:31.292 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:31.292 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:31.292 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:31.292 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:31.292 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:31.292 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:31.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.286+0000 7f07dcebc640 1 Processor -- start 2024-09-16T05:39:31.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.286+0000 7f07dcebc640 1 -- start start 2024-09-16T05:39:31.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.286+0000 7f07dcebc640 10 monclient: init 2024-09-16T05:39:31.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:31.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:31.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient(hunting): picked mon.noname-c con 0x7f07d8136960 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:31.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient(hunting): picked mon.noname-a con 0x7f07d80a1c40 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:31.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient(hunting): picked mon.noname-b con 0x7f07d8137450 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d80a09c0 con 0x7f07d80a1c40 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f07d8136960 legacy=0x7f07d8136d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d80a0f80 con 0x7f07d8136960 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d80a4bb0 con 0x7f07d8137450 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient(hunting): authenticate will time out at 1228.068433s 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d6d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f07d8137450 0x7f07d813f4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60732/0 (socket says 172.21.15.63:60732) 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d6d9d640 1 -- 172.21.15.63:0/484465047 learned_addr learned my addr 172.21.15.63:0/484465047 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d5d9b640 1 --1- 172.21.15.63:0/484465047 >> v1:172.21.15.6:6789/0 conn(0x7f07d80a1c40 0x7f07d813bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57674/0 (socket says 172.21.15.63:57674) 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3055601099 0 0) 0x7f07d80a09c0 con 0x7f07d80a1c40 2024-09-16T05:39:31.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient(hunting): my global_id is 5943 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07c0003530 con 0x7f07d80a1c40 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 164769661 0 0) 0x7f07d80a4bb0 con 0x7f07d8137450 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:31.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient(hunting): my global_id is 5918 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d80a09c0 con 0x7f07d8137450 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1347033862 0 0) 0x7f07c0003530 con 0x7f07d80a1c40 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 >> v1:172.21.15.6:6790/0 conn(0x7f07d8137450 legacy=0x7f07d813f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 >> v1:172.21.15.63:6789/0 conn(0x7f07d8136960 legacy=0x7f07d8136d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: _finish_hunting 0 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 monclient: found mon.noname-a 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d80a4e60 con 0x7f07d80a1c40 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: _finish_auth 0 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: _check_auth_tickets 2024-09-16T05:39:31.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07cc001620 con 0x7f07d80a1c40 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 5 monclient: authenticate success, global_id 5943 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: dump: 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:31.300 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07cc001880 con 0x7f07d80a1c40 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 1 -- 172.21.15.63:0/484465047 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07cc001bd0 con 0x7f07d80a1c40 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 4 set_mon_vals no callback set 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:31.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07d559a640 10 monclient: dump: 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:31.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient: shutdown 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 1 -- 172.21.15.63:0/484465047 >> v1:172.21.15.6:6789/0 conn(0x7f07d80a1c40 legacy=0x7f07d813bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 1 -- 172.21.15.63:0/484465047 shutdown_connections 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 1 -- 172.21.15.63:0/484465047 >> 172.21.15.63:0/484465047 conn(0x7f07d80a60a0 msgr2=0x7f07d80a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 1 -- 172.21.15.63:0/484465047 shutdown_connections 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 1 -- 172.21.15.63:0/484465047 wait complete. 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient: build_initial_monmap 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.290+0000 7f07dcebc640 10 monclient: monmap: 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:31.303 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:31.304 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:31.304 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:31.304 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:31.304 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:31.304 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 1 Processor -- start 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 1 -- start start 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: init 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient(hunting): picked mon.noname-c con 0x7f07d80a1c40 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient(hunting): picked mon.noname-b con 0x7f07d8136960 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient(hunting): picked mon.noname-a con 0x7f07d8137450 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d80a0500 con 0x7f07d8137450 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07d5d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f07d8136960 legacy=0x7f07d8135c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:31.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d8124030 con 0x7f07d8136960 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d80a73a0 con 0x7f07d80a1c40 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient(hunting): authenticate will time out at 1228.072916s 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07d659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f07d80a1c40 0x7f07d81cadb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60734/0 (socket says 172.21.15.63:60734) 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07d659c640 1 -- 172.21.15.63:0/2427126857 learned_addr learned my addr 172.21.15.63:0/2427126857 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07d6d9d640 1 --1- 172.21.15.63:0/2427126857 >> v1:172.21.15.6:6789/0 conn(0x7f07d8137450 0x7f07d81d30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57688/0 (socket says 172.21.15.63:57688) 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 238494373 0 0) 0x7f07d80a73a0 con 0x7f07d80a1c40 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): my global_id is 5921 2024-09-16T05:39:31.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07b8003530 con 0x7f07d80a1c40 2024-09-16T05:39:31.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3610462494 0 0) 0x7f07d0001070 con 0x7f07d8137450 2024-09-16T05:39:31.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:31.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:31.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:31.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): my global_id is 5946 2024-09-16T05:39:31.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d80a73a0 con 0x7f07d8137450 2024-09-16T05:39:31.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1426280593 0 0) 0x7f07b8003530 con 0x7f07d80a1c40 2024-09-16T05:39:31.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07d0001070 con 0x7f07d80a1c40 2024-09-16T05:39:31.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07c8004520 con 0x7f07d80a1c40 2024-09-16T05:39:31.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:31.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:31.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:31.312 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:31.312 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:31.312 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.312 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.312 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3688948978 0 0) 0x7f07d80a73a0 con 0x7f07d8137450 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07b8003530 con 0x7f07d8137450 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07d0001840 con 0x7f07d8137450 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:31.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2439522180 0 0) 0x7f07d0001070 con 0x7f07d80a1c40 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 >> v1:172.21.15.63:6789/0 conn(0x7f07d8136960 legacy=0x7f07d8135c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 >> v1:172.21.15.6:6789/0 conn(0x7f07d8137450 legacy=0x7f07d81d30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:31.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 monclient: found mon.c 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d8122be0 con 0x7f07d80a1c40 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 5 monclient: authenticate success, global_id 5921 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: _renew_subs 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07d8136330 con 0x7f07d80a1c40 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: _renew_subs 2024-09-16T05:39:31.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f07d81d49e0 con 0x7f07d80a1c40 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07c8004740 con 0x7f07d80a1c40 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f07c8004b10 con 0x7f07d80a1c40 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07d4d99640 4 set_mon_vals no callback set 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07c6ffd640 10 monclient: dump: 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.316 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.294+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07bc005090 con 0x7f07d80a1c40 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.298+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f07c8004d30 con 0x7f07d80a1c40 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.298+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(179..179 src has 1..179) ==== 7193+0+0 (unknown 1791180605 0 0) 0x7f07c8045410 con 0x7f07d80a1c40 2024-09-16T05:39:31.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.298+0000 7f07d5d9b640 10 monclient: get_auth_request con 0x7f07b80369e0 auth_method 0 2024-09-16T05:39:31.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.310+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f07c8015870 con 0x7f07d80a1c40 2024-09-16T05:39:31.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.310+0000 7f07c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:31.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.310+0000 7f07c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:31.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.478+0000 7f07dcebc640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T05:39:31.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.478+0000 7f07dcebc640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T05:39:31.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.478+0000 7f07dcebc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:31.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:31.478+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) -- 0x7f07bc005b90 con 0x7f07d80a1c40 2024-09-16T05:39:32.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.222+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 moved item id -16 name 'fooo' to location {rack=barr} in crush map v180) ==== 167+0+0 (unknown 1441227628 0 0) 0x7f07c8015170 con 0x7f07d80a1c40 2024-09-16T05:39:32.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.222+0000 7f07c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T05:39:32.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.222+0000 7f07c6ffd640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T05:39:32.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.326+0000 7f07dcebc640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T05:39:32.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.326+0000 7f07dcebc640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T05:39:32.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.326+0000 7f07dcebc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:32.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.326+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) -- 0x7f07bc0010a0 con 0x7f07d80a1c40 2024-09-16T05:39:32.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07c6ffd640 1 -- 172.21.15.63:0/2427126857 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map v180) ==== 177+0+0 (unknown 1271692965 0 0) 0x7f07c80194c0 con 0x7f07d80a1c40 2024-09-16T05:39:32.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07c6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T05:39:32.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07c6ffd640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T05:39:32.333 INFO:tasks.workunit.client.0.smithi063.stderr:no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T05:39:32.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 >> v1:172.21.15.63:6812/3643207211 conn(0x7f07b80369e0 legacy=0x7f07b8038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:32.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07dcebc640 10 monclient: shutdown 2024-09-16T05:39:32.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07dcebc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:32.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 >> v1:172.21.15.6:6790/0 conn(0x7f07d80a1c40 legacy=0x7f07d81cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:32.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 shutdown_connections 2024-09-16T05:39:32.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 >> 172.21.15.63:0/2427126857 conn(0x7f07d80a60a0 msgr2=0x7f07d80a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:32.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 shutdown_connections 2024-09-16T05:39:32.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.330+0000 7f07dcebc640 1 -- 172.21.15.63:0/2427126857 wait complete. 2024-09-16T05:39:32.350 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm fooo 2024-09-16T05:39:32.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient: get_monmap_and_config 2024-09-16T05:39:32.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient: build_initial_monmap 2024-09-16T05:39:32.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient: monmap: 2024-09-16T05:39:32.474 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:32.474 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:32.474 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:32.473919+0000 2024-09-16T05:39:32.474 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:32.473919+0000 2024-09-16T05:39:32.474 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:32.475 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:32.475 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:32.475 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:32.475 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:32.475 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 1 Processor -- start 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 1 -- start start 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient: init 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient(hunting): picked mon.noname-b con 0x7f201809ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient(hunting): picked mon.noname-a con 0x7f201806dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient(hunting): picked mon.noname-c con 0x7f20180a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f201809f800 con 0x7f201806dd30 2024-09-16T05:39:32.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20180a0ab0 con 0x7f20180a0210 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20180a0d90 con 0x7f201809ac90 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f201da65640 10 monclient(hunting): authenticate will time out at 1229.251007s 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f2017fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f20180a0210 legacy=0x7f201813d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20177fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f201809ac90 0x7f201809b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60744/0 (socket says 172.21.15.63:60744) 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20177fe640 1 -- 172.21.15.63:0/3451439667 learned_addr learned my addr 172.21.15.63:0/3451439667 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f2016ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f201806dd30 0x7f20181399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57702/0 (socket says 172.21.15.63:57702) 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 42824149 0 0) 0x7f201809f800 con 0x7f201806dd30 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:32.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 10 monclient(hunting): my global_id is 5949 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2000003530 con 0x7f201806dd30 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965016283 0 0) 0x7f20180a0d90 con 0x7f201809ac90 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 10 monclient(hunting): my global_id is 5924 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.470+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f201809f800 con 0x7f201809ac90 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3024103569 0 0) 0x7f2000003530 con 0x7f201806dd30 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 >> v1:172.21.15.6:6790/0 conn(0x7f201809ac90 legacy=0x7f201809b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 >> v1:172.21.15.63:6789/0 conn(0x7f20180a0210 legacy=0x7f201813d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:32.485 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 1 monclient: found mon.noname-a 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201809e770 con 0x7f201806dd30 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f200c001550 con 0x7f201806dd30 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 5 monclient: authenticate success, global_id 5949 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:32.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:32.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:32.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: dump: 2024-09-16T05:39:32.487 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:32.487 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:32.487 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.487 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f200c0017b0 con 0x7f201806dd30 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:32.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 1 -- 172.21.15.63:0/3451439667 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f200c001b80 con 0x7f201806dd30 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 4 set_mon_vals no callback set 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f20167fc640 10 monclient: dump: 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:32.489 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient: shutdown 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- 172.21.15.63:0/3451439667 >> v1:172.21.15.6:6789/0 conn(0x7f201806dd30 legacy=0x7f20181399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- 172.21.15.63:0/3451439667 shutdown_connections 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- 172.21.15.63:0/3451439667 >> 172.21.15.63:0/3451439667 conn(0x7f20180a1c90 msgr2=0x7f20180a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- 172.21.15.63:0/3451439667 shutdown_connections 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- 172.21.15.63:0/3451439667 wait complete. 2024-09-16T05:39:32.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient: build_initial_monmap 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient: monmap: 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 Processor -- start 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- start start 2024-09-16T05:39:32.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient: init 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient(hunting): picked mon.noname-c con 0x7f201806dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient(hunting): picked mon.noname-b con 0x7f201809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient(hunting): picked mon.noname-a con 0x7f20180a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f201811eef0 con 0x7f20180a0210 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f201806e1f0 con 0x7f201809ac90 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f2016ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f201809ac90 legacy=0x7f20181cdfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f201809c0f0 con 0x7f201806dd30 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f201da65640 10 monclient(hunting): authenticate will time out at 1229.254744s 2024-09-16T05:39:32.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f2017fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f20180a0210 0x7f20181d1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57718/0 (socket says 172.21.15.63:57718) 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f2017fff640 1 -- 172.21.15.63:0/3065274276 learned_addr learned my addr 172.21.15.63:0/3065274276 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1675373336 0 0) 0x7f201811eef0 con 0x7f20180a0210 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 10 monclient(hunting): my global_id is 5952 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2004003530 con 0x7f20180a0210 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102656730 0 0) 0x7f201809c0f0 con 0x7f201806dd30 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 10 monclient(hunting): my global_id is 5927 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.474+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f201811eef0 con 0x7f201806dd30 2024-09-16T05:39:32.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 404926094 0 0) 0x7f2004003530 con 0x7f20180a0210 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f201809c0f0 con 0x7f20180a0210 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20100043c0 con 0x7f20180a0210 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient(hunting): dump: 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:32.494 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1333654512 0 0) 0x7f201811eef0 con 0x7f201806dd30 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2004003530 con 0x7f201806dd30 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2008001a00 con 0x7f201806dd30 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient(hunting): dump: 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.495 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1259164437 0 0) 0x7f201809c0f0 con 0x7f20180a0210 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 >> v1:172.21.15.6:6790/0 conn(0x7f201806dd30 legacy=0x7f2018133820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 >> v1:172.21.15.63:6789/0 conn(0x7f201809ac90 legacy=0x7f20181cdfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 monclient: found mon.a 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:32.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20181211e0 con 0x7f20180a0210 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient: _finish_auth 0 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 5 monclient: authenticate success, global_id 5952 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 10 monclient: _renew_subs 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2018134490 con 0x7f20180a0210 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 10 monclient: _renew_subs 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:32.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2018134100 con 0x7f20180a0210 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2010004560 con 0x7f20180a0210 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2010004930 con 0x7f20180a0210 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f20157fa640 4 set_mon_vals no callback set 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 10 monclient: dump: 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:32.498 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff8005090 con 0x7f20180a0210 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2010004b50 con 0x7f20180a0210 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(180..180 src has 1..180) ==== 7193+0+0 (unknown 1822393393 0 0) 0x7f2010045130 con 0x7f20180a0210 2024-09-16T05:39:32.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.478+0000 7f20177fe640 10 monclient: get_auth_request con 0x7f20040369e0 auth_method 0 2024-09-16T05:39:32.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.490+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2010015590 con 0x7f20180a0210 2024-09-16T05:39:32.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.490+0000 7f1ffffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:32.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.490+0000 7f1ffffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:32.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.662+0000 7f201da65640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T05:39:32.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.662+0000 7f201da65640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T05:39:32.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.662+0000 7f201da65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:32.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:32.662+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) -- 0x7f1ff8005380 con 0x7f20180a0210 2024-09-16T05:39:33.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.226+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 removed item id -16 name 'fooo' from crush map v181) ==== 122+0+0 (unknown 3509789452 0 0) 0x7f2010014e90 con 0x7f20180a0210 2024-09-16T05:39:33.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.226+0000 7f1ffffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T05:39:33.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.226+0000 7f1ffffff640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'fooo' from crush map 2024-09-16T05:39:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.330+0000 7f201da65640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T05:39:33.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.330+0000 7f201da65640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T05:39:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.330+0000 7f201da65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.330+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) -- 0x7f1ff80010a0 con 0x7f20180a0210 2024-09-16T05:39:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f1ffffff640 1 -- 172.21.15.63:0/3065274276 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 device 'fooo' does not appear in the crush map v181) ==== 122+0+0 (unknown 4187174365 0 0) 0x7f20100191e0 con 0x7f20180a0210 2024-09-16T05:39:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f1ffffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T05:39:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f1ffffff640 10 monclient: _finish_command 3 = system:0 device 'fooo' does not appear in the crush map 2024-09-16T05:39:33.336 INFO:tasks.workunit.client.0.smithi063.stderr:device 'fooo' does not appear in the crush map 2024-09-16T05:39:33.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 >> v1:172.21.15.63:6812/3643207211 conn(0x7f20040369e0 legacy=0x7f2004038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:33.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f201da65640 10 monclient: shutdown 2024-09-16T05:39:33.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f201da65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:33.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 >> v1:172.21.15.6:6789/0 conn(0x7f20180a0210 legacy=0x7f20181d1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:33.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 shutdown_connections 2024-09-16T05:39:33.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 >> 172.21.15.63:0/3065274276 conn(0x7f20180a1c90 msgr2=0x7f20180a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:33.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 shutdown_connections 2024-09-16T05:39:33.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.334+0000 7f201da65640 1 -- 172.21.15.63:0/3065274276 wait complete. 2024-09-16T05:39:33.353 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm barr 2024-09-16T05:39:33.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient: get_monmap_and_config 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient: build_initial_monmap 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient: monmap: 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:33.474724+0000 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:33.474724+0000 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:33.475 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 1 Processor -- start 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 1 -- start start 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient: init 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient(hunting): picked mon.noname-b con 0x7f5c0c06dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0c0a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:33.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c09f800 con 0x7f5c0c09ac90 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c0a0ab0 con 0x7f5c0c0a0210 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c0a0d90 con 0x7f5c0c06dd30 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c12f14640 10 monclient(hunting): authenticate will time out at 1230.251764s 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.470+0000 7f5c114b1640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5c0c0a0210 legacy=0x7f5c0c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c10cb0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5c0c09ac90 0x7f5c0c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:57734/0 (socket says 172.21.15.63:57734) 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5c0c06dd30 0x7f5c0c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60754/0 (socket says 172.21.15.63:60754) 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c10cb0640 1 -- 172.21.15.63:0/236562660 learned_addr learned my addr 172.21.15.63:0/236562660 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 876371757 0 0) 0x7f5c0c09f800 con 0x7f5c0c09ac90 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:33.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient(hunting): my global_id is 5955 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bfc003530 con 0x7f5c0c09ac90 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745687828 0 0) 0x7f5c0c0a0d90 con 0x7f5c0c06dd30 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient(hunting): my global_id is 5930 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c0c09f800 con 0x7f5c0c06dd30 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3614831932 0 0) 0x7f5bfc003530 con 0x7f5c0c09ac90 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 >> v1:172.21.15.6:6790/0 conn(0x7f5c0c06dd30 legacy=0x7f5c0c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 >> v1:172.21.15.63:6789/0 conn(0x7f5c0c0a0210 legacy=0x7f5c0c13d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:33.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 monclient: found mon.noname-a 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c09e770 con 0x7f5c0c09ac90 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5c00001620 con 0x7f5c0c09ac90 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 5 monclient: authenticate success, global_id 5955 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:33.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: dump: 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:33.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c000018c0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 1 -- 172.21.15.63:0/236562660 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5c00001cd0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 4 set_mon_vals no callback set 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c0b7fe640 10 monclient: dump: 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:33.483 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:33.484 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:33.484 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:33.484 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:33.484 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:33.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient: shutdown 2024-09-16T05:39:33.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:33.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- 172.21.15.63:0/236562660 >> v1:172.21.15.6:6789/0 conn(0x7f5c0c09ac90 legacy=0x7f5c0c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:33.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- 172.21.15.63:0/236562660 shutdown_connections 2024-09-16T05:39:33.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- 172.21.15.63:0/236562660 >> 172.21.15.63:0/236562660 conn(0x7f5c0c0a1c90 msgr2=0x7f5c0c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:33.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- 172.21.15.63:0/236562660 shutdown_connections 2024-09-16T05:39:33.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- 172.21.15.63:0/236562660 wait complete. 2024-09-16T05:39:33.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient: build_initial_monmap 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient: monmap: 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:33.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 Processor -- start 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- start start 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient: init 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0c06dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient(hunting): picked mon.noname-b con 0x7f5c0c0a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c127770 con 0x7f5c0c09ac90 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c127a50 con 0x7f5c0c0a0210 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c06e6b0 con 0x7f5c0c06dd30 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c12f14640 10 monclient(hunting): authenticate will time out at 1230.255759s 2024-09-16T05:39:33.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.474+0000 7f5c114b1640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5c0c0a0210 legacy=0x7f5c0c1d3810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c10cb0640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5c0c06dd30 0x7f5c0c09c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:60768/0 (socket says 172.21.15.63:60768) 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c10cb0640 1 -- 172.21.15.63:0/371533525 learned_addr learned my addr 172.21.15.63:0/371533525 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3524219633 0 0) 0x7f5c0c127770 con 0x7f5c0c09ac90 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): my global_id is 5958 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf0003530 con 0x7f5c0c09ac90 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097405225 0 0) 0x7f5c0c06e6b0 con 0x7f5c0c06dd30 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:33.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): my global_id is 5933 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c0c127770 con 0x7f5c0c06dd30 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 415934715 0 0) 0x7f5bf0003530 con 0x7f5c0c09ac90 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c0c06e6b0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5bf80017a0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): dump: 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.492 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3241724311 0 0) 0x7f5c0c127770 con 0x7f5c0c06dd30 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bf0003530 con 0x7f5c0c06dd30 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5c00001520 con 0x7f5c0c06dd30 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:33.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient(hunting): dump: 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2411434684 0 0) 0x7f5c0c06e6b0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 >> v1:172.21.15.6:6790/0 conn(0x7f5c0c06dd30 legacy=0x7f5c0c09c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 >> v1:172.21.15.63:6789/0 conn(0x7f5c0c0a0210 legacy=0x7f5c0c1d3810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 monclient: found mon.a 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c06e160 con 0x7f5c0c09ac90 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 5 monclient: authenticate success, global_id 5958 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 10 monclient: _renew_subs 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:33.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0c09c0f0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 10 monclient: _renew_subs 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c0c126040 con 0x7f5c0c09ac90 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5bf80047f0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5bf8004b40 con 0x7f5c0c09ac90 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c0affd640 4 set_mon_vals no callback set 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 10 monclient: dump: 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:33.496 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bec005090 con 0x7f5c0c09ac90 2024-09-16T05:39:33.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5bf800f8e0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(181..181 src has 1..181) ==== 6977+0+0 (unknown 748488134 0 0) 0x7f5bf8044cd0 con 0x7f5c0c09ac90 2024-09-16T05:39:33.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.478+0000 7f5c10cb0640 10 monclient: get_auth_request con 0x7f5bf0036aa0 auth_method 0 2024-09-16T05:39:33.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.490+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5bf8015260 con 0x7f5c0c09ac90 2024-09-16T05:39:33.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.490+0000 7f5c097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:33.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.490+0000 7f5c097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:33.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.662+0000 7f5c12f14640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T05:39:33.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.662+0000 7f5c12f14640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T05:39:33.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.662+0000 7f5c12f14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:33.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:33.662+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) -- 0x7f5bec005380 con 0x7f5c0c09ac90 2024-09-16T05:39:34.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.234+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 removed item id -19 name 'barr' from crush map v182) ==== 122+0+0 (unknown 4069422836 0 0) 0x7f5bf8014b60 con 0x7f5c0c09ac90 2024-09-16T05:39:34.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.234+0000 7f5c097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T05:39:34.241 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.234+0000 7f5c097fa640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'barr' from crush map 2024-09-16T05:39:34.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.342+0000 7f5c12f14640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T05:39:34.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.342+0000 7f5c12f14640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T05:39:34.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.342+0000 7f5c12f14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:34.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.342+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) -- 0x7f5bec0010a0 con 0x7f5c0c09ac90 2024-09-16T05:39:34.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.342+0000 7f5c097fa640 1 -- 172.21.15.63:0/371533525 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 device 'barr' does not appear in the crush map v182) ==== 122+0+0 (unknown 3109098347 0 0) 0x7f5bf801d710 con 0x7f5c0c09ac90 2024-09-16T05:39:34.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.342+0000 7f5c097fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T05:39:34.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.342+0000 7f5c097fa640 10 monclient: _finish_command 3 = system:0 device 'barr' does not appear in the crush map 2024-09-16T05:39:34.348 INFO:tasks.workunit.client.0.smithi063.stderr:device 'barr' does not appear in the crush map 2024-09-16T05:39:34.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.346+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5bf0036aa0 legacy=0x7f5bf0038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:34.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.346+0000 7f5c12f14640 10 monclient: shutdown 2024-09-16T05:39:34.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.346+0000 7f5c12f14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:34.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.346+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 >> v1:172.21.15.6:6789/0 conn(0x7f5c0c09ac90 legacy=0x7f5c0c09cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:34.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.346+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 shutdown_connections 2024-09-16T05:39:34.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.346+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 >> 172.21.15.63:0/371533525 conn(0x7f5c0c0a1c90 msgr2=0x7f5c0c11b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:34.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.346+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 shutdown_connections 2024-09-16T05:39:34.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.346+0000 7f5c12f14640 1 -- 172.21.15.63:0/371533525 wait complete. 2024-09-16T05:39:34.364 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T05:39:34.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient: get_monmap_and_config 2024-09-16T05:39:34.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient: build_initial_monmap 2024-09-16T05:39:34.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient: monmap: 2024-09-16T05:39:34.487 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:34.487 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:34.487 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:34.486643+0000 2024-09-16T05:39:34.487 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:34.486643+0000 2024-09-16T05:39:34.487 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:34.487 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:34.488 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:34.488 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:34.488 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:34.488 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 1 Processor -- start 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 1 -- start start 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient: init 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient(hunting): picked mon.noname-b con 0x7f42b40a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient(hunting): picked mon.noname-a con 0x7f42b409b450 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient(hunting): picked mon.noname-c con 0x7f42b409b8a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b409e970 con 0x7f42b409b450 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b409c370 con 0x7f42b409b8a0 2024-09-16T05:39:34.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b409d620 con 0x7f42b40a08a0 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42ba4ef640 10 monclient(hunting): authenticate will time out at 1231.263696s 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.482+0000 7f42b8a8c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f42b409b8a0 legacy=0x7f42b413c0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f42b40a08a0 0x7f42b409ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59296/0 (socket says 172.21.15.63:59296) 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f42b409b450 0x7f42b41389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35280/0 (socket says 172.21.15.63:35280) 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b3fff640 1 -- 172.21.15.63:0/1238912952 learned_addr learned my addr 172.21.15.63:0/1238912952 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423085849 0 0) 0x7f42b409e970 con 0x7f42b409b450 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient(hunting): my global_id is 5961 2024-09-16T05:39:34.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f429c003530 con 0x7f42b409b450 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 974770202 0 0) 0x7f42b409d620 con 0x7f42b40a08a0 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient(hunting): my global_id is 5936 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b409e970 con 0x7f42b40a08a0 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1606348443 0 0) 0x7f429c003530 con 0x7f42b409b450 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 >> v1:172.21.15.6:6790/0 conn(0x7f42b40a08a0 legacy=0x7f42b409ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 >> v1:172.21.15.63:6789/0 conn(0x7f42b409b8a0 legacy=0x7f42b413c0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 monclient: found mon.noname-a 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:34.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b409d900 con 0x7f42b409b450 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 5 monclient: authenticate success, global_id 5961 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a8001620 con 0x7f42b409b450 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: dump: 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:34.494 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42a80018d0 con 0x7f42b409b450 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:34.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 1 -- 172.21.15.63:0/1238912952 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a8001ca0 con 0x7f42b409b450 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 4 set_mon_vals no callback set 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b2ffd640 10 monclient: dump: 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:34.496 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient: shutdown 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1238912952 >> v1:172.21.15.6:6789/0 conn(0x7f42b409b450 legacy=0x7f42b41389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1238912952 shutdown_connections 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1238912952 >> 172.21.15.63:0/1238912952 conn(0x7f42b40a1c90 msgr2=0x7f42b40a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1238912952 shutdown_connections 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1238912952 wait complete. 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient: build_initial_monmap 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient: monmap: 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:34.497 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 Processor -- start 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- start start 2024-09-16T05:39:34.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient: init 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient(hunting): picked mon.noname-a con 0x7f42b409b450 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient(hunting): picked mon.noname-b con 0x7f42b409b8a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient(hunting): picked mon.noname-c con 0x7f42b40a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b411e910 con 0x7f42b409b450 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b411eed0 con 0x7f42b409b8a0 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b40a41e0 con 0x7f42b40a08a0 2024-09-16T05:39:34.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42ba4ef640 10 monclient(hunting): authenticate will time out at 1231.267454s 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b37fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f42b409b8a0 legacy=0x7f42b41d0a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f42b409b450 0x7f42b4133020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35294/0 (socket says 172.21.15.63:35294) 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b8a8c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f42b40a08a0 0x7f42b41d41b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59300/0 (socket says 172.21.15.63:59300) 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.486+0000 7f42b3fff640 1 -- 172.21.15.63:0/1099413732 learned_addr learned my addr 172.21.15.63:0/1099413732 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4122737036 0 0) 0x7f42b411e910 con 0x7f42b409b450 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): my global_id is 5964 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42a0003530 con 0x7f42b409b450 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 915525592 0 0) 0x7f42b40a41e0 con 0x7f42b40a08a0 2024-09-16T05:39:34.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): my global_id is 5939 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42b411e910 con 0x7f42b40a08a0 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 324472147 0 0) 0x7f42a0003530 con 0x7f42b409b450 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42b40a41e0 con 0x7f42b409b450 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a4001810 con 0x7f42b409b450 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:34.504 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3021931960 0 0) 0x7f42b411e910 con 0x7f42b40a08a0 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42a0003530 con 0x7f42b40a08a0 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42ac004340 con 0x7f42b40a08a0 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:34.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient(hunting): dump: 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:34.506 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4128793622 0 0) 0x7f42b40a41e0 con 0x7f42b409b450 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 >> v1:172.21.15.6:6790/0 conn(0x7f42b40a08a0 legacy=0x7f42b41d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 >> v1:172.21.15.63:6789/0 conn(0x7f42b409b8a0 legacy=0x7f42b41d0a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 monclient: found mon.a 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b411e630 con 0x7f42b409b450 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 5 monclient: authenticate success, global_id 5964 2024-09-16T05:39:34.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 10 monclient: _renew_subs 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42b411ebf0 con 0x7f42b409b450 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 10 monclient: _renew_subs 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42b411c3f0 con 0x7f42b409b450 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42a40047f0 con 0x7f42b409b450 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f42a4004bc0 con 0x7f42b409b450 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 10 monclient: dump: 2024-09-16T05:39:34.508 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b27fc640 4 set_mon_vals no callback set 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:34.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:34.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:34.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4294005090 con 0x7f42b409b450 2024-09-16T05:39:34.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f42a4004de0 con 0x7f42b409b450 2024-09-16T05:39:34.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(182..182 src has 1..182) ==== 6761+0+0 (unknown 3658027069 0 0) 0x7f42a4044cb0 con 0x7f42b409b450 2024-09-16T05:39:34.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.490+0000 7f42b37fe640 10 monclient: get_auth_request con 0x7f42a00369d0 auth_method 0 2024-09-16T05:39:34.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.502+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f42a4019070 con 0x7f42b409b450 2024-09-16T05:39:34.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.502+0000 7f42b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:34.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.502+0000 7f42b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:34.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.666+0000 7f42ba4ef640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:34.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.666+0000 7f42ba4ef640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:34.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.666+0000 7f42ba4ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:34.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:34.666+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7f4294005380 con 0x7f42b409b450 2024-09-16T05:39:35.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:35.246+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v183) ==== 78+0+0 (unknown 279831757 0 0) 0x7f42a401d930 con 0x7f42b409b450 2024-09-16T05:39:35.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:35.246+0000 7f42b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:35.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:35.246+0000 7f42b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:35.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:35.350+0000 7f42ba4ef640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:35.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:35.350+0000 7f42ba4ef640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:35.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:35.350+0000 7f42ba4ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:35.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:35.350+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7f4294005770 con 0x7f42b409b450 2024-09-16T05:39:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.250+0000 7f42b0ff9640 1 -- 172.21.15.63:0/1099413732 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v184) ==== 78+0+0 (unknown 3587561485 0 0) 0x7f42a40404b0 con 0x7f42b409b450 2024-09-16T05:39:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.250+0000 7f42b0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:39:36.256 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.250+0000 7f42b0ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:39:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.254+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 >> v1:172.21.15.63:6812/3643207211 conn(0x7f42a00369d0 legacy=0x7f42a0038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.254+0000 7f42ba4ef640 10 monclient: shutdown 2024-09-16T05:39:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.254+0000 7f42ba4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.254+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 >> v1:172.21.15.6:6789/0 conn(0x7f42b409b450 legacy=0x7f42b4133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:36.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.254+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 shutdown_connections 2024-09-16T05:39:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.254+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 >> 172.21.15.63:0/1099413732 conn(0x7f42b40a1c90 msgr2=0x7f42b413cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.254+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 shutdown_connections 2024-09-16T05:39:36.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.254+0000 7f42ba4ef640 1 -- 172.21.15.63:0/1099413732 wait complete. 2024-09-16T05:39:36.281 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T05:39:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient: get_monmap_and_config 2024-09-16T05:39:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient: build_initial_monmap 2024-09-16T05:39:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient: monmap: 2024-09-16T05:39:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:36.406158+0000 2024-09-16T05:39:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:36.406158+0000 2024-09-16T05:39:36.406 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:36.407 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 1 Processor -- start 2024-09-16T05:39:36.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 1 -- start start 2024-09-16T05:39:36.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient: init 2024-09-16T05:39:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:36.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient(hunting): picked mon.noname-a con 0x7f15440a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient(hunting): picked mon.noname-c con 0x7f154409b3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 10 monclient(hunting): picked mon.noname-b con 0x7f154409b830 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.402+0000 7f154c470640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f154409ea70 con 0x7f15440a08a0 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f154409c5f0 con 0x7f154409b3a0 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f154409c370 con 0x7f154409b830 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient(hunting): authenticate will time out at 1233.184049s 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f1549a0b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f154409b3a0 legacy=0x7f15441389b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154aa0d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f154409b830 0x7f154413c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59306/0 (socket says 172.21.15.63:59306) 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154aa0d640 1 -- 172.21.15.63:0/2879228976 learned_addr learned my addr 172.21.15.63:0/2879228976 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:36.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154a20c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f15440a08a0 0x7f154409ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35304/0 (socket says 172.21.15.63:35304) 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183171647 0 0) 0x7f154409c370 con 0x7f154409b830 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient(hunting): my global_id is 5942 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f152c003530 con 0x7f154409b830 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554474354 0 0) 0x7f15340014a0 con 0x7f15440a08a0 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:36.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient(hunting): my global_id is 5967 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f154409c370 con 0x7f15440a08a0 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3933572247 0 0) 0x7f154409c370 con 0x7f15440a08a0 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 >> v1:172.21.15.6:6790/0 conn(0x7f154409b830 legacy=0x7f154413c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 >> v1:172.21.15.63:6789/0 conn(0x7f154409b3a0 legacy=0x7f15441389b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: _finish_hunting 0 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 monclient: found mon.noname-a 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154409d620 con 0x7f15440a08a0 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: _finish_auth 0 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: _check_auth_tickets 2024-09-16T05:39:36.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1534001ad0 con 0x7f15440a08a0 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 5 monclient: authenticate success, global_id 5967 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: dump: 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:36.414 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1534001dc0 con 0x7f15440a08a0 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 1 -- 172.21.15.63:0/2879228976 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1534004b70 con 0x7f15440a08a0 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154920a640 10 monclient: dump: 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:36.415 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 4 set_mon_vals no callback set 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient: shutdown 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:36.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- 172.21.15.63:0/2879228976 >> v1:172.21.15.6:6789/0 conn(0x7f15440a08a0 legacy=0x7f154409ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:36.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- 172.21.15.63:0/2879228976 shutdown_connections 2024-09-16T05:39:36.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- 172.21.15.63:0/2879228976 >> 172.21.15.63:0/2879228976 conn(0x7f15440a1c90 msgr2=0x7f15440a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:36.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- 172.21.15.63:0/2879228976 shutdown_connections 2024-09-16T05:39:36.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- 172.21.15.63:0/2879228976 wait complete. 2024-09-16T05:39:36.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient: build_initial_monmap 2024-09-16T05:39:36.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient: monmap: 2024-09-16T05:39:36.421 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:36.421 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:36.421 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 Processor -- start 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- start start 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient: init 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:36.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient(hunting): picked mon.noname-c con 0x7f154409b3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient(hunting): picked mon.noname-a con 0x7f154409b830 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient(hunting): picked mon.noname-b con 0x7f15440a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1544124400 con 0x7f154409b830 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15441343e0 con 0x7f15440a08a0 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15440a3d80 con 0x7f154409b3a0 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154c470640 10 monclient(hunting): authenticate will time out at 1233.187985s 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.406+0000 7f154aa0d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f15440a08a0 legacy=0x7f15441d3950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154a20c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f154409b3a0 0x7f1544133cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59318/0 (socket says 172.21.15.63:59318) 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154a20c640 1 -- 172.21.15.63:0/1085082871 learned_addr learned my addr 172.21.15.63:0/1085082871 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421815912 0 0) 0x7f15440a3d80 con 0x7f154409b3a0 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:36.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): my global_id is 5945 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1530003530 con 0x7f154409b3a0 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325329418 0 0) 0x7f1544124400 con 0x7f154409b830 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): my global_id is 5970 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15440a3d80 con 0x7f154409b830 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1121860571 0 0) 0x7f1530003530 con 0x7f154409b3a0 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1544124400 con 0x7f154409b3a0 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2115013101 0 0) 0x7f15440a3d80 con 0x7f154409b830 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1530003530 con 0x7f154409b830 2024-09-16T05:39:36.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f15340018f0 con 0x7f154409b3a0 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:36.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1538004050 con 0x7f154409b830 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:36.426 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3482525159 0 0) 0x7f1530003530 con 0x7f154409b830 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 >> v1:172.21.15.6:6790/0 conn(0x7f154409b3a0 legacy=0x7f1544133cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 >> v1:172.21.15.63:6789/0 conn(0x7f15440a08a0 legacy=0x7f15441d3950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 monclient: found mon.a 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15441cbd20 con 0x7f154409b830 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:36.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 5 monclient: authenticate success, global_id 5970 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 10 monclient: _renew_subs 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f154406f830 con 0x7f154409b830 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 10 monclient: _renew_subs 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1544123220 con 0x7f154409b830 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f15380045d0 con 0x7f154409b830 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1538004920 con 0x7f154409b830 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1548a09640 4 set_mon_vals no callback set 2024-09-16T05:39:36.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 10 monclient: dump: 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:36.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:36.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1524005090 con 0x7f154409b830 2024-09-16T05:39:36.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1538004ac0 con 0x7f154409b830 2024-09-16T05:39:36.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(184..184 src has 1..184) ==== 6433+0+0 (unknown 3732296802 0 0) 0x7f1538044a60 con 0x7f154409b830 2024-09-16T05:39:36.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.410+0000 7f154a20c640 10 monclient: get_auth_request con 0x7f15300369f0 auth_method 0 2024-09-16T05:39:36.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.426+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1538015270 con 0x7f154409b830 2024-09-16T05:39:36.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.426+0000 7f1542ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:36.430 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.426+0000 7f1542ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:36.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.594+0000 7f154c470640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:36.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.594+0000 7f154c470640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:36.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.594+0000 7f154c470640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:36.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:36.594+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7f1524005380 con 0x7f154409b830 2024-09-16T05:39:37.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.254+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v185) ==== 82+0+0 (unknown 163878420 0 0) 0x7f1538014b70 con 0x7f154409b830 2024-09-16T05:39:37.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.254+0000 7f1542ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:37.259 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.254+0000 7f1542ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:39:37.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.362+0000 7f154c470640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:37.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.362+0000 7f154c470640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:37.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.362+0000 7f154c470640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:37.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.362+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7f1524005770 con 0x7f154409b830 2024-09-16T05:39:37.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f1542ffd640 1 -- 172.21.15.63:0/1085082871 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v185) ==== 115+0+0 (unknown 437855205 0 0) 0x7f153801d720 con 0x7f154409b830 2024-09-16T05:39:37.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f1542ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T05:39:37.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f1542ffd640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T05:39:37.368 INFO:tasks.workunit.client.0.smithi063.stderr:compat weight-set already created 2024-09-16T05:39:37.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 >> v1:172.21.15.63:6812/3643207211 conn(0x7f15300369f0 legacy=0x7f1530038eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:37.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f154c470640 10 monclient: shutdown 2024-09-16T05:39:37.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f154c470640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:37.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 >> v1:172.21.15.6:6789/0 conn(0x7f154409b830 legacy=0x7f15441d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:37.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 shutdown_connections 2024-09-16T05:39:37.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 >> 172.21.15.63:0/1085082871 conn(0x7f15440a1c90 msgr2=0x7f15441bb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:37.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 shutdown_connections 2024-09-16T05:39:37.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.366+0000 7f154c470640 1 -- 172.21.15.63:0/1085082871 wait complete. 2024-09-16T05:39:37.385 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket r1 rack root=default 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient: get_monmap_and_config 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient: build_initial_monmap 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient: monmap: 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:37.509908+0000 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:37.509908+0000 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:37.510 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:37.511 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:37.511 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:37.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 1 Processor -- start 2024-09-16T05:39:37.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 1 -- start start 2024-09-16T05:39:37.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient: init 2024-09-16T05:39:37.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:37.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:37.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient(hunting): picked mon.noname-a con 0x7f18c00a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:37.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient(hunting): picked mon.noname-b con 0x7f18c00a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:37.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient(hunting): picked mon.noname-c con 0x7f18c00a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00a53b0 con 0x7f18c00a3b10 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00a0290 con 0x7f18c00a4600 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c5865640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f18c00a4600 legacy=0x7f18c013f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00a0dc0 con 0x7f18c00a07a0 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c72c8640 10 monclient(hunting): authenticate will time out at 1234.287169s 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c5064640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f18c00a3b10 0x7f18c00a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35326/0 (socket says 172.21.15.63:35326) 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c5064640 1 -- 172.21.15.63:0/766042953 learned_addr learned my addr 172.21.15.63:0/766042953 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.506+0000 7f18c4863640 1 --1- 172.21.15.63:0/766042953 >> v1:172.21.15.6:6790/0 conn(0x7f18c00a07a0 0x7f18c013be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59320/0 (socket says 172.21.15.63:59320) 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2007516832 0 0) 0x7f18c00a53b0 con 0x7f18c00a3b10 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:37.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient(hunting): my global_id is 5973 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18ac003530 con 0x7f18c00a3b10 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 637335034 0 0) 0x7f18c00a0dc0 con 0x7f18c00a07a0 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient(hunting): my global_id is 5948 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18c00a53b0 con 0x7f18c00a07a0 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2277709291 0 0) 0x7f18c00a53b0 con 0x7f18c00a07a0 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 >> v1:172.21.15.63:6789/0 conn(0x7f18c00a4600 legacy=0x7f18c013f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:37.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 >> v1:172.21.15.6:6789/0 conn(0x7f18c00a3b10 legacy=0x7f18c00a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:37.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:37.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 monclient: found mon.noname-b 2024-09-16T05:39:37.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:37.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:37.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c00a10a0 con 0x7f18c00a07a0 2024-09-16T05:39:37.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: _finish_auth 0 2024-09-16T05:39:37.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:37.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:37.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18b4001600 con 0x7f18c00a07a0 2024-09-16T05:39:37.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 5 monclient: authenticate success, global_id 5948 2024-09-16T05:39:37.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: dump: 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:37.521 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18b4001860 con 0x7f18c00a07a0 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 1 -- 172.21.15.63:0/766042953 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18b4001c30 con 0x7f18c00a07a0 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18b3fff640 10 monclient: dump: 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:37.522 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 4 set_mon_vals no callback set 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient: shutdown 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- 172.21.15.63:0/766042953 >> v1:172.21.15.6:6790/0 conn(0x7f18c00a07a0 legacy=0x7f18c013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- 172.21.15.63:0/766042953 shutdown_connections 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- 172.21.15.63:0/766042953 >> 172.21.15.63:0/766042953 conn(0x7f18c0075470 msgr2=0x7f18c0075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- 172.21.15.63:0/766042953 shutdown_connections 2024-09-16T05:39:37.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- 172.21.15.63:0/766042953 wait complete. 2024-09-16T05:39:37.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient: build_initial_monmap 2024-09-16T05:39:37.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient: monmap: 2024-09-16T05:39:37.528 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:37.528 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:37.528 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:37.528 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:37.528 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:37.528 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 Processor -- start 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- start start 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient: init 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient(hunting): picked mon.noname-c con 0x7f18c00a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient(hunting): picked mon.noname-a con 0x7f18c00a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient(hunting): picked mon.noname-b con 0x7f18c00a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c012a000 con 0x7f18c00a3b10 2024-09-16T05:39:37.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00a84d0 con 0x7f18c00a4600 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c0129370 con 0x7f18c00a07a0 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c72c8640 10 monclient(hunting): authenticate will time out at 1234.291437s 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c5865640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f18c00a4600 legacy=0x7f18c01d8080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c5064640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f18c00a07a0 0x7f18c01358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59336/0 (socket says 172.21.15.63:59336) 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c4863640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f18c00a3b10 0x7f18c01d4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35336/0 (socket says 172.21.15.63:35336) 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.510+0000 7f18c5064640 1 -- 172.21.15.63:0/3256340695 learned_addr learned my addr 172.21.15.63:0/3256340695 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 268036695 0 0) 0x7f18c0129370 con 0x7f18c00a07a0 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:37.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): my global_id is 5951 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18a4003530 con 0x7f18c00a07a0 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1085152865 0 0) 0x7f18c012a000 con 0x7f18c00a3b10 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): my global_id is 5976 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18c0129370 con 0x7f18c00a3b10 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2775882853 0 0) 0x7f18c0129370 con 0x7f18c00a3b10 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18c012a000 con 0x7f18c00a3b10 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 49952787 0 0) 0x7f18a4003530 con 0x7f18c00a07a0 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f18c0129370 con 0x7f18c00a07a0 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18b4001500 con 0x7f18c00a3b10 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:37.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): dump: 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:37.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18bc001a40 con 0x7f18c00a07a0 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient(hunting): dump: 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:37.533 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3299950821 0 0) 0x7f18c012a000 con 0x7f18c00a3b10 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 >> v1:172.21.15.6:6790/0 conn(0x7f18c00a07a0 legacy=0x7f18c01358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 >> v1:172.21.15.63:6789/0 conn(0x7f18c00a4600 legacy=0x7f18c01d8080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 monclient: found mon.a 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c0135fc0 con 0x7f18c00a3b10 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:37.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 5 monclient: authenticate success, global_id 5976 2024-09-16T05:39:37.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 10 monclient: _renew_subs 2024-09-16T05:39:37.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:37.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18c0128b50 con 0x7f18c00a3b10 2024-09-16T05:39:37.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 10 monclient: _renew_subs 2024-09-16T05:39:37.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:37.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f18c01cf7a0 con 0x7f18c00a3b10 2024-09-16T05:39:37.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f18b4001790 con 0x7f18c00a3b10 2024-09-16T05:39:37.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:37.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b37fe640 4 set_mon_vals no callback set 2024-09-16T05:39:37.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f18b4004ce0 con 0x7f18c00a3b10 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 10 monclient: dump: 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:37.537 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:37.538 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:37.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:37.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:37.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:37.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a0005090 con 0x7f18c00a3b10 2024-09-16T05:39:37.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f18b400fa80 con 0x7f18c00a3b10 2024-09-16T05:39:37.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(185..185 src has 1..185) ==== 6761+0+0 (unknown 1613038251 0 0) 0x7f18b4044c70 con 0x7f18c00a3b10 2024-09-16T05:39:37.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.514+0000 7f18c5064640 10 monclient: get_auth_request con 0x7f18a40369e0 auth_method 0 2024-09-16T05:39:37.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.530+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f18b4015280 con 0x7f18c00a3b10 2024-09-16T05:39:37.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.530+0000 7f18b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:37.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.530+0000 7f18b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:37.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.698+0000 7f18c72c8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T05:39:37.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.698+0000 7f18c72c8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T05:39:37.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.698+0000 7f18c72c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:37.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:37.698+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) -- 0x7f18a0005380 con 0x7f18c00a3b10 2024-09-16T05:39:38.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.270+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 added bucket r1 type rack to location {root=default} v186) ==== 176+0+0 (unknown 53346236 0 0) 0x7f18b4018ed0 con 0x7f18c00a3b10 2024-09-16T05:39:38.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.270+0000 7f18b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T05:39:38.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.270+0000 7f18b1ffb640 10 monclient: _finish_command 2 = system:0 added bucket r1 type rack to location {root=default} 2024-09-16T05:39:38.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.374+0000 7f18c72c8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T05:39:38.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.374+0000 7f18c72c8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T05:39:38.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.374+0000 7f18c72c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:38.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.374+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) -- 0x7f18a0005c50 con 0x7f18c00a3b10 2024-09-16T05:39:38.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.374+0000 7f18b1ffb640 1 -- 172.21.15.63:0/3256340695 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 bucket 'r1' already exists v186) ==== 150+0+0 (unknown 3777832355 0 0) 0x7f18b4041df0 con 0x7f18c00a3b10 2024-09-16T05:39:38.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.374+0000 7f18b1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T05:39:38.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.374+0000 7f18b1ffb640 10 monclient: _finish_command 3 = system:0 bucket 'r1' already exists 2024-09-16T05:39:38.380 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'r1' already exists 2024-09-16T05:39:38.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.378+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 >> v1:172.21.15.63:6812/3643207211 conn(0x7f18a40369e0 legacy=0x7f18a4038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:38.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.378+0000 7f18c72c8640 10 monclient: shutdown 2024-09-16T05:39:38.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.378+0000 7f18c72c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:38.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.378+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 >> v1:172.21.15.6:6789/0 conn(0x7f18c00a3b10 legacy=0x7f18c01d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:38.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.378+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 shutdown_connections 2024-09-16T05:39:38.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.378+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 >> 172.21.15.63:0/3256340695 conn(0x7f18c0075470 msgr2=0x7f18c00a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:38.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.378+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 shutdown_connections 2024-09-16T05:39:38.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.378+0000 7f18c72c8640 1 -- 172.21.15.63:0/3256340695 wait complete. 2024-09-16T05:39:38.398 INFO:tasks.workunit.client.0.smithi063.stderr:++ seq 1 32 2024-09-16T05:39:38.412 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:38.412 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h1 host rack=r1 2024-09-16T05:39:38.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient: get_monmap_and_config 2024-09-16T05:39:38.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient: build_initial_monmap 2024-09-16T05:39:38.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient: monmap: 2024-09-16T05:39:38.529 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:38.529 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:38.529 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:38.529243+0000 2024-09-16T05:39:38.529 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:38.529243+0000 2024-09-16T05:39:38.530 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:38.530 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:38.530 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:38.530 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:38.530 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:38.530 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:38.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 1 Processor -- start 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 1 -- start start 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient: init 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient(hunting): picked mon.noname-c con 0x7fd42409c300 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient(hunting): picked mon.noname-a con 0x7fd4240a09c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient(hunting): picked mon.noname-b con 0x7fd42409d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4240a0ed0 con 0x7fd4240a09c0 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42409d950 con 0x7fd42409c300 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42409ebd0 con 0x7fd42409d3a0 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 10 monclient(hunting): authenticate will time out at 1235.306360s 2024-09-16T05:39:38.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd422ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd42409c300 legacy=0x7fd42409c6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd4237fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd42409d3a0 0x7fd42413c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59350/0 (socket says 172.21.15.63:59350) 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd4237fe640 1 -- 172.21.15.63:0/4288390800 learned_addr learned my addr 172.21.15.63:0/4288390800 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd4227fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd4240a09c0 0x7fd424138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35346/0 (socket says 172.21.15.63:35346) 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 633082798 0 0) 0x7fd42409ebd0 con 0x7fd42409d3a0 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient(hunting): my global_id is 5954 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd40c003530 con 0x7fd42409d3a0 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156963843 0 0) 0x7fd4240a0ed0 con 0x7fd4240a09c0 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:38.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient(hunting): my global_id is 5979 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd42409ebd0 con 0x7fd4240a09c0 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3497075061 0 0) 0x7fd40c003530 con 0x7fd42409d3a0 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 >> v1:172.21.15.63:6789/0 conn(0x7fd42409c300 legacy=0x7fd42409c6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 >> v1:172.21.15.6:6789/0 conn(0x7fd4240a09c0 legacy=0x7fd424138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 monclient: found mon.noname-b 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4240a2470 con 0x7fd42409d3a0 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:38.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd4140019a0 con 0x7fd42409d3a0 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 5 monclient: authenticate success, global_id 5954 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd421ffb640 10 monclient: dump: 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:38.537 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:38.538 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:38.538 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:38.538 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:38.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.526+0000 7fd429325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:38.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd414001c00 con 0x7fd42409d3a0 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd421ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd421ffb640 1 -- 172.21.15.63:0/4288390800 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd414001fd0 con 0x7fd42409d3a0 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd421ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd421ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd421ffb640 10 monclient: dump: 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.539 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 4 set_mon_vals no callback set 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient: shutdown 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- 172.21.15.63:0/4288390800 >> v1:172.21.15.6:6790/0 conn(0x7fd42409d3a0 legacy=0x7fd42413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- 172.21.15.63:0/4288390800 shutdown_connections 2024-09-16T05:39:38.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- 172.21.15.63:0/4288390800 >> 172.21.15.63:0/4288390800 conn(0x7fd4240a1e90 msgr2=0x7fd424075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- 172.21.15.63:0/4288390800 shutdown_connections 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- 172.21.15.63:0/4288390800 wait complete. 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient: build_initial_monmap 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient: monmap: 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:38.541 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:38.542 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:38.542 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:38.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 Processor -- start 2024-09-16T05:39:38.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- start start 2024-09-16T05:39:38.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient: init 2024-09-16T05:39:38.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient(hunting): picked mon.noname-a con 0x7fd42409c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient(hunting): picked mon.noname-b con 0x7fd42409d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient(hunting): picked mon.noname-c con 0x7fd42406e8e0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4240a4560 con 0x7fd42409c300 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42409c0f0 con 0x7fd42409d3a0 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4241274c0 con 0x7fd42406e8e0 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd429325640 10 monclient(hunting): authenticate will time out at 1235.310301s 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd4227fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd42409d3a0 legacy=0x7fd42406e1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd4237fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd42406e8e0 0x7fd4241d3880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59354/0 (socket says 172.21.15.63:59354) 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd4237fe640 1 -- 172.21.15.63:0/1713985011 learned_addr learned my addr 172.21.15.63:0/1713985011 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:38.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 988114677 0 0) 0x7fd4241274c0 con 0x7fd42406e8e0 2024-09-16T05:39:38.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:38.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:38.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:38.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): my global_id is 5957 2024-09-16T05:39:38.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd410003530 con 0x7fd42406e8e0 2024-09-16T05:39:38.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942210707 0 0) 0x7fd4240a4560 con 0x7fd42409c300 2024-09-16T05:39:38.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:38.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): my global_id is 5982 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4241274c0 con 0x7fd42409c300 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3866629923 0 0) 0x7fd410003530 con 0x7fd42406e8e0 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4240a4560 con 0x7fd42406e8e0 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd4140018a0 con 0x7fd42406e8e0 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): dump: 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:38.551 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4094464366 0 0) 0x7fd4241274c0 con 0x7fd42409c300 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd410003530 con 0x7fd42409c300 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd41c0040d0 con 0x7fd42409c300 2024-09-16T05:39:38.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.530+0000 7fd40b7fe640 10 monclient(hunting): dump: 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:38.553 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3009808454 0 0) 0x7fd4240a4560 con 0x7fd42406e8e0 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 >> v1:172.21.15.63:6789/0 conn(0x7fd42409d3a0 legacy=0x7fd42406e1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 >> v1:172.21.15.6:6789/0 conn(0x7fd42409c300 legacy=0x7fd42406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 monclient: found mon.c 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4241277a0 con 0x7fd42406e8e0 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:38.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 5 monclient: authenticate success, global_id 5957 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 10 monclient: _renew_subs 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4241262e0 con 0x7fd42406e8e0 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 10 monclient: _renew_subs 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd4241d4fa0 con 0x7fd42406e8e0 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd414001c00 con 0x7fd42406e8e0 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd414005120 con 0x7fd42406e8e0 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:38.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 10 monclient: dump: 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd4217fa640 4 set_mon_vals no callback set 2024-09-16T05:39:38.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:38.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:38.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:38.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd404005090 con 0x7fd42406e8e0 2024-09-16T05:39:38.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd4140052c0 con 0x7fd42406e8e0 2024-09-16T05:39:38.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(186..186 src has 1..186) ==== 6971+0+0 (unknown 2482326172 0 0) 0x7fd414045230 con 0x7fd42406e8e0 2024-09-16T05:39:38.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.534+0000 7fd422ffd640 10 monclient: get_auth_request con 0x7fd410036a70 auth_method 0 2024-09-16T05:39:38.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.546+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd4140193c0 con 0x7fd42406e8e0 2024-09-16T05:39:38.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.546+0000 7fd40b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:38.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.546+0000 7fd40b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:38.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.714+0000 7fd429325640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:38.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.714+0000 7fd429325640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:38.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.714+0000 7fd429325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:38.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:38.714+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fd404005380 con 0x7fd42406e8e0 2024-09-16T05:39:39.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.278+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 added bucket h1 type host to location {rack=r1} v187) ==== 166+0+0 (unknown 1769572012 0 0) 0x7fd41401dc80 con 0x7fd42406e8e0 2024-09-16T05:39:39.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.278+0000 7fd40b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:39.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.278+0000 7fd40b7fe640 10 monclient: _finish_command 2 = system:0 added bucket h1 type host to location {rack=r1} 2024-09-16T05:39:39.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.382+0000 7fd429325640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:39.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.382+0000 7fd429325640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:39.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.382+0000 7fd429325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:39.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.382+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fd404005ac0 con 0x7fd42406e8e0 2024-09-16T05:39:39.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.382+0000 7fd40b7fe640 1 -- 172.21.15.63:0/1713985011 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v187) ==== 145+0+0 (unknown 3528628542 0 0) 0x7fd4140423e0 con 0x7fd42406e8e0 2024-09-16T05:39:39.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.382+0000 7fd40b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:39.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.382+0000 7fd40b7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h1' already exists 2024-09-16T05:39:39.387 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h1' already exists 2024-09-16T05:39:39.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.386+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd410036a70 legacy=0x7fd410038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:39.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.386+0000 7fd429325640 10 monclient: shutdown 2024-09-16T05:39:39.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.386+0000 7fd429325640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:39.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.386+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 >> v1:172.21.15.6:6790/0 conn(0x7fd42406e8e0 legacy=0x7fd4241d3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:39.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.386+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 shutdown_connections 2024-09-16T05:39:39.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.386+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 >> 172.21.15.63:0/1713985011 conn(0x7fd4240a1e90 msgr2=0x7fd42411b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:39.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.386+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 shutdown_connections 2024-09-16T05:39:39.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.386+0000 7fd429325640 1 -- 172.21.15.63:0/1713985011 wait complete. 2024-09-16T05:39:39.404 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:39.404 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h2 host rack=r1 2024-09-16T05:39:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient: get_monmap_and_config 2024-09-16T05:39:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient: build_initial_monmap 2024-09-16T05:39:39.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient: monmap: 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:39.528556+0000 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:39.528556+0000 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:39.529 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 1 Processor -- start 2024-09-16T05:39:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 1 -- start start 2024-09-16T05:39:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient: init 2024-09-16T05:39:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:39.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient(hunting): picked mon.noname-a con 0x7f217809c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient(hunting): picked mon.noname-c con 0x7f21780a09c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient(hunting): picked mon.noname-b con 0x7f217809d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21780a0ed0 con 0x7f217809c300 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f217809d950 con 0x7f21780a09c0 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f217809ebd0 con 0x7f217809d3a0 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient(hunting): authenticate will time out at 1236.305689s 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f21767fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f21780a09c0 legacy=0x7f2178138a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:39.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f21777fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f217809d3a0 0x7f217813c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59368/0 (socket says 172.21.15.63:59368) 2024-09-16T05:39:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2176ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f217809c300 0x7f217809c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35362/0 (socket says 172.21.15.63:35362) 2024-09-16T05:39:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f21777fe640 1 -- 172.21.15.63:0/1283702294 learned_addr learned my addr 172.21.15.63:0/1283702294 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3205880665 0 0) 0x7f217809ebd0 con 0x7f217809d3a0 2024-09-16T05:39:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient(hunting): my global_id is 5960 2024-09-16T05:39:39.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2164003530 con 0x7f217809d3a0 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2561694012 0 0) 0x7f21780a0ed0 con 0x7f217809c300 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient(hunting): my global_id is 5985 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f217809ebd0 con 0x7f217809c300 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1202601874 0 0) 0x7f2164003530 con 0x7f217809d3a0 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 >> v1:172.21.15.63:6789/0 conn(0x7f21780a09c0 legacy=0x7f2178138a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 >> v1:172.21.15.6:6789/0 conn(0x7f217809c300 legacy=0x7f217809c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 monclient: found mon.noname-b 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:39.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21780a2470 con 0x7f217809d3a0 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f21700019a0 con 0x7f217809d3a0 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: dump: 2024-09-16T05:39:39.537 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 5 monclient: authenticate success, global_id 5960 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2170001c00 con 0x7f217809d3a0 2024-09-16T05:39:39.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 1 -- 172.21.15.63:0/1283702294 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2170001fd0 con 0x7f217809d3a0 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f2175ffb640 10 monclient: dump: 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:39.539 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:39.540 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:39.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:39.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 4 set_mon_vals no callback set 2024-09-16T05:39:39.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 10 monclient: shutdown 2024-09-16T05:39:39.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:39.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 1 -- 172.21.15.63:0/1283702294 >> v1:172.21.15.6:6790/0 conn(0x7f217809d3a0 legacy=0x7f217813c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:39.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 1 -- 172.21.15.63:0/1283702294 shutdown_connections 2024-09-16T05:39:39.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.526+0000 7f217d9a6640 1 -- 172.21.15.63:0/1283702294 >> 172.21.15.63:0/1283702294 conn(0x7f21780a1e90 msgr2=0x7f2178075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:39.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 -- 172.21.15.63:0/1283702294 shutdown_connections 2024-09-16T05:39:39.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 -- 172.21.15.63:0/1283702294 wait complete. 2024-09-16T05:39:39.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: build_initial_monmap 2024-09-16T05:39:39.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: monmap: 2024-09-16T05:39:39.541 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:39.541 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:39.541 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:39.542 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:39.542 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:39.542 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:39.542 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:39.542 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:39.542 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:39.543 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:39.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 Processor -- start 2024-09-16T05:39:39.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 -- start start 2024-09-16T05:39:39.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: init 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient(hunting): picked mon.noname-c con 0x7f217809c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient(hunting): picked mon.noname-b con 0x7f217809d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient(hunting): picked mon.noname-a con 0x7f21781334d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21780a4320 con 0x7f21781334d0 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2178133940 con 0x7f217809d3a0 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f21767fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f217809d3a0 legacy=0x7f21781d0440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f217809c0f0 con 0x7f217809c300 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:39.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient(hunting): authenticate will time out at 1236.309940s 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f2176ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f217809c300 0x7f2178132dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59376/0 (socket says 172.21.15.63:59376) 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f2176ffd640 1 -- 172.21.15.63:0/271213103 learned_addr learned my addr 172.21.15.63:0/271213103 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221197232 0 0) 0x7f217809c0f0 con 0x7f217809c300 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): my global_id is 5963 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2158003530 con 0x7f217809c300 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 976090313 0 0) 0x7f21780a4320 con 0x7f21781334d0 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:39.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): my global_id is 5988 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f217809c0f0 con 0x7f21781334d0 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2190295502 0 0) 0x7f2158003530 con 0x7f217809c300 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21780a4320 con 0x7f217809c300 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f21600018d0 con 0x7f217809c300 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): dump: 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.550 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 408640118 0 0) 0x7f217809c0f0 con 0x7f21781334d0 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2158003530 con 0x7f21781334d0 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2170001db0 con 0x7f21781334d0 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:39.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient(hunting): dump: 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2001737510 0 0) 0x7f21780a4320 con 0x7f217809c300 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 >> v1:172.21.15.63:6789/0 conn(0x7f217809d3a0 legacy=0x7f21781d0440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:39.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 >> v1:172.21.15.6:6789/0 conn(0x7f21781334d0 legacy=0x7f21781d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 monclient: found mon.c 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2178071710 con 0x7f217809c300 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient: _finish_auth 0 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f216ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 5 monclient: authenticate success, global_id 5963 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: _renew_subs 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2178120780 con 0x7f217809c300 2024-09-16T05:39:39.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: _renew_subs 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.530+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f21781204a0 con 0x7f217809c300 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21600047f0 con 0x7f217809c300 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f216ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2160004b40 con 0x7f217809c300 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f216ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f216ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f2174ff9640 4 set_mon_vals no callback set 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f216ffff640 10 monclient: dump: 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:39.554 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f217d9a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f217d9a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:39.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f217d9a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:39.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f215c005090 con 0x7f217809c300 2024-09-16T05:39:39.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f216000f8e0 con 0x7f217809c300 2024-09-16T05:39:39.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(187..187 src has 1..187) ==== 7181+0+0 (unknown 4099791575 0 0) 0x7f2160044fa0 con 0x7f217809c300 2024-09-16T05:39:39.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.534+0000 7f21767fc640 10 monclient: get_auth_request con 0x7f2158036a70 auth_method 0 2024-09-16T05:39:39.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.546+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2160015410 con 0x7f217809c300 2024-09-16T05:39:39.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.546+0000 7f216ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:39.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.546+0000 7f216ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:39.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.710+0000 7f217d9a6640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:39.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.710+0000 7f217d9a6640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:39.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.710+0000 7f217d9a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:39.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:39.710+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f215c005380 con 0x7f217809c300 2024-09-16T05:39:40.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.294+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 added bucket h2 type host to location {rack=r1} v188) ==== 166+0+0 (unknown 3659760870 0 0) 0x7f2160014d10 con 0x7f217809c300 2024-09-16T05:39:40.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.294+0000 7f216ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:40.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.294+0000 7f216ffff640 10 monclient: _finish_command 2 = system:0 added bucket h2 type host to location {rack=r1} 2024-09-16T05:39:40.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.398+0000 7f217d9a6640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:40.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.398+0000 7f217d9a6640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:40.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.398+0000 7f217d9a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:40.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.398+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f215c005ac0 con 0x7f217809c300 2024-09-16T05:39:40.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f216ffff640 1 -- 172.21.15.63:0/271213103 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h2' already exists v188) ==== 145+0+0 (unknown 821990746 0 0) 0x7f216001d920 con 0x7f217809c300 2024-09-16T05:39:40.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f216ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:40.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f216ffff640 10 monclient: _finish_command 3 = system:0 bucket 'h2' already exists 2024-09-16T05:39:40.405 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h2' already exists 2024-09-16T05:39:40.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2158036a70 legacy=0x7f2158038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:40.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f217d9a6640 10 monclient: shutdown 2024-09-16T05:39:40.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f217d9a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:40.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 >> v1:172.21.15.6:6790/0 conn(0x7f217809c300 legacy=0x7f2178132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:40.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 shutdown_connections 2024-09-16T05:39:40.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 >> 172.21.15.63:0/271213103 conn(0x7f21780a1e90 msgr2=0x7f21781be940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:40.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 shutdown_connections 2024-09-16T05:39:40.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.402+0000 7f217d9a6640 1 -- 172.21.15.63:0/271213103 wait complete. 2024-09-16T05:39:40.422 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:40.422 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h3 host rack=r1 2024-09-16T05:39:40.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.542+0000 7f11d13b3640 10 monclient: get_monmap_and_config 2024-09-16T05:39:40.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.542+0000 7f11d13b3640 10 monclient: build_initial_monmap 2024-09-16T05:39:40.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.542+0000 7f11d13b3640 10 monclient: monmap: 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:40.546633+0000 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:40.546633+0000 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:40.547 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:40.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.542+0000 7f11d13b3640 1 Processor -- start 2024-09-16T05:39:40.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.542+0000 7f11d13b3640 1 -- start start 2024-09-16T05:39:40.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.542+0000 7f11d13b3640 10 monclient: init 2024-09-16T05:39:40.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:40.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient(hunting): picked mon.noname-a con 0x7f11cc09e9c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient(hunting): picked mon.noname-c con 0x7f11cc09eda0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient(hunting): picked mon.noname-b con 0x7f11cc09dcf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11cc09f1b0 con 0x7f11cc09e9c0 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11cc0a1ce0 con 0x7f11cc09eda0 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11cc09ca50 con 0x7f11cc09dcf0 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient(hunting): authenticate will time out at 1237.324588s 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11ca7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f11cc09eda0 legacy=0x7f11cc0a15d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11caffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f11cc09e9c0 0x7f11cc0a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35372/0 (socket says 172.21.15.63:35372) 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11cb7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f11cc09dcf0 0x7f11cc13d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59392/0 (socket says 172.21.15.63:59392) 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11caffd640 1 -- 172.21.15.63:0/3493780773 learned_addr learned my addr 172.21.15.63:0/3493780773 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:40.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473003899 0 0) 0x7f11cc09f1b0 con 0x7f11cc09e9c0 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient(hunting): my global_id is 5991 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11b4003530 con 0x7f11cc09e9c0 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 996779632 0 0) 0x7f11cc09ca50 con 0x7f11cc09dcf0 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient(hunting): my global_id is 5966 2024-09-16T05:39:40.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11cc09f1b0 con 0x7f11cc09dcf0 2024-09-16T05:39:40.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2273132709 0 0) 0x7f11b4003530 con 0x7f11cc09e9c0 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 >> v1:172.21.15.6:6790/0 conn(0x7f11cc09dcf0 legacy=0x7f11cc13d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 >> v1:172.21.15.63:6789/0 conn(0x7f11cc09eda0 legacy=0x7f11cc0a15d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 monclient: found mon.noname-a 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11cc09d010 con 0x7f11cc09e9c0 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11bc001550 con 0x7f11cc09e9c0 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:40.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: dump: 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 5 monclient: authenticate success, global_id 5991 2024-09-16T05:39:40.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11bc001870 con 0x7f11cc09e9c0 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 1 -- 172.21.15.63:0/3493780773 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11bc001c40 con 0x7f11cc09e9c0 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 4 set_mon_vals no callback set 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11c9ffb640 10 monclient: dump: 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.559 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient: shutdown 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- 172.21.15.63:0/3493780773 >> v1:172.21.15.6:6789/0 conn(0x7f11cc09e9c0 legacy=0x7f11cc0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- 172.21.15.63:0/3493780773 shutdown_connections 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- 172.21.15.63:0/3493780773 >> 172.21.15.63:0/3493780773 conn(0x7f11cc0a2180 msgr2=0x7f11cc0a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- 172.21.15.63:0/3493780773 shutdown_connections 2024-09-16T05:39:40.560 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- 172.21.15.63:0/3493780773 wait complete. 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient: build_initial_monmap 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient: monmap: 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 Processor -- start 2024-09-16T05:39:40.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 1 -- start start 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.546+0000 7f11d13b3640 10 monclient: init 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient(hunting): picked mon.noname-b con 0x7f11cc09dcf0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient(hunting): picked mon.noname-c con 0x7f11cc09e9c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient(hunting): picked mon.noname-a con 0x7f11cc09eda0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11cc09c6a0 con 0x7f11cc09eda0 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11cc132610 con 0x7f11cc09dcf0 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11caffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f11cc09dcf0 legacy=0x7f11cc1c9760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11cc1328f0 con 0x7f11cc09e9c0 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient(hunting): authenticate will time out at 1237.328317s 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11cb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f11cc09eda0 0x7f11cc1d1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35376/0 (socket says 172.21.15.63:35376) 2024-09-16T05:39:40.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11ca7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f11cc09e9c0 0x7f11cc1ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59408/0 (socket says 172.21.15.63:59408) 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11cb7fe640 1 -- 172.21.15.63:0/769571554 learned_addr learned my addr 172.21.15.63:0/769571554 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2526249492 0 0) 0x7f11cc09c6a0 con 0x7f11cc09eda0 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): my global_id is 5994 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11b8003530 con 0x7f11cc09eda0 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4037940785 0 0) 0x7f11cc1328f0 con 0x7f11cc09e9c0 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): my global_id is 5969 2024-09-16T05:39:40.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11cc09c6a0 con 0x7f11cc09e9c0 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 680343312 0 0) 0x7f11b8003530 con 0x7f11cc09eda0 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11cc1328f0 con 0x7f11cc09eda0 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11c4001b00 con 0x7f11cc09eda0 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): dump: 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:40.564 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4075758962 0 0) 0x7f11cc09c6a0 con 0x7f11cc09e9c0 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11b8003530 con 0x7f11cc09e9c0 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11c00040d0 con 0x7f11cc09e9c0 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient(hunting): dump: 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:40.565 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1867974786 0 0) 0x7f11cc1328f0 con 0x7f11cc09eda0 2024-09-16T05:39:40.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 >> v1:172.21.15.6:6790/0 conn(0x7f11cc09e9c0 legacy=0x7f11cc1ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 >> v1:172.21.15.63:6789/0 conn(0x7f11cc09dcf0 legacy=0x7f11cc1c9760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 monclient: found mon.a 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11cc130c80 con 0x7f11cc09eda0 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 5 monclient: authenticate success, global_id 5994 2024-09-16T05:39:40.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient: _renew_subs 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f11cc1310a0 con 0x7f11cc09eda0 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient: _renew_subs 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f11cc130310 con 0x7f11cc09eda0 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11c4004b70 con 0x7f11cc09eda0 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11c4004d20 con 0x7f11cc09eda0 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11c97fa640 4 set_mon_vals no callback set 2024-09-16T05:39:40.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 10 monclient: dump: 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:40.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ac005090 con 0x7f11cc09eda0 2024-09-16T05:39:40.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f11c400fac0 con 0x7f11cc09eda0 2024-09-16T05:39:40.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(188..188 src has 1..188) ==== 7391+0+0 (unknown 805530518 0 0) 0x7f11c4045130 con 0x7f11cc09eda0 2024-09-16T05:39:40.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.550+0000 7f11caffd640 10 monclient: get_auth_request con 0x7f11b8036af0 auth_method 0 2024-09-16T05:39:40.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.562+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f11c4015530 con 0x7f11cc09eda0 2024-09-16T05:39:40.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.562+0000 7f11b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:40.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.562+0000 7f11b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:40.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.722+0000 7f11d13b3640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:40.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.722+0000 7f11d13b3640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:40.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.722+0000 7f11d13b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:40.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:40.722+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f11ac005380 con 0x7f11cc09eda0 2024-09-16T05:39:41.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.038+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 added bucket h3 type host to location {rack=r1} v189) ==== 166+0+0 (unknown 1085621746 0 0) 0x7f11c4014e30 con 0x7f11cc09eda0 2024-09-16T05:39:41.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.038+0000 7f11b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:41.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.038+0000 7f11b37fe640 10 monclient: _finish_command 2 = system:0 added bucket h3 type host to location {rack=r1} 2024-09-16T05:39:41.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:41.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:41.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:41.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f11ac005ac0 con 0x7f11cc09eda0 2024-09-16T05:39:41.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11b37fe640 1 -- 172.21.15.63:0/769571554 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h3' already exists v189) ==== 145+0+0 (unknown 1906824499 0 0) 0x7f11c401d9e0 con 0x7f11cc09eda0 2024-09-16T05:39:41.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11b37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:41.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11b37fe640 10 monclient: _finish_command 3 = system:0 bucket 'h3' already exists 2024-09-16T05:39:41.142 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h3' already exists 2024-09-16T05:39:41.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 >> v1:172.21.15.63:6812/3643207211 conn(0x7f11b8036af0 legacy=0x7f11b8038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:41.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 10 monclient: shutdown 2024-09-16T05:39:41.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:41.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 >> v1:172.21.15.6:6789/0 conn(0x7f11cc09eda0 legacy=0x7f11cc1d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:41.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 shutdown_connections 2024-09-16T05:39:41.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 >> 172.21.15.63:0/769571554 conn(0x7f11cc0a2180 msgr2=0x7f11cc09fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:41.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 shutdown_connections 2024-09-16T05:39:41.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.138+0000 7f11d13b3640 1 -- 172.21.15.63:0/769571554 wait complete. 2024-09-16T05:39:41.157 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:41.158 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h4 host rack=r1 2024-09-16T05:39:41.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient: get_monmap_and_config 2024-09-16T05:39:41.280 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient: build_initial_monmap 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient: monmap: 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:41.280744+0000 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:41.280744+0000 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:41.281 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:41.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 Processor -- start 2024-09-16T05:39:41.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- start start 2024-09-16T05:39:41.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient: init 2024-09-16T05:39:41.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:41.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:41.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient(hunting): picked mon.noname-b con 0x7f9ae809bac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:41.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient(hunting): picked mon.noname-c con 0x7f9ae809bea0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:41.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae813d2c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae80a12e0 con 0x7f9ae813d2c0 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae809c280 con 0x7f9ae809bea0 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae8073090 con 0x7f9ae809bac0 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient(hunting): authenticate will time out at 1238.057775s 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae67fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9ae809bea0 legacy=0x7f9ae813bba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae6ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9ae809bac0 0x7f9ae8134c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59420/0 (socket says 172.21.15.63:59420) 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9ae813d2c0 0x7f9ae813f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35390/0 (socket says 172.21.15.63:35390) 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae6ffd640 1 -- 172.21.15.63:0/3924634194 learned_addr learned my addr 172.21.15.63:0/3924634194 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736209034 0 0) 0x7f9ae80a12e0 con 0x7f9ae813d2c0 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:41.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient(hunting): my global_id is 5997 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ad0003530 con 0x7f9ae813d2c0 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3271290401 0 0) 0x7f9ae8073090 con 0x7f9ae809bac0 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient(hunting): my global_id is 5972 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ae80a12e0 con 0x7f9ae809bac0 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2833438543 0 0) 0x7f9ad0003530 con 0x7f9ae813d2c0 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 >> v1:172.21.15.6:6790/0 conn(0x7f9ae809bac0 legacy=0x7f9ae8134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 >> v1:172.21.15.63:6789/0 conn(0x7f9ae809bea0 legacy=0x7f9ae813bba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:41.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 monclient: found mon.noname-a 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae8075770 con 0x7f9ae813d2c0 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ad8001620 con 0x7f9ae813d2c0 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: dump: 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:41.288 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 5 monclient: authenticate success, global_id 5997 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ad8001940 con 0x7f9ae813d2c0 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 1 -- 172.21.15.63:0/3924634194 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ad8001d10 con 0x7f9ae813d2c0 2024-09-16T05:39:41.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 4 set_mon_vals no callback set 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9ae5ffb640 10 monclient: dump: 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:41.290 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 10 monclient: shutdown 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- 172.21.15.63:0/3924634194 >> v1:172.21.15.6:6789/0 conn(0x7f9ae813d2c0 legacy=0x7f9ae813f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- 172.21.15.63:0/3924634194 shutdown_connections 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- 172.21.15.63:0/3924634194 >> 172.21.15.63:0/3924634194 conn(0x7f9ae80a2420 msgr2=0x7f9ae80a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- 172.21.15.63:0/3924634194 shutdown_connections 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.278+0000 7f9aed954640 1 -- 172.21.15.63:0/3924634194 wait complete. 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: build_initial_monmap 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: monmap: 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:41.291 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 1 Processor -- start 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 1 -- start start 2024-09-16T05:39:41.292 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: init 2024-09-16T05:39:41.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:41.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:41.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient(hunting): picked mon.noname-b con 0x7f9ae809bac0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:41.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient(hunting): picked mon.noname-c con 0x7f9ae809bea0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:41.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae813d2c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae8137cb0 con 0x7f9ae813d2c0 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae813a9d0 con 0x7f9ae809bac0 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae8139060 con 0x7f9ae809bea0 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient(hunting): authenticate will time out at 1238.061370s 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9ae6ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9ae809bac0 legacy=0x7f9ae8136e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9ae67fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9ae809bea0 0x7f9ae81375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59436/0 (socket says 172.21.15.63:59436) 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9ae77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9ae813d2c0 0x7f9ae81d3e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35394/0 (socket says 172.21.15.63:35394) 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9ae67fc640 1 -- 172.21.15.63:0/3286242232 learned_addr learned my addr 172.21.15.63:0/3286242232 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1905112379 0 0) 0x7f9ae8139060 con 0x7f9ae809bea0 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:41.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): my global_id is 5975 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ad4003530 con 0x7f9ae809bea0 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242177418 0 0) 0x7f9ae8137cb0 con 0x7f9ae813d2c0 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): my global_id is 6000 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ae8139060 con 0x7f9ae813d2c0 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2033631569 0 0) 0x7f9ad4003530 con 0x7f9ae809bea0 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ae8137cb0 con 0x7f9ae809bea0 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1360251200 0 0) 0x7f9ae8139060 con 0x7f9ae813d2c0 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ad4003530 con 0x7f9ae813d2c0 2024-09-16T05:39:41.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9adc0044e0 con 0x7f9ae809bea0 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): dump: 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:41.298 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ad8001440 con 0x7f9ae813d2c0 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient(hunting): dump: 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:41.299 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2358420394 0 0) 0x7f9ad4003530 con 0x7f9ae813d2c0 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 >> v1:172.21.15.6:6790/0 conn(0x7f9ae809bea0 legacy=0x7f9ae81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 >> v1:172.21.15.63:6789/0 conn(0x7f9ae809bac0 legacy=0x7f9ae8136e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 monclient: found mon.a 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae813cad0 con 0x7f9ae813d2c0 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient: _finish_auth 0 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9acffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:41.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 5 monclient: authenticate success, global_id 6000 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: _renew_subs 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae8140b60 con 0x7f9ae813d2c0 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: _renew_subs 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.282+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ae813a6f0 con 0x7f9ae813d2c0 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ad8001940 con 0x7f9ae813d2c0 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9acffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9ad8004d60 con 0x7f9ae813d2c0 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9acffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9ae4ff9640 4 set_mon_vals no callback set 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9acffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:41.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9acffff640 10 monclient: dump: 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9aed954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9aed954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9aed954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:41.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac8005090 con 0x7f9ae813d2c0 2024-09-16T05:39:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9ad800fb00 con 0x7f9ae813d2c0 2024-09-16T05:39:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(189..189 src has 1..189) ==== 7601+0+0 (unknown 3192286987 0 0) 0x7f9ad8044f20 con 0x7f9ae813d2c0 2024-09-16T05:39:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.286+0000 7f9ae6ffd640 10 monclient: get_auth_request con 0x7f9ad40369a0 auth_method 0 2024-09-16T05:39:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.298+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9ad80152a0 con 0x7f9ae813d2c0 2024-09-16T05:39:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.298+0000 7f9acffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:41.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.298+0000 7f9acffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:41.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.466+0000 7f9aed954640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:41.471 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.466+0000 7f9aed954640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:41.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.466+0000 7f9aed954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:41.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:41.466+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f9ac8005380 con 0x7f9ae813d2c0 2024-09-16T05:39:42.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.314+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 added bucket h4 type host to location {rack=r1} v190) ==== 166+0+0 (unknown 2456595978 0 0) 0x7f9ad8014ba0 con 0x7f9ae813d2c0 2024-09-16T05:39:42.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.314+0000 7f9acffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:42.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.314+0000 7f9acffff640 10 monclient: _finish_command 2 = system:0 added bucket h4 type host to location {rack=r1} 2024-09-16T05:39:42.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.414+0000 7f9aed954640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:42.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.414+0000 7f9aed954640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:42.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.414+0000 7f9aed954640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:42.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.414+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f9ac8005ac0 con 0x7f9ae813d2c0 2024-09-16T05:39:42.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.414+0000 7f9acffff640 1 -- 172.21.15.63:0/3286242232 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h4' already exists v190) ==== 145+0+0 (unknown 2474814978 0 0) 0x7f9ad8014560 con 0x7f9ae813d2c0 2024-09-16T05:39:42.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.414+0000 7f9acffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:42.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.414+0000 7f9acffff640 10 monclient: _finish_command 3 = system:0 bucket 'h4' already exists 2024-09-16T05:39:42.419 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h4' already exists 2024-09-16T05:39:42.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.418+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9ad40369a0 legacy=0x7f9ad4038e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:42.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.418+0000 7f9aed954640 10 monclient: shutdown 2024-09-16T05:39:42.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.418+0000 7f9aed954640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:42.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.418+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 >> v1:172.21.15.6:6789/0 conn(0x7f9ae813d2c0 legacy=0x7f9ae81d3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:42.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.418+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 shutdown_connections 2024-09-16T05:39:42.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.418+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 >> 172.21.15.63:0/3286242232 conn(0x7f9ae80a2420 msgr2=0x7f9ae80a48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:42.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.418+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 shutdown_connections 2024-09-16T05:39:42.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.418+0000 7f9aed954640 1 -- 172.21.15.63:0/3286242232 wait complete. 2024-09-16T05:39:42.436 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:42.437 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h5 host rack=r1 2024-09-16T05:39:42.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient: get_monmap_and_config 2024-09-16T05:39:42.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient: build_initial_monmap 2024-09-16T05:39:42.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient: monmap: 2024-09-16T05:39:42.561 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:42.561 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:42.562 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:42.561388+0000 2024-09-16T05:39:42.562 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:42.561388+0000 2024-09-16T05:39:42.562 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:42.562 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:42.562 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:42.562 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:42.562 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:42.562 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:42.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 1 Processor -- start 2024-09-16T05:39:42.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 1 -- start start 2024-09-16T05:39:42.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient: init 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient(hunting): picked mon.noname-c con 0x7fa50c1339b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient(hunting): picked mon.noname-a con 0x7fa50c09fc50 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient(hunting): picked mon.noname-b con 0x7fa50c13d140 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa50c0a0710 con 0x7fa50c09fc50 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa50c0a09f0 con 0x7fa50c1339b0 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa50c0a0cd0 con 0x7fa50c13d140 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 10 monclient(hunting): authenticate will time out at 1239.338378s 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa511c87640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa50c1339b0 legacy=0x7fa50c133d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa511486640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa50c09fc50 0x7fa50c13ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35406/0 (socket says 172.21.15.63:35406) 2024-09-16T05:39:42.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa511486640 1 -- 172.21.15.63:0/4292531268 learned_addr learned my addr 172.21.15.63:0/4292531268 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151200425 0 0) 0x7fa50c0a0cd0 con 0x7fa50c13d140 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient(hunting): _negotiate 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient(hunting): my global_id is 5978 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4f8003530 con 0x7fa50c13d140 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2743348544 0 0) 0x7fa50c0a0710 con 0x7fa50c09fc50 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient(hunting): _negotiate 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient(hunting): my global_id is 6003 2024-09-16T05:39:42.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa50c0a0cd0 con 0x7fa50c09fc50 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3516906530 0 0) 0x7fa50c0a0cd0 con 0x7fa50c09fc50 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 >> v1:172.21.15.6:6790/0 conn(0x7fa50c13d140 legacy=0x7fa50c13f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 >> v1:172.21.15.63:6789/0 conn(0x7fa50c1339b0 legacy=0x7fa50c133d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient: _finish_hunting 0 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 monclient: found mon.noname-a 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50c0a15b0 con 0x7fa50c09fc50 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient: _finish_auth 0 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient: _check_auth_tickets 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:42.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa4fc001600 con 0x7fa50c09fc50 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 5 monclient: authenticate success, global_id 6003 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa510c85640 10 monclient: dump: 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:42.572 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.558+0000 7fa513eeb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4fc001960 con 0x7fa50c09fc50 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa510c85640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:42.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 4 set_mon_vals no callback set 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa510c85640 1 -- 172.21.15.63:0/4292531268 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa4fc0048b0 con 0x7fa50c09fc50 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa510c85640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa510c85640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa510c85640 10 monclient: dump: 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:42.574 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: shutdown 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- 172.21.15.63:0/4292531268 >> v1:172.21.15.6:6789/0 conn(0x7fa50c09fc50 legacy=0x7fa50c13ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- 172.21.15.63:0/4292531268 shutdown_connections 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- 172.21.15.63:0/4292531268 >> 172.21.15.63:0/4292531268 conn(0x7fa50c0a1e90 msgr2=0x7fa50c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- 172.21.15.63:0/4292531268 shutdown_connections 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- 172.21.15.63:0/4292531268 wait complete. 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: build_initial_monmap 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: monmap: 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:42.575 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 Processor -- start 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- start start 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: init 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:42.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient(hunting): picked mon.noname-b con 0x7fa50c09fc50 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient(hunting): picked mon.noname-c con 0x7fa50c1339b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient(hunting): picked mon.noname-a con 0x7fa50c13d140 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa50c1320a0 con 0x7fa50c13d140 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa50c132380 con 0x7fa50c09fc50 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa50c1cbdd0 con 0x7fa50c1339b0 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa511c87640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa50c09fc50 legacy=0x7fa50c1cafb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient(hunting): authenticate will time out at 1239.342109s 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa511486640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa50c1339b0 0x7fa50c1cb6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59450/0 (socket says 172.21.15.63:59450) 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa511486640 1 -- 172.21.15.63:0/1598913135 learned_addr learned my addr 172.21.15.63:0/1598913135 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2165789553 0 0) 0x7fa50c1cbdd0 con 0x7fa50c1339b0 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:42.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): my global_id is 5981 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4f0003530 con 0x7fa50c1339b0 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1571702881 0 0) 0x7fa50c1320a0 con 0x7fa50c13d140 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): my global_id is 6006 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa50c1cbdd0 con 0x7fa50c13d140 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1563267433 0 0) 0x7fa4f0003530 con 0x7fa50c1339b0 2024-09-16T05:39:42.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa50c1320a0 con 0x7fa50c1339b0 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa4fc001420 con 0x7fa50c1339b0 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): dump: 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:42.579 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3940593304 0 0) 0x7fa50c1cbdd0 con 0x7fa50c13d140 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4f0003530 con 0x7fa50c13d140 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa504001ca0 con 0x7fa50c13d140 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient(hunting): dump: 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:42.580 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3555232161 0 0) 0x7fa50c1320a0 con 0x7fa50c1339b0 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 >> v1:172.21.15.63:6789/0 conn(0x7fa50c09fc50 legacy=0x7fa50c1cafb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 >> v1:172.21.15.6:6789/0 conn(0x7fa50c13d140 legacy=0x7fa50c1d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 monclient: found mon.c 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50c130ed0 con 0x7fa50c1339b0 2024-09-16T05:39:42.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa5027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 5 monclient: authenticate success, global_id 5981 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: _renew_subs 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa50c12e750 con 0x7fa50c1339b0 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: _renew_subs 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.562+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa50c131b00 con 0x7fa50c1339b0 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4fc001960 con 0x7fa50c1339b0 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa4fc004dd0 con 0x7fa50c1339b0 2024-09-16T05:39:42.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5037fe640 4 set_mon_vals no callback set 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5027fc640 10 monclient: dump: 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa513eeb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:42.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa513eeb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:42.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa513eeb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:42.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f4005090 con 0x7fa50c1339b0 2024-09-16T05:39:42.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa4fc00fb70 con 0x7fa50c1339b0 2024-09-16T05:39:42.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(190..190 src has 1..190) ==== 7811+0+0 (unknown 818820096 0 0) 0x7fa4fc0450a0 con 0x7fa50c1339b0 2024-09-16T05:39:42.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.566+0000 7fa511c87640 10 monclient: get_auth_request con 0x7fa4f0036a30 auth_method 0 2024-09-16T05:39:42.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.578+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa4fc0153b0 con 0x7fa50c1339b0 2024-09-16T05:39:42.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.578+0000 7fa5027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:42.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.578+0000 7fa5027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:42.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.746+0000 7fa513eeb640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:42.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.746+0000 7fa513eeb640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:42.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.746+0000 7fa513eeb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:42.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:42.746+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fa4f4005380 con 0x7fa50c1339b0 2024-09-16T05:39:43.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.322+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 added bucket h5 type host to location {rack=r1} v191) ==== 166+0+0 (unknown 150630174 0 0) 0x7fa4fc014cb0 con 0x7fa50c1339b0 2024-09-16T05:39:43.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.322+0000 7fa5027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:43.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.322+0000 7fa5027fc640 10 monclient: _finish_command 2 = system:0 added bucket h5 type host to location {rack=r1} 2024-09-16T05:39:43.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.426+0000 7fa513eeb640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:43.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.426+0000 7fa513eeb640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:43.431 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.426+0000 7fa513eeb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:43.432 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.426+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fa4f4005ac0 con 0x7fa50c1339b0 2024-09-16T05:39:43.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa5027fc640 1 -- 172.21.15.63:0/1598913135 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h5' already exists v191) ==== 145+0+0 (unknown 3537628779 0 0) 0x7fa4fc041e00 con 0x7fa50c1339b0 2024-09-16T05:39:43.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa5027fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:43.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa5027fc640 10 monclient: _finish_command 3 = system:0 bucket 'h5' already exists 2024-09-16T05:39:43.434 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h5' already exists 2024-09-16T05:39:43.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa4f0036a30 legacy=0x7fa4f0038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:43.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa513eeb640 10 monclient: shutdown 2024-09-16T05:39:43.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa513eeb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:43.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 >> v1:172.21.15.6:6790/0 conn(0x7fa50c1339b0 legacy=0x7fa50c1cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:43.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 shutdown_connections 2024-09-16T05:39:43.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 >> 172.21.15.63:0/1598913135 conn(0x7fa50c0a1e90 msgr2=0x7fa50c09cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:43.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 shutdown_connections 2024-09-16T05:39:43.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.430+0000 7fa513eeb640 1 -- 172.21.15.63:0/1598913135 wait complete. 2024-09-16T05:39:43.451 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:43.452 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h6 host rack=r1 2024-09-16T05:39:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient: get_monmap_and_config 2024-09-16T05:39:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient: build_initial_monmap 2024-09-16T05:39:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient: monmap: 2024-09-16T05:39:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:43.569 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:43.569504+0000 2024-09-16T05:39:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:43.569504+0000 2024-09-16T05:39:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:43.570 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:43.570 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:43.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 1 Processor -- start 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 1 -- start start 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient: init 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient(hunting): picked mon.noname-a con 0x7fad50134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient(hunting): picked mon.noname-b con 0x7fad500a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient(hunting): picked mon.noname-c con 0x7fad5006dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad50135420 con 0x7fad50134930 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad5009e830 con 0x7fad5006dfa0 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad500a0dc0 con 0x7fad500a0810 2024-09-16T05:39:43.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient(hunting): authenticate will time out at 1240.346176s 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad554b2640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fad5006dfa0 legacy=0x7fad5013cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad54cb1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fad50134930 0x7fad50134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35420/0 (socket says 172.21.15.63:35420) 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fad500a0810 0x7fad50139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59462/0 (socket says 172.21.15.63:59462) 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad54cb1640 1 -- 172.21.15.63:0/2222732868 learned_addr learned my addr 172.21.15.63:0/2222732868 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278775685 0 0) 0x7fad500a0dc0 con 0x7fad500a0810 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient(hunting): my global_id is 5984 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad40003530 con 0x7fad500a0810 2024-09-16T05:39:43.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3311302726 0 0) 0x7fad50135420 con 0x7fad50134930 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient(hunting): my global_id is 6009 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad500a0dc0 con 0x7fad50134930 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3739234609 0 0) 0x7fad500a0dc0 con 0x7fad50134930 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 >> v1:172.21.15.6:6790/0 conn(0x7fad500a0810 legacy=0x7fad50139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 >> v1:172.21.15.63:6789/0 conn(0x7fad5006dfa0 legacy=0x7fad5013cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 monclient: found mon.noname-a 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:43.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad500a2500 con 0x7fad50134930 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad44001530 con 0x7fad50134930 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 5 monclient: authenticate success, global_id 6009 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: dump: 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.582 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad44001890 con 0x7fad50134930 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad56f15640 4 set_mon_vals no callback set 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 1 -- 172.21.15.63:0/2222732868 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad44001ca0 con 0x7fad50134930 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:43.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.566+0000 7fad4f7fe640 10 monclient: dump: 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:43.584 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: shutdown 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- 172.21.15.63:0/2222732868 >> v1:172.21.15.6:6789/0 conn(0x7fad50134930 legacy=0x7fad50134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- 172.21.15.63:0/2222732868 shutdown_connections 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- 172.21.15.63:0/2222732868 >> 172.21.15.63:0/2222732868 conn(0x7fad500a1e90 msgr2=0x7fad50075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- 172.21.15.63:0/2222732868 shutdown_connections 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- 172.21.15.63:0/2222732868 wait complete. 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: build_initial_monmap 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: monmap: 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:43.585 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 Processor -- start 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- start start 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: init 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient(hunting): picked mon.noname-a con 0x7fad5006dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:43.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient(hunting): picked mon.noname-b con 0x7fad500a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient(hunting): picked mon.noname-c con 0x7fad50134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad50132440 con 0x7fad5006dfa0 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fad500a0810 legacy=0x7fad5009cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad50132720 con 0x7fad500a0810 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad50071eb0 con 0x7fad50134930 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient(hunting): authenticate will time out at 1240.350191s 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad554b2640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fad50134930 0x7fad501d3780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59474/0 (socket says 172.21.15.63:59474) 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad554b2640 1 -- 172.21.15.63:0/2840212568 learned_addr learned my addr 172.21.15.63:0/2840212568 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 856066728 0 0) 0x7fad50071eb0 con 0x7fad50134930 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:43.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): my global_id is 5987 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad34003530 con 0x7fad50134930 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007893800 0 0) 0x7fad50132440 con 0x7fad5006dfa0 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): my global_id is 6012 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad50071eb0 con 0x7fad5006dfa0 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3880924844 0 0) 0x7fad34003530 con 0x7fad50134930 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad50132440 con 0x7fad50134930 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 151793828 0 0) 0x7fad50071eb0 con 0x7fad5006dfa0 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad34003530 con 0x7fad5006dfa0 2024-09-16T05:39:43.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad480043c0 con 0x7fad50134930 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): dump: 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:43.589 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad44001be0 con 0x7fad5006dfa0 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient(hunting): dump: 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:43.590 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1246467497 0 0) 0x7fad50132440 con 0x7fad50134930 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 >> v1:172.21.15.63:6789/0 conn(0x7fad500a0810 legacy=0x7fad5009cca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 >> v1:172.21.15.6:6789/0 conn(0x7fad5006dfa0 legacy=0x7fad5009c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 monclient: found mon.c 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5013e400 con 0x7fad50134930 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:43.591 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:43.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:43.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 5 monclient: authenticate success, global_id 5987 2024-09-16T05:39:43.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: _renew_subs 2024-09-16T05:39:43.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:43.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fad5009c0f0 con 0x7fad50134930 2024-09-16T05:39:43.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: _renew_subs 2024-09-16T05:39:43.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:43.592 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.570+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fad5009d3b0 con 0x7fad50134930 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad48004660 con 0x7fad50134930 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad48004a30 con 0x7fad50134930 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4effd640 4 set_mon_vals no callback set 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4d7fa640 10 monclient: dump: 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:43.593 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad56f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad56f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad56f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad30005090 con 0x7fad50134930 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fad48004bd0 con 0x7fad50134930 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(191..191 src has 1..191) ==== 8149+0+0 (unknown 3106545977 0 0) 0x7fad480454e0 con 0x7fad50134930 2024-09-16T05:39:43.594 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.574+0000 7fad54cb1640 10 monclient: get_auth_request con 0x7fad340369c0 auth_method 0 2024-09-16T05:39:43.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.586+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fad480155e0 con 0x7fad50134930 2024-09-16T05:39:43.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.586+0000 7fad4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:43.595 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.586+0000 7fad4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:43.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.754+0000 7fad56f15640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:43.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.754+0000 7fad56f15640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:43.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.754+0000 7fad56f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:43.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:43.754+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fad30005380 con 0x7fad50134930 2024-09-16T05:39:44.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.346+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 added bucket h6 type host to location {rack=r1} v192) ==== 166+0+0 (unknown 2966974207 0 0) 0x7fad48014ee0 con 0x7fad50134930 2024-09-16T05:39:44.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.346+0000 7fad4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:44.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.346+0000 7fad4d7fa640 10 monclient: _finish_command 2 = system:0 added bucket h6 type host to location {rack=r1} 2024-09-16T05:39:44.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.450+0000 7fad56f15640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:44.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.450+0000 7fad56f15640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:44.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.450+0000 7fad56f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:44.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.450+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fad30005ac0 con 0x7fad50134930 2024-09-16T05:39:44.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad4d7fa640 1 -- 172.21.15.63:0/2840212568 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h6' already exists v192) ==== 145+0+0 (unknown 3885070550 0 0) 0x7fad480149d0 con 0x7fad50134930 2024-09-16T05:39:44.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad4d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:44.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad4d7fa640 10 monclient: _finish_command 3 = system:0 bucket 'h6' already exists 2024-09-16T05:39:44.458 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h6' already exists 2024-09-16T05:39:44.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 >> v1:172.21.15.63:6812/3643207211 conn(0x7fad340369c0 legacy=0x7fad34038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:44.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad56f15640 10 monclient: shutdown 2024-09-16T05:39:44.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad56f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:44.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 >> v1:172.21.15.6:6790/0 conn(0x7fad50134930 legacy=0x7fad501d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:44.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 shutdown_connections 2024-09-16T05:39:44.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 >> 172.21.15.63:0/2840212568 conn(0x7fad500a1e90 msgr2=0x7fad5009f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:44.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 shutdown_connections 2024-09-16T05:39:44.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.454+0000 7fad56f15640 1 -- 172.21.15.63:0/2840212568 wait complete. 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h7 host rack=r1 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient: get_monmap_and_config 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient: build_initial_monmap 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient: monmap: 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:44.601090+0000 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:44.601090+0000 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:44.601 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:44.602 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:44.602 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:44.602 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:44.602 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:44.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 1 Processor -- start 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 1 -- start start 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient: init 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient(hunting): picked mon.noname-a con 0x7f59e4134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient(hunting): picked mon.noname-b con 0x7f59e40a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient(hunting): picked mon.noname-c con 0x7f59e406dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e4135420 con 0x7f59e4134930 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e409e830 con 0x7f59e406dfa0 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e40a0dc0 con 0x7f59e40a0810 2024-09-16T05:39:44.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient(hunting): authenticate will time out at 1241.378041s 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e912e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f59e406dfa0 legacy=0x7f59e413cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e892d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f59e4134930 0x7f59e4134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35436/0 (socket says 172.21.15.63:35436) 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e892d640 1 -- 172.21.15.63:0/2890800250 learned_addr learned my addr 172.21.15.63:0/2890800250 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e3fff640 1 --1- 172.21.15.63:0/2890800250 >> v1:172.21.15.6:6790/0 conn(0x7f59e40a0810 0x7f59e4139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44116/0 (socket says 172.21.15.63:44116) 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784897252 0 0) 0x7f59e40a0dc0 con 0x7f59e40a0810 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient(hunting): my global_id is 5990 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d0003530 con 0x7f59e40a0810 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 468014938 0 0) 0x7f59e4135420 con 0x7f59e4134930 2024-09-16T05:39:44.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient(hunting): my global_id is 6015 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59e40a0dc0 con 0x7f59e4134930 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3122656638 0 0) 0x7f59e40a0dc0 con 0x7f59e4134930 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 >> v1:172.21.15.6:6790/0 conn(0x7f59e40a0810 legacy=0x7f59e4139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 >> v1:172.21.15.63:6789/0 conn(0x7f59e406dfa0 legacy=0x7f59e413cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 monclient: found mon.noname-a 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e40a2500 con 0x7f59e4134930 2024-09-16T05:39:44.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f59d4001980 con 0x7f59e4134930 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: dump: 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:44.608 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 5 monclient: authenticate success, global_id 6015 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59d4001be0 con 0x7f59e4134930 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 1 -- 172.21.15.63:0/2890800250 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f59d4002030 con 0x7f59e4134930 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59eab91640 4 set_mon_vals no callback set 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:44.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.598+0000 7f59e37fe640 10 monclient: dump: 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: shutdown 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:44.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- 172.21.15.63:0/2890800250 >> v1:172.21.15.6:6789/0 conn(0x7f59e4134930 legacy=0x7f59e4134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:44.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- 172.21.15.63:0/2890800250 shutdown_connections 2024-09-16T05:39:44.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- 172.21.15.63:0/2890800250 >> 172.21.15.63:0/2890800250 conn(0x7f59e40a1e90 msgr2=0x7f59e4075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:44.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- 172.21.15.63:0/2890800250 shutdown_connections 2024-09-16T05:39:44.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- 172.21.15.63:0/2890800250 wait complete. 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: build_initial_monmap 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: monmap: 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:44.615 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 Processor -- start 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- start start 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: init 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient(hunting): picked mon.noname-b con 0x7f59e406dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient(hunting): picked mon.noname-c con 0x7f59e40a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient(hunting): picked mon.noname-a con 0x7f59e4134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e409d650 con 0x7f59e4134930 2024-09-16T05:39:44.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e4139cb0 con 0x7f59e406dfa0 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e413a270 con 0x7f59e40a0810 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient(hunting): authenticate will time out at 1241.381906s 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e892d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f59e406dfa0 legacy=0x7f59e409c0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e912e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f59e4134930 0x7f59e41d3780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35448/0 (socket says 172.21.15.63:35448) 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e912e640 1 -- 172.21.15.63:0/1555332421 learned_addr learned my addr 172.21.15.63:0/1555332421 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e3fff640 1 --1- 172.21.15.63:0/1555332421 >> v1:172.21.15.6:6790/0 conn(0x7f59e40a0810 0x7f59e409c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44128/0 (socket says 172.21.15.63:44128) 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2788485476 0 0) 0x7f59e409d650 con 0x7f59e4134930 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): my global_id is 6018 2024-09-16T05:39:44.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59c4003530 con 0x7f59e4134930 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672318231 0 0) 0x7f59e413a270 con 0x7f59e40a0810 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): my global_id is 5993 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59e409d650 con 0x7f59e40a0810 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 391525771 0 0) 0x7f59c4003530 con 0x7f59e4134930 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59e413a270 con 0x7f59e4134930 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f59dc004040 con 0x7f59e4134930 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): dump: 2024-09-16T05:39:44.618 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 642259947 0 0) 0x7f59e409d650 con 0x7f59e40a0810 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59c4003530 con 0x7f59e40a0810 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f59cc0018a0 con 0x7f59e40a0810 2024-09-16T05:39:44.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient(hunting): dump: 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:44.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3293038934 0 0) 0x7f59e413a270 con 0x7f59e4134930 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 >> v1:172.21.15.6:6790/0 conn(0x7f59e40a0810 legacy=0x7f59e409c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 >> v1:172.21.15.63:6789/0 conn(0x7f59e406dfa0 legacy=0x7f59e409c0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 monclient: found mon.a 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e413e2c0 con 0x7f59e4134930 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 5 monclient: authenticate success, global_id 6018 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: _renew_subs 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:44.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59e409cf10 con 0x7f59e4134930 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: _renew_subs 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.602+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f59e4139f90 con 0x7f59e4134930 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59dc0042a0 con 0x7f59e4134930 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f59dc0045f0 con 0x7f59e4134930 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e27fc640 4 set_mon_vals no callback set 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e17fa640 10 monclient: dump: 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:44.622 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59eab91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59eab91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59eab91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c8005090 con 0x7f59e4134930 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f59dc004810 con 0x7f59e4134930 2024-09-16T05:39:44.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(192..192 src has 1..192) ==== 8359+0+0 (unknown 2987149281 0 0) 0x7f59dc045390 con 0x7f59e4134930 2024-09-16T05:39:44.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.606+0000 7f59e892d640 10 monclient: get_auth_request con 0x7f59c4036ac0 auth_method 0 2024-09-16T05:39:44.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.618+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f59dc015360 con 0x7f59e4134930 2024-09-16T05:39:44.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.618+0000 7f59e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:44.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.618+0000 7f59e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:44.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.786+0000 7f59eab91640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:44.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.786+0000 7f59eab91640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:44.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.786+0000 7f59eab91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:44.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:44.786+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f59c8005380 con 0x7f59e4134930 2024-09-16T05:39:45.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.346+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 added bucket h7 type host to location {rack=r1} v193) ==== 166+0+0 (unknown 709799915 0 0) 0x7f59dc014c60 con 0x7f59e4134930 2024-09-16T05:39:45.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.346+0000 7f59e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:45.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.346+0000 7f59e17fa640 10 monclient: _finish_command 2 = system:0 added bucket h7 type host to location {rack=r1} 2024-09-16T05:39:45.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.450+0000 7f59eab91640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:45.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.450+0000 7f59eab91640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:45.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.450+0000 7f59eab91640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:45.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.450+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f59c8005be0 con 0x7f59e4134930 2024-09-16T05:39:45.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59e17fa640 1 -- 172.21.15.63:0/1555332421 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h7' already exists v193) ==== 145+0+0 (unknown 2798139583 0 0) 0x7f59dc018fb0 con 0x7f59e4134930 2024-09-16T05:39:45.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59e17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:45.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59e17fa640 10 monclient: _finish_command 3 = system:0 bucket 'h7' already exists 2024-09-16T05:39:45.457 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h7' already exists 2024-09-16T05:39:45.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 >> v1:172.21.15.63:6812/3643207211 conn(0x7f59c4036ac0 legacy=0x7f59c4038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:45.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59eab91640 10 monclient: shutdown 2024-09-16T05:39:45.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59eab91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:45.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 >> v1:172.21.15.6:6789/0 conn(0x7f59e4134930 legacy=0x7f59e41d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:45.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 shutdown_connections 2024-09-16T05:39:45.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 >> 172.21.15.63:0/1555332421 conn(0x7f59e40a1e90 msgr2=0x7f59e40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:45.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 shutdown_connections 2024-09-16T05:39:45.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.454+0000 7f59eab91640 1 -- 172.21.15.63:0/1555332421 wait complete. 2024-09-16T05:39:45.474 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:45.474 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h8 host rack=r1 2024-09-16T05:39:45.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient: get_monmap_and_config 2024-09-16T05:39:45.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient: build_initial_monmap 2024-09-16T05:39:45.597 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient: monmap: 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:45.597512+0000 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:45.597512+0000 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:45.598 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 1 Processor -- start 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 1 -- start start 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient: init 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef80a22a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef80731d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef81395a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef809c580 con 0x7f5ef81395a0 2024-09-16T05:39:45.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef80a1300 con 0x7f5ef80a22a0 2024-09-16T05:39:45.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef809d0f0 con 0x7f5ef80731d0 2024-09-16T05:39:45.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:45.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5efcb3f640 10 monclient(hunting): authenticate will time out at 1242.374991s 2024-09-16T05:39:45.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5ef80a22a0 legacy=0x7f5ef809f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:45.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef6d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ef81395a0 0x7f5ef813b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35464/0 (socket says 172.21.15.63:35464) 2024-09-16T05:39:45.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef6d9d640 1 -- 172.21.15.63:0/3742497280 learned_addr learned my addr 172.21.15.63:0/3742497280 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:45.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2718576174 0 0) 0x7f5ef809d0f0 con 0x7f5ef80731d0 2024-09-16T05:39:45.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:45.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:45.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:45.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 10 monclient(hunting): my global_id is 5996 2024-09-16T05:39:45.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ee0003450 con 0x7f5ef80731d0 2024-09-16T05:39:45.604 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4136070938 0 0) 0x7f5ef809c580 con 0x7f5ef81395a0 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 10 monclient(hunting): my global_id is 6021 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.594+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef809d0f0 con 0x7f5ef81395a0 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3254009825 0 0) 0x7f5ee0003450 con 0x7f5ef80731d0 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 >> v1:172.21.15.63:6789/0 conn(0x7f5ef80a22a0 legacy=0x7f5ef809f790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 >> v1:172.21.15.6:6789/0 conn(0x7f5ef81395a0 legacy=0x7f5ef813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:45.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: _finish_hunting 0 2024-09-16T05:39:45.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 1 monclient: found mon.noname-b 2024-09-16T05:39:45.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:45.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:45.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef809d3d0 con 0x7f5ef80731d0 2024-09-16T05:39:45.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: _finish_auth 0 2024-09-16T05:39:45.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: _check_auth_tickets 2024-09-16T05:39:45.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:45.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5eec001620 con 0x7f5ef80731d0 2024-09-16T05:39:45.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:45.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:45.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: dump: 2024-09-16T05:39:45.609 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:45.609 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:45.609 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.609 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.610 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:45.610 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:45.610 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:45.610 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:45.610 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:45.610 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:45.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 5 monclient: authenticate success, global_id 5996 2024-09-16T05:39:45.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:45.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5eec0018c0 con 0x7f5ef80731d0 2024-09-16T05:39:45.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:45.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 1 -- 172.21.15.63:0/3742497280 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5eec001cd0 con 0x7f5ef80731d0 2024-09-16T05:39:45.612 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:45.613 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 4 set_mon_vals no callback set 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef559a640 10 monclient: dump: 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:45.614 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient: shutdown 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- 172.21.15.63:0/3742497280 >> v1:172.21.15.6:6790/0 conn(0x7f5ef80731d0 legacy=0x7f5ef809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- 172.21.15.63:0/3742497280 shutdown_connections 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- 172.21.15.63:0/3742497280 >> 172.21.15.63:0/3742497280 conn(0x7f5ef80a1e90 msgr2=0x7f5ef8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- 172.21.15.63:0/3742497280 shutdown_connections 2024-09-16T05:39:45.615 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- 172.21.15.63:0/3742497280 wait complete. 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient: build_initial_monmap 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient: monmap: 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:45.616 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:45.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 Processor -- start 2024-09-16T05:39:45.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- start start 2024-09-16T05:39:45.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient: init 2024-09-16T05:39:45.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:45.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:45.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient(hunting): picked mon.noname-a con 0x7f5ef80731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:45.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient(hunting): picked mon.noname-b con 0x7f5ef80a22a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:45.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient(hunting): picked mon.noname-c con 0x7f5ef81395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:45.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef811f560 con 0x7f5ef80731d0 2024-09-16T05:39:45.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef8120f00 con 0x7f5ef80a22a0 2024-09-16T05:39:45.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef81223b0 con 0x7f5ef81395a0 2024-09-16T05:39:45.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef5d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f5ef80a22a0 legacy=0x7f5ef811ee50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:45.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:45.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5efcb3f640 10 monclient(hunting): authenticate will time out at 1242.378765s 2024-09-16T05:39:45.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ef80731d0 0x7f5ef811e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35470/0 (socket says 172.21.15.63:35470) 2024-09-16T05:39:45.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ef659c640 1 -- 172.21.15.63:0/2149999687 learned_addr learned my addr 172.21.15.63:0/2149999687 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:45.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3180420511 0 0) 0x7f5ef811f560 con 0x7f5ef80731d0 2024-09-16T05:39:45.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:45.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:45.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:45.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 10 monclient(hunting): my global_id is 6024 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed8003530 con 0x7f5ef80731d0 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2269504169 0 0) 0x7f5ef81223b0 con 0x7f5ef81395a0 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 10 monclient(hunting): my global_id is 5999 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef811f560 con 0x7f5ef81395a0 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.598+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2349057581 0 0) 0x7f5ed8003530 con 0x7f5ef80731d0 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ef81223b0 con 0x7f5ef80731d0 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ee8004160 con 0x7f5ef80731d0 2024-09-16T05:39:45.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:45.624 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2179061380 0 0) 0x7f5ef811f560 con 0x7f5ef81395a0 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ed8003530 con 0x7f5ef81395a0 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ef0001c00 con 0x7f5ef81395a0 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:45.625 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1376750959 0 0) 0x7f5ef81223b0 con 0x7f5ef80731d0 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 >> v1:172.21.15.6:6790/0 conn(0x7f5ef81395a0 legacy=0x7f5ef81d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 >> v1:172.21.15.63:6789/0 conn(0x7f5ef80a22a0 legacy=0x7f5ef811ee50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:45.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 monclient: found mon.a 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef8136590 con 0x7f5ef80731d0 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 5 monclient: authenticate success, global_id 6024 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 10 monclient: _renew_subs 2024-09-16T05:39:45.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef80a2e50 con 0x7f5ef80731d0 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 10 monclient: _renew_subs 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ef8120c80 con 0x7f5ef80731d0 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ee8004410 con 0x7f5ef80731d0 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ee80047e0 con 0x7f5ef80731d0 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 10 monclient: dump: 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:45.628 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:45.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee7fff640 4 set_mon_vals no callback set 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5edc005090 con 0x7f5ef80731d0 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5ee8004980 con 0x7f5ef80731d0 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(193..193 src has 1..193) ==== 8569+0+0 (unknown 1429286820 0 0) 0x7f5ee80454c0 con 0x7f5ef80731d0 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.602+0000 7f5ef5d9b640 10 monclient: get_auth_request con 0x7f5ed8036a70 auth_method 0 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.614+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5ee80153b0 con 0x7f5ef80731d0 2024-09-16T05:39:45.630 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.614+0000 7f5ee6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:45.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.614+0000 7f5ee6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:45.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.778+0000 7f5efcb3f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:45.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.778+0000 7f5efcb3f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:45.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.778+0000 7f5efcb3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:45.784 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:45.778+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f5edc005380 con 0x7f5ef80731d0 2024-09-16T05:39:46.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.350+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 added bucket h8 type host to location {rack=r1} v194) ==== 166+0+0 (unknown 160102521 0 0) 0x7f5ee8014cb0 con 0x7f5ef80731d0 2024-09-16T05:39:46.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.350+0000 7f5ee6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:46.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.350+0000 7f5ee6ffd640 10 monclient: _finish_command 2 = system:0 added bucket h8 type host to location {rack=r1} 2024-09-16T05:39:46.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.458+0000 7f5efcb3f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:46.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.458+0000 7f5efcb3f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:46.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.458+0000 7f5efcb3f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:46.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.458+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f5edc005ac0 con 0x7f5ef80731d0 2024-09-16T05:39:46.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.458+0000 7f5ee6ffd640 1 -- 172.21.15.63:0/2149999687 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h8' already exists v194) ==== 145+0+0 (unknown 124794010 0 0) 0x7f5ee8019000 con 0x7f5ef80731d0 2024-09-16T05:39:46.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.458+0000 7f5ee6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:46.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.458+0000 7f5ee6ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h8' already exists 2024-09-16T05:39:46.462 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h8' already exists 2024-09-16T05:39:46.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.462+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5ed8036a70 legacy=0x7f5ed8038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:46.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.462+0000 7f5efcb3f640 10 monclient: shutdown 2024-09-16T05:39:46.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.462+0000 7f5efcb3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:46.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.462+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 >> v1:172.21.15.6:6789/0 conn(0x7f5ef80731d0 legacy=0x7f5ef811e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:46.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.462+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 shutdown_connections 2024-09-16T05:39:46.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.462+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 >> 172.21.15.63:0/2149999687 conn(0x7f5ef80a1e90 msgr2=0x7f5ef8138970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:46.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.462+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 shutdown_connections 2024-09-16T05:39:46.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.462+0000 7f5efcb3f640 1 -- 172.21.15.63:0/2149999687 wait complete. 2024-09-16T05:39:46.479 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:46.479 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h9 host rack=r1 2024-09-16T05:39:46.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient: get_monmap_and_config 2024-09-16T05:39:46.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient: build_initial_monmap 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient: monmap: 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:46.602690+0000 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:46.602690+0000 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:46.603 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 1 Processor -- start 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 1 -- start start 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient: init 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient(hunting): picked mon.noname-a con 0x7f3b101339b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient(hunting): picked mon.noname-b con 0x7f3b1009fc50 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient(hunting): picked mon.noname-c con 0x7f3b1013d140 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b100a0710 con 0x7f3b101339b0 2024-09-16T05:39:46.606 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b100a09f0 con 0x7f3b1013d140 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b100a0cd0 con 0x7f3b1009fc50 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b178eb640 10 monclient(hunting): authenticate will time out at 1243.379615s 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.598+0000 7f3b15e88640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3b1013d140 legacy=0x7f3b1013f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b15687640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3b101339b0 0x7f3b10133d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35484/0 (socket says 172.21.15.63:35484) 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b14e86640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3b1009fc50 0x7f3b1013ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44146/0 (socket says 172.21.15.63:44146) 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b15687640 1 -- 172.21.15.63:0/2716399193 learned_addr learned my addr 172.21.15.63:0/2716399193 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952475936 0 0) 0x7f3b100a0cd0 con 0x7f3b1009fc50 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient(hunting): my global_id is 6002 2024-09-16T05:39:46.607 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3afc003530 con 0x7f3b1009fc50 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1512423791 0 0) 0x7f3b100a0710 con 0x7f3b101339b0 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient(hunting): my global_id is 6027 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b100a0cd0 con 0x7f3b101339b0 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3439540536 0 0) 0x7f3afc003530 con 0x7f3b1009fc50 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 >> v1:172.21.15.63:6789/0 conn(0x7f3b1013d140 legacy=0x7f3b1013f530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 >> v1:172.21.15.6:6789/0 conn(0x7f3b101339b0 legacy=0x7f3b10133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 monclient: found mon.noname-b 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:46.608 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b100a15b0 con 0x7f3b1009fc50 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: _finish_auth 0 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3b00001620 con 0x7f3b1009fc50 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: dump: 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.609 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 5 monclient: authenticate success, global_id 6002 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b00001880 con 0x7f3b1009fc50 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 1 -- 172.21.15.63:0/2716399193 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3b00001c90 con 0x7f3b1009fc50 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 4 set_mon_vals no callback set 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:46.610 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b07fff640 10 monclient: dump: 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:46.611 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:46.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 10 monclient: shutdown 2024-09-16T05:39:46.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:46.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 1 -- 172.21.15.63:0/2716399193 >> v1:172.21.15.6:6790/0 conn(0x7f3b1009fc50 legacy=0x7f3b1013ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:46.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 1 -- 172.21.15.63:0/2716399193 shutdown_connections 2024-09-16T05:39:46.616 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 1 -- 172.21.15.63:0/2716399193 >> 172.21.15.63:0/2716399193 conn(0x7f3b100a1e90 msgr2=0x7f3b10075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 1 -- 172.21.15.63:0/2716399193 shutdown_connections 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 1 -- 172.21.15.63:0/2716399193 wait complete. 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 10 monclient: build_initial_monmap 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 10 monclient: monmap: 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:46.617 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 1 Processor -- start 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 1 -- start start 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 10 monclient: init 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.602+0000 7f3b178eb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient(hunting): picked mon.noname-a con 0x7f3b1009fc50 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient(hunting): picked mon.noname-b con 0x7f3b101339b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient(hunting): picked mon.noname-c con 0x7f3b1013d140 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b1013ca70 con 0x7f3b1009fc50 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b101326c0 con 0x7f3b101339b0 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b101329a0 con 0x7f3b1013d140 2024-09-16T05:39:46.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b14e86640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3b101339b0 legacy=0x7f3b101cfbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient(hunting): authenticate will time out at 1243.384114s 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b15e88640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3b1013d140 0x7f3b101d3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44158/0 (socket says 172.21.15.63:44158) 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b15687640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3b1009fc50 0x7f3b10132dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35490/0 (socket says 172.21.15.63:35490) 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b15e88640 1 -- 172.21.15.63:0/794674256 learned_addr learned my addr 172.21.15.63:0/794674256 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944850253 0 0) 0x7f3b0c0011e0 con 0x7f3b1009fc50 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): my global_id is 6030 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3af4003560 con 0x7f3b1009fc50 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 941081451 0 0) 0x7f3b101329a0 con 0x7f3b1013d140 2024-09-16T05:39:46.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): my global_id is 6005 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b0c0011e0 con 0x7f3b1013d140 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 217701408 0 0) 0x7f3af4003560 con 0x7f3b1009fc50 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b101329a0 con 0x7f3b1009fc50 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2998483847 0 0) 0x7f3b0c0011e0 con 0x7f3b1013d140 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3af4003560 con 0x7f3b1013d140 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3b0c0014b0 con 0x7f3b1009fc50 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): dump: 2024-09-16T05:39:46.620 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3b08004300 con 0x7f3b1013d140 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:46.621 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient(hunting): dump: 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2625281665 0 0) 0x7f3b101329a0 con 0x7f3b1009fc50 2024-09-16T05:39:46.622 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 >> v1:172.21.15.6:6790/0 conn(0x7f3b1013d140 legacy=0x7f3b101d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 >> v1:172.21.15.63:6789/0 conn(0x7f3b101339b0 legacy=0x7f3b101cfbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 monclient: found mon.a 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b101409c0 con 0x7f3b1009fc50 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 5 monclient: authenticate success, global_id 6030 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient: _renew_subs 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1009c0f0 con 0x7f3b1009fc50 2024-09-16T05:39:46.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient: _renew_subs 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3b101d4a40 con 0x7f3b1009fc50 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b0c004aa0 con 0x7f3b1009fc50 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3b0c004df0 con 0x7f3b1009fc50 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 10 monclient: dump: 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:46.624 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b077fe640 4 set_mon_vals no callback set 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af0005090 con 0x7f3b1009fc50 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3b0c00fb90 con 0x7f3b1009fc50 2024-09-16T05:39:46.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(194..194 src has 1..194) ==== 8779+0+0 (unknown 795337938 0 0) 0x7f3b0c0458c0 con 0x7f3b1009fc50 2024-09-16T05:39:46.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.606+0000 7f3b14e86640 10 monclient: get_auth_request con 0x7f3af40369f0 auth_method 0 2024-09-16T05:39:46.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.622+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3b1013ca70 con 0x7f3b1009fc50 2024-09-16T05:39:46.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.622+0000 7f3b05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:46.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.622+0000 7f3b05ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:46.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.786+0000 7f3b178eb640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:46.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.786+0000 7f3b178eb640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:46.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.786+0000 7f3b178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:46.791 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:46.786+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3af0005380 con 0x7f3b1009fc50 2024-09-16T05:39:47.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.370+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 added bucket h9 type host to location {rack=r1} v195) ==== 166+0+0 (unknown 2468100461 0 0) 0x7f3b0c015760 con 0x7f3b1009fc50 2024-09-16T05:39:47.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.370+0000 7f3b05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:47.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.370+0000 7f3b05ffb640 10 monclient: _finish_command 2 = system:0 added bucket h9 type host to location {rack=r1} 2024-09-16T05:39:47.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.474+0000 7f3b178eb640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:47.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.474+0000 7f3b178eb640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:47.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.474+0000 7f3b178eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:47.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.474+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3af0005ac0 con 0x7f3b1009fc50 2024-09-16T05:39:47.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.474+0000 7f3b05ffb640 1 -- 172.21.15.63:0/794674256 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h9' already exists v195) ==== 145+0+0 (unknown 1177122035 0 0) 0x7f3b0c015060 con 0x7f3b1009fc50 2024-09-16T05:39:47.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.474+0000 7f3b05ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:47.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.474+0000 7f3b05ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h9' already exists 2024-09-16T05:39:47.479 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h9' already exists 2024-09-16T05:39:47.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.478+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3af40369f0 legacy=0x7f3af4038eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:47.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.478+0000 7f3b178eb640 10 monclient: shutdown 2024-09-16T05:39:47.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.478+0000 7f3b178eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:47.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.478+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 >> v1:172.21.15.6:6789/0 conn(0x7f3b1009fc50 legacy=0x7f3b10132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:47.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.478+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 shutdown_connections 2024-09-16T05:39:47.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.478+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 >> 172.21.15.63:0/794674256 conn(0x7f3b100a1e90 msgr2=0x7f3b10072140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:47.481 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.478+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 shutdown_connections 2024-09-16T05:39:47.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.478+0000 7f3b178eb640 1 -- 172.21.15.63:0/794674256 wait complete. 2024-09-16T05:39:47.495 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:47.496 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h10 host rack=r1 2024-09-16T05:39:47.618 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.614+0000 7f71ef6d8640 10 monclient: get_monmap_and_config 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.614+0000 7f71ef6d8640 10 monclient: build_initial_monmap 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.614+0000 7f71ef6d8640 10 monclient: monmap: 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:47.618797+0000 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:47.618797+0000 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:47.619 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:47.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.614+0000 7f71ef6d8640 1 Processor -- start 2024-09-16T05:39:47.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.614+0000 7f71ef6d8640 1 -- start start 2024-09-16T05:39:47.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.614+0000 7f71ef6d8640 10 monclient: init 2024-09-16T05:39:47.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:47.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:47.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient(hunting): picked mon.noname-c con 0x7f71e8134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:47.623 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient(hunting): picked mon.noname-b con 0x7f71e80a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient(hunting): picked mon.noname-a con 0x7f71e806dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e8135420 con 0x7f71e806dfa0 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ed474640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f71e8134930 legacy=0x7f71e8134d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e809e830 con 0x7f71e8134930 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e80a0dc0 con 0x7f71e80a0810 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient(hunting): authenticate will time out at 1244.396243s 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ecc73640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f71e80a0810 0x7f71e8139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44174/0 (socket says 172.21.15.63:44174) 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71edc75640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f71e806dfa0 0x7f71e813cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35492/0 (socket says 172.21.15.63:35492) 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ecc73640 1 -- 172.21.15.63:0/3445059412 learned_addr learned my addr 172.21.15.63:0/3445059412 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1445738699 0 0) 0x7f71e80a0dc0 con 0x7f71e80a0810 2024-09-16T05:39:47.624 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient(hunting): my global_id is 6008 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d4003530 con 0x7f71e80a0810 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075627811 0 0) 0x7f71e8135420 con 0x7f71e806dfa0 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient(hunting): my global_id is 6033 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e80a0dc0 con 0x7f71e806dfa0 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1919372928 0 0) 0x7f71d4003530 con 0x7f71e80a0810 2024-09-16T05:39:47.625 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 >> v1:172.21.15.63:6789/0 conn(0x7f71e8134930 legacy=0x7f71e8134d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 >> v1:172.21.15.6:6789/0 conn(0x7f71e806dfa0 legacy=0x7f71e813cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 monclient: found mon.noname-b 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e80a2500 con 0x7f71e80a0810 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: _finish_auth 0 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f71e4001620 con 0x7f71e80a0810 2024-09-16T05:39:47.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: dump: 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:47.627 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 5 monclient: authenticate success, global_id 6008 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71e4001940 con 0x7f71e80a0810 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 1 -- 172.21.15.63:0/3445059412 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f71e4001d10 con 0x7f71e80a0810 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 4 set_mon_vals no callback set 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71dffff640 10 monclient: dump: 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.628 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient: shutdown 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- 172.21.15.63:0/3445059412 >> v1:172.21.15.6:6790/0 conn(0x7f71e80a0810 legacy=0x7f71e8139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- 172.21.15.63:0/3445059412 shutdown_connections 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- 172.21.15.63:0/3445059412 >> 172.21.15.63:0/3445059412 conn(0x7f71e80a1e90 msgr2=0x7f71e8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- 172.21.15.63:0/3445059412 shutdown_connections 2024-09-16T05:39:47.629 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- 172.21.15.63:0/3445059412 wait complete. 2024-09-16T05:39:47.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient: build_initial_monmap 2024-09-16T05:39:47.634 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient: monmap: 2024-09-16T05:39:47.634 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:47.634 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 Processor -- start 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 1 -- start start 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.618+0000 7f71ef6d8640 10 monclient: init 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:47.635 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient(hunting): picked mon.noname-b con 0x7f71e806dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient(hunting): picked mon.noname-a con 0x7f71e80a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient(hunting): picked mon.noname-c con 0x7f71e8134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e813e2a0 con 0x7f71e80a0810 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e81380d0 con 0x7f71e806dfa0 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ed474640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f71e806dfa0 legacy=0x7f71e8134d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e8135b30 con 0x7f71e8134930 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient(hunting): authenticate will time out at 1244.400595s 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ecc73640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f71e80a0810 0x7f71e8135420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35498/0 (socket says 172.21.15.63:35498) 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ecc73640 1 -- 172.21.15.63:0/873633316 learned_addr learned my addr 172.21.15.63:0/873633316 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 849319631 0 0) 0x7f71e8135b30 con 0x7f71e8134930 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:47.636 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): my global_id is 6011 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71cc003530 con 0x7f71e8134930 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 495510500 0 0) 0x7f71e80a2500 con 0x7f71e80a0810 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): my global_id is 6036 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e8135b30 con 0x7f71e80a0810 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3313283978 0 0) 0x7f71cc003530 con 0x7f71e8134930 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71e80a2500 con 0x7f71e8134930 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f71e0001c40 con 0x7f71e8134930 2024-09-16T05:39:47.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): dump: 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:47.638 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 627204887 0 0) 0x7f71e8135b30 con 0x7f71e80a0810 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71cc003530 con 0x7f71e80a0810 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f71e40015e0 con 0x7f71e80a0810 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient(hunting): dump: 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:47.639 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 377552868 0 0) 0x7f71e80a2500 con 0x7f71e8134930 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 >> v1:172.21.15.63:6789/0 conn(0x7f71e806dfa0 legacy=0x7f71e8134d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 >> v1:172.21.15.6:6789/0 conn(0x7f71e80a0810 legacy=0x7f71e8135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 monclient: found mon.c 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e8071e10 con 0x7f71e8134930 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 5 monclient: authenticate success, global_id 6011 2024-09-16T05:39:47.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient: _renew_subs 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71e8136e40 con 0x7f71e8134930 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient: _renew_subs 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f71e81383b0 con 0x7f71e8134930 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71e0004b70 con 0x7f71e8134930 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f71e0004ec0 con 0x7f71e8134930 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71df7fe640 4 set_mon_vals no callback set 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 10 monclient: dump: 2024-09-16T05:39:47.641 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71c8005090 con 0x7f71e8134930 2024-09-16T05:39:47.642 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f71e000fc60 con 0x7f71e8134930 2024-09-16T05:39:47.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(195..195 src has 1..195) ==== 8989+0+0 (unknown 4033596835 0 0) 0x7f71e0045830 con 0x7f71e8134930 2024-09-16T05:39:47.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.622+0000 7f71ed474640 10 monclient: get_auth_request con 0x7f71cc036a00 auth_method 0 2024-09-16T05:39:47.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.638+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f71e0015580 con 0x7f71e8134930 2024-09-16T05:39:47.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.638+0000 7f71ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:47.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.638+0000 7f71ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:47.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.798+0000 7f71ef6d8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:47.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.798+0000 7f71ef6d8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:47.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.798+0000 7f71ef6d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:47.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:47.798+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f71c8005380 con 0x7f71e8134930 2024-09-16T05:39:48.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.390+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 added bucket h10 type host to location {rack=r1} v196) ==== 168+0+0 (unknown 2115945814 0 0) 0x7f71e0014e80 con 0x7f71e8134930 2024-09-16T05:39:48.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.390+0000 7f71ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:48.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.390+0000 7f71ddffb640 10 monclient: _finish_command 2 = system:0 added bucket h10 type host to location {rack=r1} 2024-09-16T05:39:48.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.490+0000 7f71ef6d8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:48.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.490+0000 7f71ef6d8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:48.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.490+0000 7f71ef6d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:48.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.490+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f71c8005ac0 con 0x7f71e8134930 2024-09-16T05:39:48.497 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ddffb640 1 -- 172.21.15.63:0/873633316 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h10' already exists v196) ==== 147+0+0 (unknown 2504927727 0 0) 0x7f71e00191d0 con 0x7f71e8134930 2024-09-16T05:39:48.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ddffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:48.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ddffb640 10 monclient: _finish_command 3 = system:0 bucket 'h10' already exists 2024-09-16T05:39:48.498 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h10' already exists 2024-09-16T05:39:48.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 >> v1:172.21.15.63:6812/3643207211 conn(0x7f71cc036a00 legacy=0x7f71cc038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:48.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ef6d8640 10 monclient: shutdown 2024-09-16T05:39:48.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ef6d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:48.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 >> v1:172.21.15.6:6790/0 conn(0x7f71e8134930 legacy=0x7f71e81d17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:48.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 shutdown_connections 2024-09-16T05:39:48.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.494+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 >> 172.21.15.63:0/873633316 conn(0x7f71e80a1e90 msgr2=0x7f71e8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:48.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.498+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 shutdown_connections 2024-09-16T05:39:48.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.498+0000 7f71ef6d8640 1 -- 172.21.15.63:0/873633316 wait complete. 2024-09-16T05:39:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:48.515 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h11 host rack=r1 2024-09-16T05:39:48.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient: get_monmap_and_config 2024-09-16T05:39:48.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient: build_initial_monmap 2024-09-16T05:39:48.640 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient: monmap: 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:48.640652+0000 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:48.640652+0000 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:48.641 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:48.643 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 1 Processor -- start 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 1 -- start start 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient: init 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient(hunting): picked mon.noname-b con 0x7f64dc134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient(hunting): picked mon.noname-a con 0x7f64dc0a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient(hunting): picked mon.noname-c con 0x7f64dc06dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc135420 con 0x7f64dc0a0810 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc09e830 con 0x7f64dc06dfa0 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc0a0dc0 con 0x7f64dc134930 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:48.644 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient(hunting): authenticate will time out at 1245.417694s 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e12b6640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f64dc06dfa0 legacy=0x7f64dc13cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e0ab5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f64dc134930 0x7f64dc134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44188/0 (socket says 172.21.15.63:44188) 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64dbfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f64dc0a0810 0x7f64dc139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35502/0 (socket says 172.21.15.63:35502) 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e0ab5640 1 -- 172.21.15.63:0/763157260 learned_addr learned my addr 172.21.15.63:0/763157260 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053477588 0 0) 0x7f64dc135420 con 0x7f64dc0a0810 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient(hunting): my global_id is 6039 2024-09-16T05:39:48.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c4003530 con 0x7f64dc0a0810 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3900497255 0 0) 0x7f64dc0a0dc0 con 0x7f64dc134930 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient(hunting): my global_id is 6014 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64dc135420 con 0x7f64dc134930 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3282595361 0 0) 0x7f64c4003530 con 0x7f64dc0a0810 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 >> v1:172.21.15.6:6790/0 conn(0x7f64dc134930 legacy=0x7f64dc134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 >> v1:172.21.15.63:6789/0 conn(0x7f64dc06dfa0 legacy=0x7f64dc13cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:48.646 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 monclient: found mon.noname-a 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc0a2500 con 0x7f64dc0a0810 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f64d0001620 con 0x7f64dc0a0810 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: dump: 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:48.647 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 5 monclient: authenticate success, global_id 6039 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64d0001880 con 0x7f64dc0a0810 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:48.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:48.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 4 set_mon_vals no callback set 2024-09-16T05:39:48.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 1 -- 172.21.15.63:0/763157260 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f64d0001c90 con 0x7f64dc0a0810 2024-09-16T05:39:48.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:48.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:48.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64db7fe640 10 monclient: dump: 2024-09-16T05:39:48.649 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:48.649 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:48.650 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.650 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.650 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:48.650 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:48.650 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:48.650 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:48.650 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:48.650 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:48.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 10 monclient: shutdown 2024-09-16T05:39:48.654 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.638+0000 7f64e2d19640 1 -- 172.21.15.63:0/763157260 >> v1:172.21.15.6:6789/0 conn(0x7f64dc0a0810 legacy=0x7f64dc139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- 172.21.15.63:0/763157260 shutdown_connections 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- 172.21.15.63:0/763157260 >> 172.21.15.63:0/763157260 conn(0x7f64dc0a1e90 msgr2=0x7f64dc075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- 172.21.15.63:0/763157260 shutdown_connections 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- 172.21.15.63:0/763157260 wait complete. 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: build_initial_monmap 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: monmap: 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:48.655 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 Processor -- start 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- start start 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: init 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient(hunting): picked mon.noname-a con 0x7f64dc06dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient(hunting): picked mon.noname-b con 0x7f64dc0a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient(hunting): picked mon.noname-c con 0x7f64dc134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:48.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc09e120 con 0x7f64dc06dfa0 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc071d90 con 0x7f64dc0a0810 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc13e2a0 con 0x7f64dc134930 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient(hunting): authenticate will time out at 1245.421671s 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64dbfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f64dc0a0810 legacy=0x7f64dc09c800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e0ab5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f64dc06dfa0 0x7f64dc09c0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35510/0 (socket says 172.21.15.63:35510) 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e0ab5640 1 -- 172.21.15.63:0/1132434373 learned_addr learned my addr 172.21.15.63:0/1132434373 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 77116 0 0) 0x7f64dc09e120 con 0x7f64dc06dfa0 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): my global_id is 6042 2024-09-16T05:39:48.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c8003530 con 0x7f64dc06dfa0 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4216241346 0 0) 0x7f64dc13e2a0 con 0x7f64dc134930 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): my global_id is 6017 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64dc09e120 con 0x7f64dc134930 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 535706944 0 0) 0x7f64dc09e120 con 0x7f64dc134930 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64dc13e2a0 con 0x7f64dc134930 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f64d4004340 con 0x7f64dc134930 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): dump: 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:48.658 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:48.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1491307140 0 0) 0x7f64c8003530 con 0x7f64dc06dfa0 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64dc09e120 con 0x7f64dc06dfa0 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f64cc001a00 con 0x7f64dc06dfa0 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient(hunting): dump: 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.660 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3663395992 0 0) 0x7f64dc13e2a0 con 0x7f64dc134930 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 >> v1:172.21.15.63:6789/0 conn(0x7f64dc0a0810 legacy=0x7f64dc09c800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 >> v1:172.21.15.6:6789/0 conn(0x7f64dc06dfa0 legacy=0x7f64dc09c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 monclient: found mon.c 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:48.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc139cf0 con 0x7f64dc134930 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 5 monclient: authenticate success, global_id 6017 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: _renew_subs 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc139fd0 con 0x7f64dc134930 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: _renew_subs 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f64dc09cf10 con 0x7f64dc134930 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64d40045a0 con 0x7f64dc134930 2024-09-16T05:39:48.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f64d40048f0 con 0x7f64dc134930 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64d97fa640 10 monclient: dump: 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:48.663 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:48.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.642+0000 7f64daffd640 4 set_mon_vals no callback set 2024-09-16T05:39:48.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.646+0000 7f64e2d19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:48.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.646+0000 7f64e2d19640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:48.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.646+0000 7f64e2d19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:48.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.646+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64bc005090 con 0x7f64dc134930 2024-09-16T05:39:48.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.646+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f64d4004b10 con 0x7f64dc134930 2024-09-16T05:39:48.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.646+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(196..196 src has 1..196) ==== 9202+0+0 (unknown 3399056260 0 0) 0x7f64d40459a0 con 0x7f64dc134930 2024-09-16T05:39:48.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.646+0000 7f64e0ab5640 10 monclient: get_auth_request con 0x7f64c8036a20 auth_method 0 2024-09-16T05:39:48.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.658+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f64d40155c0 con 0x7f64dc134930 2024-09-16T05:39:48.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.658+0000 7f64d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:48.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.658+0000 7f64d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:48.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.826+0000 7f64e2d19640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:48.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.826+0000 7f64e2d19640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:48.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.826+0000 7f64e2d19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:48.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:48.826+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f64bc005380 con 0x7f64dc134930 2024-09-16T05:39:49.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.402+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 added bucket h11 type host to location {rack=r1} v197) ==== 168+0+0 (unknown 34357419 0 0) 0x7f64d4019270 con 0x7f64dc134930 2024-09-16T05:39:49.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.402+0000 7f64d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:49.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.402+0000 7f64d97fa640 10 monclient: _finish_command 2 = system:0 added bucket h11 type host to location {rack=r1} 2024-09-16T05:39:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.506+0000 7f64e2d19640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.506+0000 7f64e2d19640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.506+0000 7f64e2d19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:49.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.506+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f64bc005be0 con 0x7f64dc134930 2024-09-16T05:39:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64d97fa640 1 -- 172.21.15.63:0/1132434373 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h11' already exists v197) ==== 147+0+0 (unknown 2667065346 0 0) 0x7f64d401db30 con 0x7f64dc134930 2024-09-16T05:39:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64d97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64d97fa640 10 monclient: _finish_command 3 = system:0 bucket 'h11' already exists 2024-09-16T05:39:49.513 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h11' already exists 2024-09-16T05:39:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 >> v1:172.21.15.63:6812/3643207211 conn(0x7f64c8036a20 legacy=0x7f64c8038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64e2d19640 10 monclient: shutdown 2024-09-16T05:39:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64e2d19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 >> v1:172.21.15.6:6790/0 conn(0x7f64dc134930 legacy=0x7f64dc1d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 shutdown_connections 2024-09-16T05:39:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 >> 172.21.15.63:0/1132434373 conn(0x7f64dc0a1e90 msgr2=0x7f64dc0a34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 shutdown_connections 2024-09-16T05:39:49.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.510+0000 7f64e2d19640 1 -- 172.21.15.63:0/1132434373 wait complete. 2024-09-16T05:39:49.530 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:49.531 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h12 host rack=r1 2024-09-16T05:39:49.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient: get_monmap_and_config 2024-09-16T05:39:49.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient: build_initial_monmap 2024-09-16T05:39:49.653 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient: monmap: 2024-09-16T05:39:49.654 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:49.654 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:49.654 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:49.653118+0000 2024-09-16T05:39:49.654 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:49.653118+0000 2024-09-16T05:39:49.654 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:49.654 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:49.655 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:49.655 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:49.655 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:49.655 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:49.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 1 Processor -- start 2024-09-16T05:39:49.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 1 -- start start 2024-09-16T05:39:49.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient: init 2024-09-16T05:39:49.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient(hunting): picked mon.noname-c con 0x7f7170134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient(hunting): picked mon.noname-a con 0x7f71700a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient(hunting): picked mon.noname-b con 0x7f717006dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7170135420 con 0x7f71700a0810 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717009e830 con 0x7f7170134930 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71700a0dc0 con 0x7f717006dfa0 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 10 monclient(hunting): authenticate will time out at 1246.430062s 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f717637a640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7170134930 legacy=0x7f7170134d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7176b7b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f717006dfa0 0x7f717013cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44212/0 (socket says 172.21.15.63:44212) 2024-09-16T05:39:49.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7176b7b640 1 -- 172.21.15.63:0/500747703 learned_addr learned my addr 172.21.15.63:0/500747703 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364040905 0 0) 0x7f7170135420 con 0x7f71700a0810 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient(hunting): _negotiate 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient(hunting): my global_id is 6045 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f715c003530 con 0x7f71700a0810 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3459442653 0 0) 0x7f71700a0dc0 con 0x7f717006dfa0 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient(hunting): _negotiate 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient(hunting): my global_id is 6020 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7170135420 con 0x7f717006dfa0 2024-09-16T05:39:49.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1848031019 0 0) 0x7f7170135420 con 0x7f717006dfa0 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 >> v1:172.21.15.63:6789/0 conn(0x7f7170134930 legacy=0x7f7170134d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 >> v1:172.21.15.6:6789/0 conn(0x7f71700a0810 legacy=0x7f7170139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient: _finish_hunting 0 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 monclient: found mon.noname-b 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71700a2500 con 0x7f717006dfa0 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient: _finish_auth 0 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient: _check_auth_tickets 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7168001830 con 0x7f717006dfa0 2024-09-16T05:39:49.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:49.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:49.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f7175378640 10 monclient: dump: 2024-09-16T05:39:49.662 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:49.662 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:49.662 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.662 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.662 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:49.663 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:49.663 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:49.663 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:49.664 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:49.664 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:49.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 5 monclient: authenticate success, global_id 6020 2024-09-16T05:39:49.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.650+0000 7f71785de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:49.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7168001a90 con 0x7f717006dfa0 2024-09-16T05:39:49.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7175378640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7175378640 1 -- 172.21.15.63:0/500747703 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7168001ee0 con 0x7f717006dfa0 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 4 set_mon_vals no callback set 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7175378640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7175378640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7175378640 10 monclient: dump: 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.665 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient: shutdown 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- 172.21.15.63:0/500747703 >> v1:172.21.15.6:6790/0 conn(0x7f717006dfa0 legacy=0x7f717013cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- 172.21.15.63:0/500747703 shutdown_connections 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- 172.21.15.63:0/500747703 >> 172.21.15.63:0/500747703 conn(0x7f71700a1e90 msgr2=0x7f7170075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- 172.21.15.63:0/500747703 shutdown_connections 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- 172.21.15.63:0/500747703 wait complete. 2024-09-16T05:39:49.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient: build_initial_monmap 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient: monmap: 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 Processor -- start 2024-09-16T05:39:49.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- start start 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient: init 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient(hunting): picked mon.noname-a con 0x7f717006dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient(hunting): picked mon.noname-c con 0x7f71700a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient(hunting): picked mon.noname-b con 0x7f7170134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7170134340 con 0x7f717006dfa0 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7170071de0 con 0x7f7170134930 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7170139f70 con 0x7f71700a0810 2024-09-16T05:39:49.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:49.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f71785de640 10 monclient(hunting): authenticate will time out at 1246.434182s 2024-09-16T05:39:49.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7176b7b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f7170134930 legacy=0x7f71701d3830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:49.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7175b79640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f71700a0810 0x7f71701d0100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44220/0 (socket says 172.21.15.63:44220) 2024-09-16T05:39:49.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7175b79640 1 -- 172.21.15.63:0/4193020163 learned_addr learned my addr 172.21.15.63:0/4193020163 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:49.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f717637a640 1 --1- 172.21.15.63:0/4193020163 >> v1:172.21.15.6:6789/0 conn(0x7f717006dfa0 0x7f71701cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35526/0 (socket says 172.21.15.63:35526) 2024-09-16T05:39:49.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048063130 0 0) 0x7f7170134340 con 0x7f717006dfa0 2024-09-16T05:39:49.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:49.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): my global_id is 6048 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7154003530 con 0x7f717006dfa0 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101785843 0 0) 0x7f7170139f70 con 0x7f71700a0810 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): my global_id is 6023 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7170134340 con 0x7f71700a0810 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4139760355 0 0) 0x7f7170134340 con 0x7f71700a0810 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7170139f70 con 0x7f71700a0810 2024-09-16T05:39:49.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1927926048 0 0) 0x7f7154003530 con 0x7f717006dfa0 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7170134340 con 0x7f717006dfa0 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7164001a80 con 0x7f71700a0810 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:49.674 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:49.675 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:49.675 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:49.675 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:49.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:49.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f716c004050 con 0x7f717006dfa0 2024-09-16T05:39:49.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:49.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:49.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.654+0000 7f7162ffd640 10 monclient(hunting): dump: 2024-09-16T05:39:49.676 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:49.676 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:49.676 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.676 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.676 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:49.676 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:49.677 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:49.677 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:49.677 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:49.677 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:49.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1375486799 0 0) 0x7f7170139f70 con 0x7f71700a0810 2024-09-16T05:39:49.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 >> v1:172.21.15.63:6789/0 conn(0x7f7170134930 legacy=0x7f71701d3830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:49.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 >> v1:172.21.15.6:6789/0 conn(0x7f717006dfa0 legacy=0x7f71701cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:49.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:49.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 monclient: found mon.c 2024-09-16T05:39:49.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:49.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:49.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71701396d0 con 0x7f71700a0810 2024-09-16T05:39:49.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:49.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:49.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:49.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 5 monclient: authenticate success, global_id 6023 2024-09-16T05:39:49.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 10 monclient: _renew_subs 2024-09-16T05:39:49.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:49.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7170139120 con 0x7f71700a0810 2024-09-16T05:39:49.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 10 monclient: _renew_subs 2024-09-16T05:39:49.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:49.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f71701cb970 con 0x7f71700a0810 2024-09-16T05:39:49.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71640038b0 con 0x7f71700a0810 2024-09-16T05:39:49.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:49.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7164003c80 con 0x7f71700a0810 2024-09-16T05:39:49.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:49.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:49.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7163fff640 4 set_mon_vals no callback set 2024-09-16T05:39:49.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 10 monclient: dump: 2024-09-16T05:39:49.680 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:49.680 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:49.681 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:49.682 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7158005090 con 0x7f71700a0810 2024-09-16T05:39:49.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7164003ea0 con 0x7f71700a0810 2024-09-16T05:39:49.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(197..197 src has 1..197) ==== 9415+0+0 (unknown 4156505495 0 0) 0x7f71640459e0 con 0x7f71700a0810 2024-09-16T05:39:49.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.658+0000 7f717637a640 10 monclient: get_auth_request con 0x7f7154036a50 auth_method 0 2024-09-16T05:39:49.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.670+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7164015530 con 0x7f71700a0810 2024-09-16T05:39:49.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.670+0000 7f7162ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:49.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.670+0000 7f7162ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:49.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.834+0000 7f71785de640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:49.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.834+0000 7f71785de640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:49.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.834+0000 7f71785de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:49.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:49.834+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f7158005380 con 0x7f71700a0810 2024-09-16T05:39:50.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.406+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 added bucket h12 type host to location {rack=r1} v198) ==== 168+0+0 (unknown 2252066476 0 0) 0x7f71640191e0 con 0x7f71700a0810 2024-09-16T05:39:50.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.406+0000 7f7162ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:50.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.406+0000 7f7162ffd640 10 monclient: _finish_command 2 = system:0 added bucket h12 type host to location {rack=r1} 2024-09-16T05:39:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.510+0000 7f71785de640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:50.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.510+0000 7f71785de640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.510+0000 7f71785de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:50.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.510+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f7158005ac0 con 0x7f71700a0810 2024-09-16T05:39:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.510+0000 7f7162ffd640 1 -- 172.21.15.63:0/4193020163 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h12' already exists v198) ==== 147+0+0 (unknown 2183273013 0 0) 0x7f7164042100 con 0x7f71700a0810 2024-09-16T05:39:50.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.510+0000 7f7162ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.510+0000 7f7162ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h12' already exists 2024-09-16T05:39:50.516 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h12' already exists 2024-09-16T05:39:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.514+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7154036a50 legacy=0x7f7154038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.514+0000 7f71785de640 10 monclient: shutdown 2024-09-16T05:39:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.514+0000 7f71785de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.514+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 >> v1:172.21.15.6:6790/0 conn(0x7f71700a0810 legacy=0x7f71701d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.514+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 shutdown_connections 2024-09-16T05:39:50.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.514+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 >> 172.21.15.63:0/4193020163 conn(0x7f71700a1e90 msgr2=0x7f717009f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.514+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 shutdown_connections 2024-09-16T05:39:50.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.514+0000 7f71785de640 1 -- 172.21.15.63:0/4193020163 wait complete. 2024-09-16T05:39:50.532 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:50.532 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h13 host rack=r1 2024-09-16T05:39:50.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient: get_monmap_and_config 2024-09-16T05:39:50.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient: build_initial_monmap 2024-09-16T05:39:50.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient: monmap: 2024-09-16T05:39:50.655 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:50.655 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:50.656 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:50.654901+0000 2024-09-16T05:39:50.656 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:50.654901+0000 2024-09-16T05:39:50.656 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:50.656 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:50.656 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:50.656 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:50.657 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:50.657 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:50.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 1 Processor -- start 2024-09-16T05:39:50.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 1 -- start start 2024-09-16T05:39:50.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient: init 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient(hunting): picked mon.noname-b con 0x7fdbd00a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient(hunting): picked mon.noname-c con 0x7fdbd00a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient(hunting): picked mon.noname-a con 0x7fdbd00a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbd00a53b0 con 0x7fdbd00a4600 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbd00a0290 con 0x7fdbd00a07a0 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbd00a0dc0 con 0x7fdbd00a3b10 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbd592b640 10 monclient(hunting): authenticate will time out at 1247.431907s 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.650+0000 7fdbce7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fdbd00a07a0 legacy=0x7fdbd013be20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:50.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbceffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdbd00a3b10 0x7fdbd00a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44234/0 (socket says 172.21.15.63:44234) 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcf7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdbd00a4600 0x7fdbd013f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35528/0 (socket says 172.21.15.63:35528) 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbceffd640 1 -- 172.21.15.63:0/4129468729 learned_addr learned my addr 172.21.15.63:0/4129468729 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1318873366 0 0) 0x7fdbd00a53b0 con 0x7fdbd00a4600 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient(hunting): my global_id is 6051 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb8003530 con 0x7fdbd00a4600 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595881916 0 0) 0x7fdbd00a0dc0 con 0x7fdbd00a3b10 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:50.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient(hunting): my global_id is 6026 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbd00a53b0 con 0x7fdbd00a3b10 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2983059776 0 0) 0x7fdbb8003530 con 0x7fdbd00a4600 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 >> v1:172.21.15.6:6790/0 conn(0x7fdbd00a3b10 legacy=0x7fdbd00a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 >> v1:172.21.15.63:6789/0 conn(0x7fdbd00a07a0 legacy=0x7fdbd013be20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 monclient: found mon.noname-a 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd00a10a0 con 0x7fdbd00a4600 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:50.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 5 monclient: authenticate success, global_id 6051 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdbc80019a0 con 0x7fdbd00a4600 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: dump: 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:50.670 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbc8001c00 con 0x7fdbd00a4600 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 1 -- 172.21.15.63:0/4129468729 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdbc8001f90 con 0x7fdbd00a4600 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 4 set_mon_vals no callback set 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbcdffb640 10 monclient: dump: 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:50.671 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 10 monclient: shutdown 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 1 -- 172.21.15.63:0/4129468729 >> v1:172.21.15.6:6789/0 conn(0x7fdbd00a4600 legacy=0x7fdbd013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 1 -- 172.21.15.63:0/4129468729 shutdown_connections 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 1 -- 172.21.15.63:0/4129468729 >> 172.21.15.63:0/4129468729 conn(0x7fdbd0075470 msgr2=0x7fdbd0075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:50.672 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 1 -- 172.21.15.63:0/4129468729 shutdown_connections 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 1 -- 172.21.15.63:0/4129468729 wait complete. 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 10 monclient: build_initial_monmap 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 10 monclient: monmap: 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:50.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 1 Processor -- start 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 1 -- start start 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 10 monclient: init 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.654+0000 7fdbd592b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient(hunting): picked mon.noname-c con 0x7fdbd00a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient(hunting): picked mon.noname-b con 0x7fdbd00a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient(hunting): picked mon.noname-a con 0x7fdbd00a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbd00a0500 con 0x7fdbd00a4600 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbd01272f0 con 0x7fdbd00a3b10 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbd0125a60 con 0x7fdbd00a07a0 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbce7fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fdbd00a3b10 legacy=0x7fdbd01d0320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:50.674 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient(hunting): authenticate will time out at 1247.436119s 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbceffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdbd00a07a0 0x7fdbd0136ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44244/0 (socket says 172.21.15.63:44244) 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbceffd640 1 -- 172.21.15.63:0/961939670 learned_addr learned my addr 172.21.15.63:0/961939670 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900089913 0 0) 0x7fdbd00a10a0 con 0x7fdbd00a4600 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): my global_id is 6054 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbbc003530 con 0x7fdbd00a4600 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1180496986 0 0) 0x7fdbd0125a60 con 0x7fdbd00a07a0 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:50.675 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): my global_id is 6029 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbd00a10a0 con 0x7fdbd00a07a0 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2378375727 0 0) 0x7fdbd00a10a0 con 0x7fdbd00a07a0 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbd0125a60 con 0x7fdbd00a07a0 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdbc00019a0 con 0x7fdbd00a07a0 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): dump: 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.676 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 754595094 0 0) 0x7fdbbc003530 con 0x7fdbd00a4600 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbd00a10a0 con 0x7fdbd00a4600 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdbc80018a0 con 0x7fdbd00a4600 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient(hunting): dump: 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:50.677 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1123774522 0 0) 0x7fdbd0125a60 con 0x7fdbd00a07a0 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 >> v1:172.21.15.63:6789/0 conn(0x7fdbd00a3b10 legacy=0x7fdbd01d0320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 >> v1:172.21.15.6:6789/0 conn(0x7fdbd00a4600 legacy=0x7fdbd01d3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 monclient: found mon.c 2024-09-16T05:39:50.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd00a8830 con 0x7fdbd00a07a0 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient: _finish_auth 0 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 5 monclient: authenticate success, global_id 6029 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient: _renew_subs 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd01275d0 con 0x7fdbd00a07a0 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient: _renew_subs 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbd01377a0 con 0x7fdbd00a07a0 2024-09-16T05:39:50.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbc00047f0 con 0x7fdbd00a07a0 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdbc0004b40 con 0x7fdbd00a07a0 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbcd7fa640 4 set_mon_vals no callback set 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 10 monclient: dump: 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:50.680 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbb0005090 con 0x7fdbd00a07a0 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdbc000f8e0 con 0x7fdbd00a07a0 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(198..198 src has 1..198) ==== 9628+0+0 (unknown 1318538917 0 0) 0x7fdbc0045780 con 0x7fdbd00a07a0 2024-09-16T05:39:50.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.658+0000 7fdbce7fc640 10 monclient: get_auth_request con 0x7fdbbc036b10 auth_method 0 2024-09-16T05:39:50.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.670+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fdbc0015270 con 0x7fdbd00a07a0 2024-09-16T05:39:50.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.670+0000 7fdbb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:50.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.670+0000 7fdbb7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:50.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.838+0000 7fdbd592b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:50.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.838+0000 7fdbd592b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:50.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.838+0000 7fdbd592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:50.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:50.838+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fdbb0005380 con 0x7fdbd00a07a0 2024-09-16T05:39:51.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.046+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 added bucket h13 type host to location {rack=r1} v199) ==== 168+0+0 (unknown 4197010257 0 0) 0x7fdbc0018f20 con 0x7fdbd00a07a0 2024-09-16T05:39:51.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.046+0000 7fdbb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:51.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.046+0000 7fdbb7fff640 10 monclient: _finish_command 2 = system:0 added bucket h13 type host to location {rack=r1} 2024-09-16T05:39:51.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.146+0000 7fdbd592b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:51.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.146+0000 7fdbd592b640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:51.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.146+0000 7fdbd592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:51.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.146+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fdbb0005ac0 con 0x7fdbd00a07a0 2024-09-16T05:39:51.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.150+0000 7fdbb7fff640 1 -- 172.21.15.63:0/961939670 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h13' already exists v199) ==== 147+0+0 (unknown 2308178904 0 0) 0x7fdbc0040520 con 0x7fdbd00a07a0 2024-09-16T05:39:51.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.150+0000 7fdbb7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:51.155 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.150+0000 7fdbb7fff640 10 monclient: _finish_command 3 = system:0 bucket 'h13' already exists 2024-09-16T05:39:51.155 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h13' already exists 2024-09-16T05:39:51.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.154+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 >> v1:172.21.15.63:6812/3643207211 conn(0x7fdbbc036b10 legacy=0x7fdbbc038fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:51.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.154+0000 7fdbd592b640 10 monclient: shutdown 2024-09-16T05:39:51.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.154+0000 7fdbd592b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:51.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.154+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 >> v1:172.21.15.6:6790/0 conn(0x7fdbd00a07a0 legacy=0x7fdbd0136ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:51.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.154+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 shutdown_connections 2024-09-16T05:39:51.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.154+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 >> 172.21.15.63:0/961939670 conn(0x7fdbd0075470 msgr2=0x7fdbd00a8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:51.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.154+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 shutdown_connections 2024-09-16T05:39:51.158 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.154+0000 7fdbd592b640 1 -- 172.21.15.63:0/961939670 wait complete. 2024-09-16T05:39:51.172 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:51.172 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h14 host rack=r1 2024-09-16T05:39:51.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient: get_monmap_and_config 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient: build_initial_monmap 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient: monmap: 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:51.296685+0000 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:51.296685+0000 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:51.297 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:51.298 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:51.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 1 Processor -- start 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 1 -- start start 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient: init 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5600a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5600a4d90 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5600a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5600a51e0 con 0x7fa5600a4790 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5600a0d60 con 0x7fa5600a4d90 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5600a1040 con 0x7fa5600a3ca0 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:51.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient(hunting): authenticate will time out at 1248.073636s 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa5655d8640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa5600a4d90 legacy=0x7fa56013bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa565dd9640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa5600a3ca0 0x7fa5600a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44248/0 (socket says 172.21.15.63:44248) 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa565dd9640 1 -- 172.21.15.63:0/3708663136 learned_addr learned my addr 172.21.15.63:0/3708663136 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3986153010 0 0) 0x7fa5600a1040 con 0x7fa5600a3ca0 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient(hunting): _negotiate 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient(hunting): my global_id is 6032 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa54c003530 con 0x7fa5600a3ca0 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293503089 0 0) 0x7fa5600a51e0 con 0x7fa5600a4790 2024-09-16T05:39:51.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient(hunting): _negotiate 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient(hunting): my global_id is 6057 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5600a1040 con 0x7fa5600a4790 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3576883362 0 0) 0x7fa54c003530 con 0x7fa5600a3ca0 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 >> v1:172.21.15.63:6789/0 conn(0x7fa5600a4d90 legacy=0x7fa56013bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 >> v1:172.21.15.6:6789/0 conn(0x7fa5600a4790 legacy=0x7fa56013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: _finish_hunting 0 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 monclient: found mon.noname-b 2024-09-16T05:39:51.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5600a0810 con 0x7fa5600a3ca0 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: _finish_auth 0 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: _check_auth_tickets 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa5580019a0 con 0x7fa5600a3ca0 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: dump: 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:51.310 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 5 monclient: authenticate success, global_id 6032 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa558001bd0 con 0x7fa5600a3ca0 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:51.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 1 -- 172.21.15.63:0/3708663136 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa558001fe0 con 0x7fa5600a3ca0 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa564dd7640 10 monclient: dump: 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:51.312 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 4 set_mon_vals no callback set 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 10 monclient: shutdown 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.294+0000 7fa56803d640 1 -- 172.21.15.63:0/3708663136 >> v1:172.21.15.6:6790/0 conn(0x7fa5600a3ca0 legacy=0x7fa5600a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- 172.21.15.63:0/3708663136 shutdown_connections 2024-09-16T05:39:51.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- 172.21.15.63:0/3708663136 >> 172.21.15.63:0/3708663136 conn(0x7fa560075470 msgr2=0x7fa560075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:51.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- 172.21.15.63:0/3708663136 shutdown_connections 2024-09-16T05:39:51.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- 172.21.15.63:0/3708663136 wait complete. 2024-09-16T05:39:51.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: build_initial_monmap 2024-09-16T05:39:51.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: monmap: 2024-09-16T05:39:51.314 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:51.314 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 Processor -- start 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- start start 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: init 2024-09-16T05:39:51.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5600a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5600a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5600a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa560138a50 con 0x7fa5600a3ca0 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5600726f0 con 0x7fa5600a4790 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa560138770 con 0x7fa5600a4d90 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5655d8640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fa5600a4790 legacy=0x7fa5601d0240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient(hunting): authenticate will time out at 1248.077621s 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa565dd9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa5600a3ca0 0x7fa5601c7c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35556/0 (socket says 172.21.15.63:35556) 2024-09-16T05:39:51.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa565dd9640 1 -- 172.21.15.63:0/704381245 learned_addr learned my addr 172.21.15.63:0/704381245 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5665da640 1 --1- 172.21.15.63:0/704381245 >> v1:172.21.15.6:6790/0 conn(0x7fa5600a4d90 0x7fa5601d3970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44264/0 (socket says 172.21.15.63:44264) 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 810144260 0 0) 0x7fa560138a50 con 0x7fa5600a3ca0 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): my global_id is 6060 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa544003530 con 0x7fa5600a3ca0 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3953076285 0 0) 0x7fa560138770 con 0x7fa5600a4d90 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): my global_id is 6035 2024-09-16T05:39:51.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa560138a50 con 0x7fa5600a4d90 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1324022517 0 0) 0x7fa544003530 con 0x7fa5600a3ca0 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa560138770 con 0x7fa5600a3ca0 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa558001f20 con 0x7fa5600a3ca0 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): dump: 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:51.318 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3220431694 0 0) 0x7fa560138a50 con 0x7fa5600a4d90 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa544003530 con 0x7fa5600a4d90 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa5500019a0 con 0x7fa5600a4d90 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient(hunting): dump: 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:51.319 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4099903759 0 0) 0x7fa560138770 con 0x7fa5600a3ca0 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 >> v1:172.21.15.6:6790/0 conn(0x7fa5600a4d90 legacy=0x7fa5601d3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 >> v1:172.21.15.63:6789/0 conn(0x7fa5600a4790 legacy=0x7fa5601d0240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:51.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 monclient: found mon.a 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5601377d0 con 0x7fa5600a3ca0 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 5 monclient: authenticate success, global_id 6060 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: _renew_subs 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa5601371c0 con 0x7fa5600a3ca0 2024-09-16T05:39:51.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: _renew_subs 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5601c8560 con 0x7fa5600a3ca0 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa558001bd0 con 0x7fa5600a3ca0 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa558005120 con 0x7fa5600a3ca0 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa557fff640 4 set_mon_vals no callback set 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.298+0000 7fa5567fc640 10 monclient: dump: 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.322 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:51.323 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:51.323 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:51.323 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:51.323 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:51.323 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:51.323 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:51.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.302+0000 7fa56803d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:51.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.302+0000 7fa56803d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:51.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.302+0000 7fa56803d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:51.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.302+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa548005090 con 0x7fa5600a3ca0 2024-09-16T05:39:51.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.302+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa55800fec0 con 0x7fa5600a3ca0 2024-09-16T05:39:51.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.302+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(199..199 src has 1..199) ==== 9841+0+0 (unknown 3673208586 0 0) 0x7fa558045dc0 con 0x7fa5600a3ca0 2024-09-16T05:39:51.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.302+0000 7fa5655d8640 10 monclient: get_auth_request con 0x7fa544036990 auth_method 0 2024-09-16T05:39:51.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.314+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fa558015820 con 0x7fa5600a3ca0 2024-09-16T05:39:51.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.314+0000 7fa5567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:51.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.314+0000 7fa5567fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:51.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.478+0000 7fa56803d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:51.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.478+0000 7fa56803d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:51.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.478+0000 7fa56803d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:51.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:51.478+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fa548005380 con 0x7fa5600a3ca0 2024-09-16T05:39:52.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.422+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 added bucket h14 type host to location {rack=r1} v200) ==== 168+0+0 (unknown 1085713285 0 0) 0x7fa558015120 con 0x7fa5600a3ca0 2024-09-16T05:39:52.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.422+0000 7fa5567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:52.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.422+0000 7fa5567fc640 10 monclient: _finish_command 2 = system:0 added bucket h14 type host to location {rack=r1} 2024-09-16T05:39:52.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.526+0000 7fa56803d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:52.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.526+0000 7fa56803d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:52.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.526+0000 7fa56803d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:52.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.526+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fa548005ac0 con 0x7fa5600a3ca0 2024-09-16T05:39:52.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.526+0000 7fa5567fc640 1 -- 172.21.15.63:0/704381245 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h14' already exists v200) ==== 147+0+0 (unknown 517624319 0 0) 0x7fa55801dcd0 con 0x7fa5600a3ca0 2024-09-16T05:39:52.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.526+0000 7fa5567fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:52.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.526+0000 7fa5567fc640 10 monclient: _finish_command 3 = system:0 bucket 'h14' already exists 2024-09-16T05:39:52.531 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h14' already exists 2024-09-16T05:39:52.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.530+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa544036990 legacy=0x7fa544038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:52.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.530+0000 7fa56803d640 10 monclient: shutdown 2024-09-16T05:39:52.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.530+0000 7fa56803d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:52.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.530+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 >> v1:172.21.15.6:6789/0 conn(0x7fa5600a3ca0 legacy=0x7fa5601c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:52.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.530+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 shutdown_connections 2024-09-16T05:39:52.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.530+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 >> 172.21.15.63:0/704381245 conn(0x7fa560075470 msgr2=0x7fa5600a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:52.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.530+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 shutdown_connections 2024-09-16T05:39:52.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.530+0000 7fa56803d640 1 -- 172.21.15.63:0/704381245 wait complete. 2024-09-16T05:39:52.548 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:52.548 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h15 host rack=r1 2024-09-16T05:39:52.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient: get_monmap_and_config 2024-09-16T05:39:52.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient: build_initial_monmap 2024-09-16T05:39:52.673 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient: monmap: 2024-09-16T05:39:52.673 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:52.673575+0000 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:52.673575+0000 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:52.674 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:52.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 1 Processor -- start 2024-09-16T05:39:52.677 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 1 -- start start 2024-09-16T05:39:52.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient: init 2024-09-16T05:39:52.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:52.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:52.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient(hunting): picked mon.noname-a con 0x7f26400a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:52.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient(hunting): picked mon.noname-b con 0x7f26400a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:52.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient(hunting): picked mon.noname-c con 0x7f26400a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:52.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26400a53b0 con 0x7f26400a3b10 2024-09-16T05:39:52.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26400a0290 con 0x7f26400a4600 2024-09-16T05:39:52.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26400a0dc0 con 0x7f26400a07a0 2024-09-16T05:39:52.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:52.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2645366640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f26400a4600 legacy=0x7f264013f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:52.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 10 monclient(hunting): authenticate will time out at 1249.450549s 2024-09-16T05:39:52.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f26400a07a0 0x7f264013be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44280/0 (socket says 172.21.15.63:44280) 2024-09-16T05:39:52.680 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2644b65640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f26400a3b10 0x7f26400a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35570/0 (socket says 172.21.15.63:35570) 2024-09-16T05:39:52.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263ffff640 1 -- 172.21.15.63:0/2393165331 learned_addr learned my addr 172.21.15.63:0/2393165331 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:52.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3998717963 0 0) 0x7f26400a53b0 con 0x7f26400a3b10 2024-09-16T05:39:52.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:52.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:52.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:52.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient(hunting): my global_id is 6063 2024-09-16T05:39:52.681 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2628003530 con 0x7f26400a3b10 2024-09-16T05:39:52.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993111922 0 0) 0x7f26400a0dc0 con 0x7f26400a07a0 2024-09-16T05:39:52.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:52.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:52.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:52.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient(hunting): my global_id is 6038 2024-09-16T05:39:52.682 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26400a53b0 con 0x7f26400a07a0 2024-09-16T05:39:52.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1576537085 0 0) 0x7f2628003530 con 0x7f26400a3b10 2024-09-16T05:39:52.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 >> v1:172.21.15.6:6790/0 conn(0x7f26400a07a0 legacy=0x7f264013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:52.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 >> v1:172.21.15.63:6789/0 conn(0x7f26400a4600 legacy=0x7f264013f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:52.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:52.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 monclient: found mon.noname-a 2024-09-16T05:39:52.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:52.683 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:52.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26400a10a0 con 0x7f26400a3b10 2024-09-16T05:39:52.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:52.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:52.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:52.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2630001620 con 0x7f26400a3b10 2024-09-16T05:39:52.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:52.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:52.685 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f263f7fe640 10 monclient: dump: 2024-09-16T05:39:52.685 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:52.685 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:52.685 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.685 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.685 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:52.686 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:52.686 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:52.686 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:52.686 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:52.686 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:52.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.670+0000 7f2646dc9640 5 monclient: authenticate success, global_id 6063 2024-09-16T05:39:52.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:52.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2630001880 con 0x7f26400a3b10 2024-09-16T05:39:52.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:52.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263f7fe640 1 -- 172.21.15.63:0/2393165331 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2630001bd0 con 0x7f26400a3b10 2024-09-16T05:39:52.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:52.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:52.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263f7fe640 10 monclient: dump: 2024-09-16T05:39:52.687 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:52.688 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:52.688 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.688 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.688 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:52.688 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:52.688 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:52.688 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:52.689 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:52.689 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:52.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:52.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 4 set_mon_vals no callback set 2024-09-16T05:39:52.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient: shutdown 2024-09-16T05:39:52.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:52.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- 172.21.15.63:0/2393165331 >> v1:172.21.15.6:6789/0 conn(0x7f26400a3b10 legacy=0x7f26400a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:52.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- 172.21.15.63:0/2393165331 shutdown_connections 2024-09-16T05:39:52.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- 172.21.15.63:0/2393165331 >> 172.21.15.63:0/2393165331 conn(0x7f2640075470 msgr2=0x7f2640075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:52.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- 172.21.15.63:0/2393165331 shutdown_connections 2024-09-16T05:39:52.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- 172.21.15.63:0/2393165331 wait complete. 2024-09-16T05:39:52.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient: build_initial_monmap 2024-09-16T05:39:52.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient: monmap: 2024-09-16T05:39:52.699 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:52.699 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:52.700 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:52.700 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:52.700 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:52.700 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:52.700 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:52.701 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:52.701 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:52.701 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:52.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 Processor -- start 2024-09-16T05:39:52.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- start start 2024-09-16T05:39:52.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient: init 2024-09-16T05:39:52.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:52.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:52.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient(hunting): picked mon.noname-c con 0x7f26400a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:52.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient(hunting): picked mon.noname-b con 0x7f26400a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:52.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient(hunting): picked mon.noname-a con 0x7f26400a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:52.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2640123110 con 0x7f26400a4600 2024-09-16T05:39:52.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2640121cc0 con 0x7f26400a3b10 2024-09-16T05:39:52.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f26400a3b10 legacy=0x7f2640135d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:52.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2640121f40 con 0x7f26400a07a0 2024-09-16T05:39:52.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:52.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2646dc9640 10 monclient(hunting): authenticate will time out at 1249.454550s 2024-09-16T05:39:52.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2644b65640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f26400a07a0 0x7f26401cb080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44282/0 (socket says 172.21.15.63:44282) 2024-09-16T05:39:52.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f2644b65640 1 -- 172.21.15.63:0/897140994 learned_addr learned my addr 172.21.15.63:0/897140994 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60090610 0 0) 0x7f2640121f40 con 0x7f26400a07a0 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): my global_id is 6041 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f262c003530 con 0x7f26400a07a0 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 919608088 0 0) 0x7f2640123110 con 0x7f26400a4600 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): my global_id is 6066 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2640121f40 con 0x7f26400a4600 2024-09-16T05:39:52.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 487607621 0 0) 0x7f262c003530 con 0x7f26400a07a0 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2640123110 con 0x7f26400a07a0 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f26300047f0 con 0x7f26400a07a0 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): dump: 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:52.705 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1687792914 0 0) 0x7f2640121f40 con 0x7f26400a4600 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f262c003530 con 0x7f26400a4600 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2638004340 con 0x7f26400a4600 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.674+0000 7f263d7fa640 10 monclient(hunting): dump: 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.706 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1071085113 0 0) 0x7f2640123110 con 0x7f26400a07a0 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 >> v1:172.21.15.63:6789/0 conn(0x7f26400a3b10 legacy=0x7f2640135d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 >> v1:172.21.15.6:6789/0 conn(0x7f26400a4600 legacy=0x7f26401d33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 monclient: found mon.c 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:52.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2640136a60 con 0x7f26400a07a0 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 5 monclient: authenticate success, global_id 6041 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 10 monclient: _renew_subs 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2640122e30 con 0x7f26400a07a0 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 10 monclient: _renew_subs 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f26401365f0 con 0x7f26400a07a0 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26300017e0 con 0x7f26400a07a0 2024-09-16T05:39:52.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263effd640 4 set_mon_vals no callback set 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2630004de0 con 0x7f26400a07a0 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 10 monclient: dump: 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:52.709 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2620005090 con 0x7f26400a07a0 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2630005040 con 0x7f26400a07a0 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(200..200 src has 1..200) ==== 10054+0+0 (unknown 480758046 0 0) 0x7f2630045c30 con 0x7f26400a07a0 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.678+0000 7f263ffff640 10 monclient: get_auth_request con 0x7f262c0369d0 auth_method 0 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.690+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f26300155c0 con 0x7f26400a07a0 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.690+0000 7f263d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:52.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.690+0000 7f263d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:52.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.854+0000 7f2646dc9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:52.859 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.854+0000 7f2646dc9640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:52.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.854+0000 7f2646dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:52.860 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:52.854+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f2620005380 con 0x7f26400a07a0 2024-09-16T05:39:53.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.442+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 added bucket h15 type host to location {rack=r1} v201) ==== 168+0+0 (unknown 1017392760 0 0) 0x7f2630014ec0 con 0x7f26400a07a0 2024-09-16T05:39:53.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.442+0000 7f263d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:53.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.442+0000 7f263d7fa640 10 monclient: _finish_command 2 = system:0 added bucket h15 type host to location {rack=r1} 2024-09-16T05:39:53.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.546+0000 7f2646dc9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:53.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.546+0000 7f2646dc9640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.546+0000 7f2646dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:53.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.546+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f2620005ac0 con 0x7f26400a07a0 2024-09-16T05:39:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.546+0000 7f263d7fa640 1 -- 172.21.15.63:0/897140994 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h15' already exists v201) ==== 147+0+0 (unknown 359417874 0 0) 0x7f263001da70 con 0x7f26400a07a0 2024-09-16T05:39:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.546+0000 7f263d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:53.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.546+0000 7f263d7fa640 10 monclient: _finish_command 3 = system:0 bucket 'h15' already exists 2024-09-16T05:39:53.551 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h15' already exists 2024-09-16T05:39:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.550+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 >> v1:172.21.15.63:6812/3643207211 conn(0x7f262c0369d0 legacy=0x7f262c038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.550+0000 7f2646dc9640 10 monclient: shutdown 2024-09-16T05:39:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.550+0000 7f2646dc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.550+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 >> v1:172.21.15.6:6790/0 conn(0x7f26400a07a0 legacy=0x7f26401cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.550+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 shutdown_connections 2024-09-16T05:39:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.550+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 >> 172.21.15.63:0/897140994 conn(0x7f2640075470 msgr2=0x7f26401beb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.550+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 shutdown_connections 2024-09-16T05:39:53.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.550+0000 7f2646dc9640 1 -- 172.21.15.63:0/897140994 wait complete. 2024-09-16T05:39:53.568 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:53.568 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h16 host rack=r1 2024-09-16T05:39:53.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient: get_monmap_and_config 2024-09-16T05:39:53.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient: build_initial_monmap 2024-09-16T05:39:53.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient: monmap: 2024-09-16T05:39:53.699 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:53.699 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:53.699 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:53.693699+0000 2024-09-16T05:39:53.699 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:53.693699+0000 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 1 Processor -- start 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 1 -- start start 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient: init 2024-09-16T05:39:53.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient(hunting): picked mon.noname-b con 0x7f2034134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient(hunting): picked mon.noname-a con 0x7f20340a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient(hunting): picked mon.noname-c con 0x7f203406dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2034135420 con 0x7f20340a0810 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203409e830 con 0x7f203406dfa0 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20340a0dc0 con 0x7f2034134930 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:53.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f203ae39640 10 monclient(hunting): authenticate will time out at 1250.470810s 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20393d6640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f203406dfa0 legacy=0x7f203413cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f2038bd5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2034134930 0x7f2034134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44286/0 (socket says 172.21.15.63:44286) 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f2038bd5640 1 -- 172.21.15.63:0/3448636964 learned_addr learned my addr 172.21.15.63:0/3448636964 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f2033fff640 1 --1- 172.21.15.63:0/3448636964 >> v1:172.21.15.6:6789/0 conn(0x7f20340a0810 0x7f2034139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35598/0 (socket says 172.21.15.63:35598) 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516563969 0 0) 0x7f20340a0dc0 con 0x7f2034134930 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 10 monclient(hunting): my global_id is 6044 2024-09-16T05:39:53.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f201c003530 con 0x7f2034134930 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778591106 0 0) 0x7f2034135420 con 0x7f20340a0810 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 10 monclient(hunting): _negotiate 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 10 monclient(hunting): my global_id is 6069 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.690+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20340a0dc0 con 0x7f20340a0810 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3268537392 0 0) 0x7f201c003530 con 0x7f2034134930 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 >> v1:172.21.15.63:6789/0 conn(0x7f203406dfa0 legacy=0x7f203413cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 >> v1:172.21.15.6:6789/0 conn(0x7f20340a0810 legacy=0x7f2034139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: _finish_hunting 0 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 1 monclient: found mon.noname-b 2024-09-16T05:39:53.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:53.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:53.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20340a2500 con 0x7f2034134930 2024-09-16T05:39:53.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: _finish_auth 0 2024-09-16T05:39:53.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: _check_auth_tickets 2024-09-16T05:39:53.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:53.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2024001620 con 0x7f2034134930 2024-09-16T05:39:53.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:53.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: dump: 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 5 monclient: authenticate success, global_id 6044 2024-09-16T05:39:53.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2024001880 con 0x7f2034134930 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 1 -- 172.21.15.63:0/3448636964 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2024001c50 con 0x7f2034134930 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 4 set_mon_vals no callback set 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20337fe640 10 monclient: dump: 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:53.706 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient: shutdown 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- 172.21.15.63:0/3448636964 >> v1:172.21.15.6:6790/0 conn(0x7f2034134930 legacy=0x7f2034134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- 172.21.15.63:0/3448636964 shutdown_connections 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- 172.21.15.63:0/3448636964 >> 172.21.15.63:0/3448636964 conn(0x7f20340a1e90 msgr2=0x7f2034075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- 172.21.15.63:0/3448636964 shutdown_connections 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- 172.21.15.63:0/3448636964 wait complete. 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient: build_initial_monmap 2024-09-16T05:39:53.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient: monmap: 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 Processor -- start 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- start start 2024-09-16T05:39:53.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient: init 2024-09-16T05:39:53.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:53.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:53.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient(hunting): picked mon.noname-a con 0x7f203406dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient(hunting): picked mon.noname-c con 0x7f20340a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient(hunting): picked mon.noname-b con 0x7f2034134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2034137d30 con 0x7f203406dfa0 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2034138010 con 0x7f2034134930 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f20393d6640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2034134930 legacy=0x7f20341d1760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203413e3c0 con 0x7f20340a0810 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f203ae39640 10 monclient(hunting): authenticate will time out at 1250.475447s 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f2038bd5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f203406dfa0 0x7f2034134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:35602/0 (socket says 172.21.15.63:35602) 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f2038bd5640 1 -- 172.21.15.63:0/335569920 learned_addr learned my addr 172.21.15.63:0/335569920 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.694+0000 7f2033fff640 1 --1- 172.21.15.63:0/335569920 >> v1:172.21.15.6:6790/0 conn(0x7f20340a0810 0x7f2034135420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:44296/0 (socket says 172.21.15.63:44296) 2024-09-16T05:39:53.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 373626099 0 0) 0x7f2034137d30 con 0x7f203406dfa0 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): my global_id is 6072 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2020003530 con 0x7f203406dfa0 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 634233788 0 0) 0x7f203413e3c0 con 0x7f20340a0810 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): _negotiate 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:53.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): my global_id is 6047 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2034137d30 con 0x7f20340a0810 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1747422330 0 0) 0x7f2020003530 con 0x7f203406dfa0 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f203413e3c0 con 0x7f203406dfa0 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2024001560 con 0x7f203406dfa0 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): dump: 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:53.715 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1563108176 0 0) 0x7f2034137d30 con 0x7f20340a0810 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2020003530 con 0x7f20340a0810 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20280018d0 con 0x7f20340a0810 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:53.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient(hunting): dump: 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2427924613 0 0) 0x7f203413e3c0 con 0x7f203406dfa0 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 >> v1:172.21.15.6:6790/0 conn(0x7f20340a0810 legacy=0x7f2034135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:53.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 >> v1:172.21.15.63:6789/0 conn(0x7f2034134930 legacy=0x7f20341d1760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient: _finish_hunting 0 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 monclient: found mon.a 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2034071eb0 con 0x7f203406dfa0 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient: _finish_auth 0 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient: _check_auth_tickets 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 5 monclient: authenticate success, global_id 6072 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 10 monclient: _renew_subs 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f203409c0f0 con 0x7f203406dfa0 2024-09-16T05:39:53.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 10 monclient: _renew_subs 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2034136600 con 0x7f203406dfa0 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2024001880 con 0x7f203406dfa0 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20327fc640 4 set_mon_vals no callback set 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2024004da0 con 0x7f203406dfa0 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 10 monclient: dump: 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:53.719 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2014005090 con 0x7f203406dfa0 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2024004fc0 con 0x7f203406dfa0 2024-09-16T05:39:53.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(201..201 src has 1..201) ==== 10267+0+0 (unknown 1357164525 0 0) 0x7f2024045c60 con 0x7f203406dfa0 2024-09-16T05:39:53.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.698+0000 7f2033fff640 10 monclient: get_auth_request con 0x7f2020036990 auth_method 0 2024-09-16T05:39:53.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.710+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2024015570 con 0x7f203406dfa0 2024-09-16T05:39:53.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.710+0000 7f20317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:53.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.710+0000 7f20317fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:53.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.878+0000 7f203ae39640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:53.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.878+0000 7f203ae39640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:53.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.878+0000 7f203ae39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:53.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:53.878+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f2014005380 con 0x7f203406dfa0 2024-09-16T05:39:54.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.438+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 added bucket h16 type host to location {rack=r1} v202) ==== 168+0+0 (unknown 3096682623 0 0) 0x7f2024014e70 con 0x7f203406dfa0 2024-09-16T05:39:54.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.438+0000 7f20317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:54.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.438+0000 7f20317fa640 10 monclient: _finish_command 2 = system:0 added bucket h16 type host to location {rack=r1} 2024-09-16T05:39:54.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:54.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:54.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:54.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f2014005ac0 con 0x7f203406dfa0 2024-09-16T05:39:54.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f20317fa640 1 -- 172.21.15.63:0/335569920 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h16' already exists v202) ==== 147+0+0 (unknown 162947621 0 0) 0x7f2024040780 con 0x7f203406dfa0 2024-09-16T05:39:54.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f20317fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:54.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f20317fa640 10 monclient: _finish_command 3 = system:0 bucket 'h16' already exists 2024-09-16T05:39:54.546 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h16' already exists 2024-09-16T05:39:54.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2020036990 legacy=0x7f2020038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:54.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 10 monclient: shutdown 2024-09-16T05:39:54.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:54.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 >> v1:172.21.15.6:6789/0 conn(0x7f203406dfa0 legacy=0x7f2034134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:54.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 shutdown_connections 2024-09-16T05:39:54.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 >> 172.21.15.63:0/335569920 conn(0x7f20340a1e90 msgr2=0x7f2034071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:54.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 shutdown_connections 2024-09-16T05:39:54.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.542+0000 7f203ae39640 1 -- 172.21.15.63:0/335569920 wait complete. 2024-09-16T05:39:54.563 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:54.563 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h17 host rack=r1 2024-09-16T05:39:54.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient: get_monmap_and_config 2024-09-16T05:39:54.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient: build_initial_monmap 2024-09-16T05:39:54.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient: monmap: 2024-09-16T05:39:54.689 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:54.689 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:54.689 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:54.688722+0000 2024-09-16T05:39:54.689 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:54.688722+0000 2024-09-16T05:39:54.690 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:54.690 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:54.690 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:54.690 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:54.690 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:54.690 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 1 Processor -- start 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 1 -- start start 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient: init 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient(hunting): picked mon.noname-c con 0x7f32900a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient(hunting): picked mon.noname-a con 0x7f32900a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient(hunting): picked mon.noname-b con 0x7f32900a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:54.693 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32900a53b0 con 0x7f32900a07a0 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32900a0290 con 0x7f32900a3b10 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32900a0dc0 con 0x7f32900a4600 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient(hunting): authenticate will time out at 1251.465574s 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328e59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f32900a3b10 legacy=0x7f32900a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f32900a07a0 0x7f329013be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37952/0 (socket says 172.21.15.63:37952) 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328ed9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f32900a4600 0x7f329013f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47106/0 (socket says 172.21.15.63:47106) 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328dd9b640 1 -- 172.21.15.63:0/2375304836 learned_addr learned my addr 172.21.15.63:0/2375304836 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310518925 0 0) 0x7f32900a0dc0 con 0x7f32900a4600 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:54.694 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient(hunting): my global_id is 6050 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3278003530 con 0x7f32900a4600 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3528488147 0 0) 0x7f32900a53b0 con 0x7f32900a07a0 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient(hunting): my global_id is 6075 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32900a0dc0 con 0x7f32900a07a0 2024-09-16T05:39:54.695 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1395929458 0 0) 0x7f32900a0dc0 con 0x7f32900a07a0 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 >> v1:172.21.15.6:6790/0 conn(0x7f32900a4600 legacy=0x7f329013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 >> v1:172.21.15.63:6789/0 conn(0x7f32900a3b10 legacy=0x7f32900a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 monclient: found mon.noname-a 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32900a10a0 con 0x7f32900a07a0 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: _finish_auth 0 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:54.696 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3284001600 con 0x7f32900a07a0 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: dump: 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:54.697 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 5 monclient: authenticate success, global_id 6075 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3284001860 con 0x7f32900a07a0 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 1 -- 172.21.15.63:0/2375304836 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3284001c30 con 0x7f32900a07a0 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f328d59a640 10 monclient: dump: 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.698 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 4 set_mon_vals no callback set 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 10 monclient: shutdown 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 1 -- 172.21.15.63:0/2375304836 >> v1:172.21.15.6:6789/0 conn(0x7f32900a07a0 legacy=0x7f329013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 1 -- 172.21.15.63:0/2375304836 shutdown_connections 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.686+0000 7f3294884640 1 -- 172.21.15.63:0/2375304836 >> 172.21.15.63:0/2375304836 conn(0x7f3290075470 msgr2=0x7f3290075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:54.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 -- 172.21.15.63:0/2375304836 shutdown_connections 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 -- 172.21.15.63:0/2375304836 wait complete. 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: build_initial_monmap 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: monmap: 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:54.700 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:54.701 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:54.701 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:54.701 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:54.701 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:54.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 Processor -- start 2024-09-16T05:39:54.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 -- start start 2024-09-16T05:39:54.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: init 2024-09-16T05:39:54.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:54.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:54.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient(hunting): picked mon.noname-b con 0x7f32900a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient(hunting): picked mon.noname-a con 0x7f32900a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient(hunting): picked mon.noname-c con 0x7f32900a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3290127480 con 0x7f32900a3b10 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f328e59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f32900a07a0 legacy=0x7f32901cae70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3290125910 con 0x7f32900a07a0 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32900a8390 con 0x7f32900a4600 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient(hunting): authenticate will time out at 1251.469621s 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f328dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f32900a3b10 0x7f3290136d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37966/0 (socket says 172.21.15.63:37966) 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f328dd9b640 1 -- 172.21.15.63:0/871856713 learned_addr learned my addr 172.21.15.63:0/871856713 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:54.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4238452200 0 0) 0x7f3290127480 con 0x7f32900a3b10 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): my global_id is 6078 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3270003530 con 0x7f32900a3b10 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1009910265 0 0) 0x7f32900a8390 con 0x7f32900a4600 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): my global_id is 6053 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3290127480 con 0x7f32900a4600 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2141810490 0 0) 0x7f3270003530 con 0x7f32900a3b10 2024-09-16T05:39:54.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32900a8390 con 0x7f32900a3b10 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3284001500 con 0x7f32900a3b10 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): dump: 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:54.707 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2412798465 0 0) 0x7f3290127480 con 0x7f32900a4600 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3270003530 con 0x7f32900a4600 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3280001970 con 0x7f32900a4600 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient(hunting): dump: 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.708 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3123156226 0 0) 0x7f32900a8390 con 0x7f32900a3b10 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 >> v1:172.21.15.6:6790/0 conn(0x7f32900a4600 legacy=0x7f32901d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 >> v1:172.21.15.63:6789/0 conn(0x7f32900a07a0 legacy=0x7f32901cae70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 monclient: found mon.a 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:54.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32901271a0 con 0x7f32900a3b10 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient: _finish_auth 0 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 5 monclient: authenticate success, global_id 6078 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: _renew_subs 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3290137480 con 0x7f32900a3b10 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: _renew_subs 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f32901d4ae0 con 0x7f32900a3b10 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3284001860 con 0x7f32900a3b10 2024-09-16T05:39:54.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:54.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3284004da0 con 0x7f32900a3b10 2024-09-16T05:39:54.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:54.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327ffff640 4 set_mon_vals no callback set 2024-09-16T05:39:54.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:54.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.690+0000 7f327effd640 10 monclient: dump: 2024-09-16T05:39:54.711 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:54.711 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:54.711 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.694+0000 7f3294884640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.694+0000 7f3294884640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.694+0000 7f3294884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.694+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3274005090 con 0x7f32900a3b10 2024-09-16T05:39:54.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.694+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f328400fb40 con 0x7f32900a3b10 2024-09-16T05:39:54.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.694+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(202..202 src has 1..202) ==== 10736+0+0 (unknown 1593764070 0 0) 0x7f3284045e10 con 0x7f32900a3b10 2024-09-16T05:39:54.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.694+0000 7f328e59c640 10 monclient: get_auth_request con 0x7f32700369e0 auth_method 0 2024-09-16T05:39:54.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.706+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f32840154f0 con 0x7f32900a3b10 2024-09-16T05:39:54.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.706+0000 7f327effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:54.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.706+0000 7f327effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:54.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.874+0000 7f3294884640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:54.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.874+0000 7f3294884640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:54.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.874+0000 7f3294884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:54.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:54.874+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3274005380 con 0x7f32900a3b10 2024-09-16T05:39:55.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.446+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 added bucket h17 type host to location {rack=r1} v203) ==== 168+0+0 (unknown 3296798082 0 0) 0x7f3284014df0 con 0x7f32900a3b10 2024-09-16T05:39:55.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.446+0000 7f327effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:55.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.446+0000 7f327effd640 10 monclient: _finish_command 2 = system:0 added bucket h17 type host to location {rack=r1} 2024-09-16T05:39:55.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.550+0000 7f3294884640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:55.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.550+0000 7f3294884640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:55.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.550+0000 7f3294884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:39:55.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.550+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3274005ac0 con 0x7f32900a3b10 2024-09-16T05:39:55.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.550+0000 7f327effd640 1 -- 172.21.15.63:0/871856713 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h17' already exists v203) ==== 147+0+0 (unknown 33586120 0 0) 0x7f328401d9a0 con 0x7f32900a3b10 2024-09-16T05:39:55.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.550+0000 7f327effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:55.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.550+0000 7f327effd640 10 monclient: _finish_command 3 = system:0 bucket 'h17' already exists 2024-09-16T05:39:55.555 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h17' already exists 2024-09-16T05:39:55.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.554+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 >> v1:172.21.15.63:6812/3643207211 conn(0x7f32700369e0 legacy=0x7f3270038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:55.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.554+0000 7f3294884640 10 monclient: shutdown 2024-09-16T05:39:55.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.554+0000 7f3294884640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:55.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.554+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 >> v1:172.21.15.6:6789/0 conn(0x7f32900a3b10 legacy=0x7f3290136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:55.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.554+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 shutdown_connections 2024-09-16T05:39:55.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.554+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 >> 172.21.15.63:0/871856713 conn(0x7f3290075470 msgr2=0x7f32900a7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:55.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.554+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 shutdown_connections 2024-09-16T05:39:55.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.554+0000 7f3294884640 1 -- 172.21.15.63:0/871856713 wait complete. 2024-09-16T05:39:55.573 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:55.573 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h18 host rack=r1 2024-09-16T05:39:55.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient: get_monmap_and_config 2024-09-16T05:39:55.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient: build_initial_monmap 2024-09-16T05:39:55.697 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient: monmap: 2024-09-16T05:39:55.697 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:55.697 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:55.698 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:55.697423+0000 2024-09-16T05:39:55.698 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:55.697423+0000 2024-09-16T05:39:55.698 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:55.698 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:55.698 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:55.698 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:55.698 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:55.698 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:55.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 1 Processor -- start 2024-09-16T05:39:55.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 1 -- start start 2024-09-16T05:39:55.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient: init 2024-09-16T05:39:55.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:55.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:55.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient(hunting): picked mon.noname-a con 0x7fc73c09c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient(hunting): picked mon.noname-c con 0x7fc73c0a09c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient(hunting): picked mon.noname-b con 0x7fc73c09d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73c0a0ed0 con 0x7fc73c09c300 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73c09d950 con 0x7fc73c0a09c0 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73c09ebd0 con 0x7fc73c09d3a0 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 10 monclient(hunting): authenticate will time out at 1252.474336s 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc7413f3640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc73c0a09c0 legacy=0x7fc73c138a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc7423f5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc73c09d3a0 0x7fc73c13c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47122/0 (socket says 172.21.15.63:47122) 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc7423f5640 1 -- 172.21.15.63:0/3754165322 learned_addr learned my addr 172.21.15.63:0/3754165322 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2812650061 0 0) 0x7fc73c0a0ed0 con 0x7fc73c09c300 2024-09-16T05:39:55.701 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient(hunting): _negotiate 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient(hunting): my global_id is 6081 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc728005740 con 0x7fc73c09c300 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461073840 0 0) 0x7fc73c09ebd0 con 0x7fc73c09d3a0 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient(hunting): _negotiate 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient(hunting): my global_id is 6056 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc73c0a0ed0 con 0x7fc73c09d3a0 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2679536746 0 0) 0x7fc728005740 con 0x7fc73c09c300 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 >> v1:172.21.15.6:6790/0 conn(0x7fc73c09d3a0 legacy=0x7fc73c13c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:55.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 >> v1:172.21.15.63:6789/0 conn(0x7fc73c0a09c0 legacy=0x7fc73c138a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient: _finish_hunting 0 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 monclient: found mon.noname-a 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c0a2470 con 0x7fc73c09c300 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient: _finish_auth 0 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient: _check_auth_tickets 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc738001550 con 0x7fc73c09c300 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc740bf2640 10 monclient: dump: 2024-09-16T05:39:55.703 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 5 monclient: authenticate success, global_id 6081 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.694+0000 7fc743e58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc738001870 con 0x7fc73c09c300 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc740bf2640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:55.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc740bf2640 1 -- 172.21.15.63:0/3754165322 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc738001c40 con 0x7fc73c09c300 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc740bf2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc740bf2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc740bf2640 10 monclient: dump: 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:55.705 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:55.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:55.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 4 set_mon_vals no callback set 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: shutdown 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- 172.21.15.63:0/3754165322 >> v1:172.21.15.6:6789/0 conn(0x7fc73c09c300 legacy=0x7fc73c09c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- 172.21.15.63:0/3754165322 shutdown_connections 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- 172.21.15.63:0/3754165322 >> 172.21.15.63:0/3754165322 conn(0x7fc73c0a1e90 msgr2=0x7fc73c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- 172.21.15.63:0/3754165322 shutdown_connections 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- 172.21.15.63:0/3754165322 wait complete. 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: build_initial_monmap 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: monmap: 2024-09-16T05:39:55.710 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 Processor -- start 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- start start 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: init 2024-09-16T05:39:55.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient(hunting): picked mon.noname-a con 0x7fc73c09c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient(hunting): picked mon.noname-b con 0x7fc73c09d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient(hunting): picked mon.noname-c con 0x7fc73c1342a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73c09c0f0 con 0x7fc73c09c300 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73c122fb0 con 0x7fc73c09d3a0 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73c0a46e0 con 0x7fc73c1342a0 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient(hunting): authenticate will time out at 1252.478145s 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7413f3640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc73c09d3a0 legacy=0x7fc73c1cbec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc741bf4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc73c09c300 0x7fc73c133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37980/0 (socket says 172.21.15.63:37980) 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc741bf4640 1 -- 172.21.15.63:0/412359853 learned_addr learned my addr 172.21.15.63:0/412359853 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:55.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7423f5640 1 --1- 172.21.15.63:0/412359853 >> v1:172.21.15.6:6790/0 conn(0x7fc73c1342a0 0x7fc73c1cf5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47138/0 (socket says 172.21.15.63:47138) 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832928282 0 0) 0x7fc73c0a46e0 con 0x7fc73c1342a0 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): my global_id is 6059 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc720003530 con 0x7fc73c1342a0 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417050700 0 0) 0x7fc73c0a2470 con 0x7fc73c09c300 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): my global_id is 6084 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc73c0a46e0 con 0x7fc73c09c300 2024-09-16T05:39:55.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 196273041 0 0) 0x7fc720003530 con 0x7fc73c1342a0 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc73c0a2470 con 0x7fc73c1342a0 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc734001bc0 con 0x7fc73c1342a0 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): dump: 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:55.714 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3186245025 0 0) 0x7fc73c0a46e0 con 0x7fc73c09c300 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc720003530 con 0x7fc73c09c300 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc738001510 con 0x7fc73c09c300 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient(hunting): dump: 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.715 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 412050808 0 0) 0x7fc73c0a2470 con 0x7fc73c1342a0 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 >> v1:172.21.15.63:6789/0 conn(0x7fc73c09d3a0 legacy=0x7fc73c1cbec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 >> v1:172.21.15.6:6789/0 conn(0x7fc73c09c300 legacy=0x7fc73c133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 monclient: found mon.c 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:55.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c123290 con 0x7fc73c1342a0 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc7327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 5 monclient: authenticate success, global_id 6059 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: _renew_subs 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc73c121b60 con 0x7fc73c1342a0 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: _renew_subs 2024-09-16T05:39:55.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.698+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc73c1c6f90 con 0x7fc73c1342a0 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc734004b70 con 0x7fc73c1342a0 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7327fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7337fe640 4 set_mon_vals no callback set 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc743e58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc743e58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc743e58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc734004d20 con 0x7fc73c1342a0 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc724005090 con 0x7fc73c1342a0 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7327fc640 10 monclient: dump: 2024-09-16T05:39:55.718 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc734004f80 con 0x7fc73c1342a0 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(203..203 src has 1..203) ==== 10949+0+0 (unknown 1646706301 0 0) 0x7fc734046010 con 0x7fc73c1342a0 2024-09-16T05:39:55.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.702+0000 7fc741bf4640 10 monclient: get_auth_request con 0x7fc7200369e0 auth_method 0 2024-09-16T05:39:55.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.714+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc734015560 con 0x7fc73c1342a0 2024-09-16T05:39:55.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.714+0000 7fc7327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:55.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.714+0000 7fc7327fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:55.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.874+0000 7fc743e58640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:55.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.874+0000 7fc743e58640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:55.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.874+0000 7fc743e58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:55.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:55.874+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fc724005380 con 0x7fc73c1342a0 2024-09-16T05:39:56.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.462+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 added bucket h18 type host to location {rack=r1} v204) ==== 168+0+0 (unknown 2428400045 0 0) 0x7fc734019210 con 0x7fc73c1342a0 2024-09-16T05:39:56.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.462+0000 7fc7327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:56.466 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.462+0000 7fc7327fc640 10 monclient: _finish_command 2 = system:0 added bucket h18 type host to location {rack=r1} 2024-09-16T05:39:56.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.562+0000 7fc743e58640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:56.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.562+0000 7fc743e58640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:56.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.562+0000 7fc743e58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:56.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.562+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fc724005ac0 con 0x7fc73c1342a0 2024-09-16T05:39:56.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.562+0000 7fc7327fc640 1 -- 172.21.15.63:0/412359853 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h18' already exists v204) ==== 147+0+0 (unknown 3372148359 0 0) 0x7fc73401dad0 con 0x7fc73c1342a0 2024-09-16T05:39:56.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.562+0000 7fc7327fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:56.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.562+0000 7fc7327fc640 10 monclient: _finish_command 3 = system:0 bucket 'h18' already exists 2024-09-16T05:39:56.568 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h18' already exists 2024-09-16T05:39:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.566+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc7200369e0 legacy=0x7fc720038ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.566+0000 7fc743e58640 10 monclient: shutdown 2024-09-16T05:39:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.566+0000 7fc743e58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.566+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 >> v1:172.21.15.6:6790/0 conn(0x7fc73c1342a0 legacy=0x7fc73c1cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.566+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 shutdown_connections 2024-09-16T05:39:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.566+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 >> 172.21.15.63:0/412359853 conn(0x7fc73c0a1e90 msgr2=0x7fc73c09fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.566+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 shutdown_connections 2024-09-16T05:39:56.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.566+0000 7fc743e58640 1 -- 172.21.15.63:0/412359853 wait complete. 2024-09-16T05:39:56.586 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:56.587 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h19 host rack=r1 2024-09-16T05:39:56.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient: get_monmap_and_config 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient: build_initial_monmap 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient: monmap: 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:56.709801+0000 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:56.709801+0000 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:56.710 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:56.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 1 Processor -- start 2024-09-16T05:39:56.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 1 -- start start 2024-09-16T05:39:56.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient: init 2024-09-16T05:39:56.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient(hunting): picked mon.noname-c con 0x7f62f40731d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient(hunting): picked mon.noname-b con 0x7f62f40a22a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient(hunting): picked mon.noname-a con 0x7f62f4139db0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f409c580 con 0x7f62f4139db0 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f40a1300 con 0x7f62f40731d0 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f409d0f0 con 0x7f62f40a22a0 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fbf5c640 10 monclient(hunting): authenticate will time out at 1253.486842s 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f9cf8640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f62f40731d0 legacy=0x7f62f4075b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f94f7640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f62f40a22a0 0x7f62f4138690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47140/0 (socket says 172.21.15.63:47140) 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62fa4f9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62f4139db0 0x7f62f413c1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37982/0 (socket says 172.21.15.63:37982) 2024-09-16T05:39:56.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f94f7640 1 -- 172.21.15.63:0/2256164693 learned_addr learned my addr 172.21.15.63:0/2256164693 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4012492618 0 0) 0x7f62f409c580 con 0x7f62f4139db0 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 10 monclient(hunting): _negotiate 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 10 monclient(hunting): my global_id is 6087 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62e0003530 con 0x7f62f4139db0 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1257248575 0 0) 0x7f62f409d0f0 con 0x7f62f40a22a0 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 10 monclient(hunting): _negotiate 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 10 monclient(hunting): my global_id is 6062 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.706+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62f409c580 con 0x7f62f40a22a0 2024-09-16T05:39:56.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1160388668 0 0) 0x7f62e0003530 con 0x7f62f4139db0 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 >> v1:172.21.15.6:6790/0 conn(0x7f62f40a22a0 legacy=0x7f62f4138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 >> v1:172.21.15.63:6789/0 conn(0x7f62f40731d0 legacy=0x7f62f4075b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: _finish_hunting 0 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 1 monclient: found mon.noname-a 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f409d3d0 con 0x7f62f4139db0 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: _finish_auth 0 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: _check_auth_tickets 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62ec0018d0 con 0x7f62f4139db0 2024-09-16T05:39:56.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: dump: 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:56.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 5 monclient: authenticate success, global_id 6087 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62ec001b30 con 0x7f62f4139db0 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 1 -- 172.21.15.63:0/2256164693 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62ec001f40 con 0x7f62f4139db0 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 4 set_mon_vals no callback set 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f8cf6640 10 monclient: dump: 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.718 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient: shutdown 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- 172.21.15.63:0/2256164693 >> v1:172.21.15.6:6789/0 conn(0x7f62f4139db0 legacy=0x7f62f413c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- 172.21.15.63:0/2256164693 shutdown_connections 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- 172.21.15.63:0/2256164693 >> 172.21.15.63:0/2256164693 conn(0x7f62f40a1e90 msgr2=0x7f62f4075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- 172.21.15.63:0/2256164693 shutdown_connections 2024-09-16T05:39:56.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- 172.21.15.63:0/2256164693 wait complete. 2024-09-16T05:39:56.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient: build_initial_monmap 2024-09-16T05:39:56.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient: monmap: 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 Processor -- start 2024-09-16T05:39:56.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- start start 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient: init 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient(hunting): picked mon.noname-a con 0x7f62f40731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient(hunting): picked mon.noname-b con 0x7f62f40a22a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient(hunting): picked mon.noname-c con 0x7f62f4139db0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f41229c0 con 0x7f62f40731d0 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f4121780 con 0x7f62f40a22a0 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f40a3cc0 con 0x7f62f4139db0 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f94f7640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f62f40a22a0 legacy=0x7f62f41d01a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62fbf5c640 10 monclient(hunting): authenticate will time out at 1253.490684s 2024-09-16T05:39:56.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f9cf8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f62f40731d0 0x7f62f41345b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:37992/0 (socket says 172.21.15.63:37992) 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62f9cf8640 1 -- 172.21.15.63:0/3092879584 learned_addr learned my addr 172.21.15.63:0/3092879584 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 319895230 0 0) 0x7f62f41229c0 con 0x7f62f40731d0 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 10 monclient(hunting): my global_id is 6090 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62d8003530 con 0x7f62f40731d0 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3305879677 0 0) 0x7f62f40a3cc0 con 0x7f62f4139db0 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 10 monclient(hunting): my global_id is 6065 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.710+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62f41229c0 con 0x7f62f4139db0 2024-09-16T05:39:56.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2766044305 0 0) 0x7f62f41229c0 con 0x7f62f4139db0 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62f40a3cc0 con 0x7f62f4139db0 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1275181227 0 0) 0x7f62d8003530 con 0x7f62f40731d0 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62f41229c0 con 0x7f62f40731d0 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62ec001e80 con 0x7f62f4139db0 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient(hunting): dump: 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.727 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62f00040e0 con 0x7f62f40731d0 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient(hunting): dump: 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:56.728 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3336119904 0 0) 0x7f62f40a3cc0 con 0x7f62f4139db0 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 >> v1:172.21.15.63:6789/0 conn(0x7f62f40a22a0 legacy=0x7f62f41d01a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 >> v1:172.21.15.6:6789/0 conn(0x7f62f40731d0 legacy=0x7f62f41345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 monclient: found mon.c 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:56.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f40a4210 con 0x7f62f4139db0 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 5 monclient: authenticate success, global_id 6065 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 10 monclient: _renew_subs 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f62f4121500 con 0x7f62f4139db0 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 10 monclient: _renew_subs 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f62f4134cc0 con 0x7f62f4139db0 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62ec004e20 con 0x7f62f4139db0 2024-09-16T05:39:56.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f62ec005170 con 0x7f62f4139db0 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ebfff640 4 set_mon_vals no callback set 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 10 monclient: dump: 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:56.731 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc005090 con 0x7f62f4139db0 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f62ec005310 con 0x7f62f4139db0 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(204..204 src has 20..204) ==== 11162+0+0 (unknown 2119527994 0 0) 0x7f62ec0462b0 con 0x7f62f4139db0 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.714+0000 7f62f9cf8640 10 monclient: get_auth_request con 0x7f62d8036a60 auth_method 0 2024-09-16T05:39:56.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.726+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f62ec0158a0 con 0x7f62f4139db0 2024-09-16T05:39:56.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.726+0000 7f62ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:56.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.726+0000 7f62ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:56.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.894+0000 7f62fbf5c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:56.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.894+0000 7f62fbf5c640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:56.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.894+0000 7f62fbf5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:56.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:56.894+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f62dc005380 con 0x7f62f4139db0 2024-09-16T05:39:57.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.470+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 added bucket h19 type host to location {rack=r1} v205) ==== 168+0+0 (unknown 3970757712 0 0) 0x7f62ec0151a0 con 0x7f62f4139db0 2024-09-16T05:39:57.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.470+0000 7f62ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:57.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.470+0000 7f62ea7fc640 10 monclient: _finish_command 2 = system:0 added bucket h19 type host to location {rack=r1} 2024-09-16T05:39:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.574+0000 7f62fbf5c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.574+0000 7f62fbf5c640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.574+0000 7f62fbf5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:57.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.574+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f62dc005ac0 con 0x7f62f4139db0 2024-09-16T05:39:57.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62ea7fc640 1 -- 172.21.15.63:0/3092879584 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h19' already exists v205) ==== 147+0+0 (unknown 3276341098 0 0) 0x7f62ec0422f0 con 0x7f62f4139db0 2024-09-16T05:39:57.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62ea7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:57.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62ea7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h19' already exists 2024-09-16T05:39:57.581 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h19' already exists 2024-09-16T05:39:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 >> v1:172.21.15.63:6812/3643207211 conn(0x7f62d8036a60 legacy=0x7f62d8038f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62fbf5c640 10 monclient: shutdown 2024-09-16T05:39:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62fbf5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 >> v1:172.21.15.6:6790/0 conn(0x7f62f4139db0 legacy=0x7f62f41d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 shutdown_connections 2024-09-16T05:39:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 >> 172.21.15.63:0/3092879584 conn(0x7f62f40a1e90 msgr2=0x7f62f40a2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 shutdown_connections 2024-09-16T05:39:57.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.578+0000 7f62fbf5c640 1 -- 172.21.15.63:0/3092879584 wait complete. 2024-09-16T05:39:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:57.598 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h20 host rack=r1 2024-09-16T05:39:57.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient: get_monmap_and_config 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient: build_initial_monmap 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient: monmap: 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:57.721889+0000 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:57.721889+0000 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:57.722 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:57.723 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:57.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 1 Processor -- start 2024-09-16T05:39:57.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 1 -- start start 2024-09-16T05:39:57.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient: init 2024-09-16T05:39:57.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:57.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:39:57.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient(hunting): picked mon.noname-b con 0x7f51b809cfe0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:57.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient(hunting): picked mon.noname-a con 0x7f51b80a1040 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:57.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient(hunting): picked mon.noname-c con 0x7f51b813afb0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b809fd70 con 0x7f51b80a1040 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b80a0330 con 0x7f51b813afb0 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b809dad0 con 0x7f51b809cfe0 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bf734640 10 monclient(hunting): authenticate will time out at 1254.498796s 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bdcd1640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f51b813afb0 legacy=0x7f51b813d3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bcccf640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f51b80a1040 0x7f51b8139890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38000/0 (socket says 172.21.15.63:38000) 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bcccf640 1 -- 172.21.15.63:0/2511416823 learned_addr learned my addr 172.21.15.63:0/2511416823 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51bd4d0640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f51b809cfe0 0x7f51b809d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47170/0 (socket says 172.21.15.63:47170) 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797280355 0 0) 0x7f51b809dad0 con 0x7f51b809cfe0 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:57.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 10 monclient(hunting): my global_id is 6068 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51a4003530 con 0x7f51b809cfe0 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015670733 0 0) 0x7f51b809fd70 con 0x7f51b80a1040 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 10 monclient(hunting): _negotiate 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 10 monclient(hunting): my global_id is 6093 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.718+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51b809dad0 con 0x7f51b80a1040 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3503453275 0 0) 0x7f51a4003530 con 0x7f51b809cfe0 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 >> v1:172.21.15.63:6789/0 conn(0x7f51b813afb0 legacy=0x7f51b813d3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:57.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 >> v1:172.21.15.6:6789/0 conn(0x7f51b80a1040 legacy=0x7f51b8139890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: _finish_hunting 0 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 1 monclient: found mon.noname-b 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b8075e40 con 0x7f51b809cfe0 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: _finish_auth 0 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: _check_auth_tickets 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f51a8001550 con 0x7f51b809cfe0 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: dump: 2024-09-16T05:39:57.728 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 5 monclient: authenticate success, global_id 6068 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51a8001870 con 0x7f51b809cfe0 2024-09-16T05:39:57.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 1 -- 172.21.15.63:0/2511416823 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f51a8001c40 con 0x7f51b809cfe0 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51affff640 10 monclient: dump: 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:57.730 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 4 set_mon_vals no callback set 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient: shutdown 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- 172.21.15.63:0/2511416823 >> v1:172.21.15.6:6790/0 conn(0x7f51b809cfe0 legacy=0x7f51b809d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- 172.21.15.63:0/2511416823 shutdown_connections 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- 172.21.15.63:0/2511416823 >> 172.21.15.63:0/2511416823 conn(0x7f51b80a3eb0 msgr2=0x7f51b8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- 172.21.15.63:0/2511416823 shutdown_connections 2024-09-16T05:39:57.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- 172.21.15.63:0/2511416823 wait complete. 2024-09-16T05:39:57.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient: build_initial_monmap 2024-09-16T05:39:57.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient: monmap: 2024-09-16T05:39:57.734 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:57.734 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 Processor -- start 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- start start 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient: init 2024-09-16T05:39:57.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient(hunting): picked mon.noname-a con 0x7f51b809cfe0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient(hunting): picked mon.noname-c con 0x7f51b80a1040 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient(hunting): picked mon.noname-b con 0x7f51b813afb0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b8127080 con 0x7f51b809cfe0 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b8125eb0 con 0x7f51b813afb0 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b809e110 con 0x7f51b80a1040 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bf734640 10 monclient(hunting): authenticate will time out at 1254.502480s 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bdcd1640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f51b813afb0 legacy=0x7f51b81d3ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bcccf640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f51b80a1040 0x7f51b81d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47178/0 (socket says 172.21.15.63:47178) 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51bcccf640 1 -- 172.21.15.63:0/901308518 learned_addr learned my addr 172.21.15.63:0/901308518 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:57.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4204888968 0 0) 0x7f51b809e110 con 0x7f51b80a1040 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): my global_id is 6071 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f519c003530 con 0x7f51b80a1040 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077932590 0 0) 0x7f51b8127080 con 0x7f51b809cfe0 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): _negotiate 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): my global_id is 6096 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51b809e110 con 0x7f51b809cfe0 2024-09-16T05:39:57.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3006706473 0 0) 0x7f519c003530 con 0x7f51b80a1040 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51b8127080 con 0x7f51b80a1040 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f51b4001900 con 0x7f51b80a1040 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): dump: 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:57.738 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4169126017 0 0) 0x7f51b8127080 con 0x7f51b80a1040 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 >> v1:172.21.15.63:6789/0 conn(0x7f51b813afb0 legacy=0x7f51b81d3ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.722+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 >> v1:172.21.15.6:6789/0 conn(0x7f51b809cfe0 legacy=0x7f51b8139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 10 monclient: _finish_hunting 0 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 1 monclient: found mon.c 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b8126490 con 0x7f51b80a1040 2024-09-16T05:39:57.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 10 monclient: _finish_auth 0 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 10 monclient: _check_auth_tickets 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 5 monclient: authenticate success, global_id 6071 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 10 monclient: _renew_subs 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f51b8126ae0 con 0x7f51b80a1040 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 10 monclient: _renew_subs 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:57.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f51b813a390 con 0x7f51b80a1040 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51b40047f0 con 0x7f51b80a1040 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f51b4004b40 con 0x7f51b80a1040 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51af7fe640 4 set_mon_vals no callback set 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 10 monclient: dump: 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:57.741 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5198005090 con 0x7f51b80a1040 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f51b4004d60 con 0x7f51b80a1040 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(205..205 src has 20..205) ==== 11375+0+0 (unknown 389723086 0 0) 0x7f51b4046040 con 0x7f51b80a1040 2024-09-16T05:39:57.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.726+0000 7f51bd4d0640 10 monclient: get_auth_request con 0x7f519c036ac0 auth_method 0 2024-09-16T05:39:57.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.738+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f51b40154c0 con 0x7f51b80a1040 2024-09-16T05:39:57.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.738+0000 7f51adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:57.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.738+0000 7f51adffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:57.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.902+0000 7f51bf734640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:57.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.902+0000 7f51bf734640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:57.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.902+0000 7f51bf734640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:57.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:57.902+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f5198005380 con 0x7f51b80a1040 2024-09-16T05:39:58.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.478+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 added bucket h20 type host to location {rack=r1} v206) ==== 168+0+0 (unknown 1883535626 0 0) 0x7f51b4014dc0 con 0x7f51b80a1040 2024-09-16T05:39:58.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.478+0000 7f51adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:58.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.478+0000 7f51adffb640 10 monclient: _finish_command 2 = system:0 added bucket h20 type host to location {rack=r1} 2024-09-16T05:39:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.582+0000 7f51bf734640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.582+0000 7f51bf734640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:58.585 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.582+0000 7f51bf734640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:58.586 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.582+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f5198005ac0 con 0x7f51b80a1040 2024-09-16T05:39:58.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.582+0000 7f51adffb640 1 -- 172.21.15.63:0/901308518 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h20' already exists v206) ==== 147+0+0 (unknown 2850835731 0 0) 0x7f51b401d970 con 0x7f51b80a1040 2024-09-16T05:39:58.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.582+0000 7f51adffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:58.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.582+0000 7f51adffb640 10 monclient: _finish_command 3 = system:0 bucket 'h20' already exists 2024-09-16T05:39:58.587 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h20' already exists 2024-09-16T05:39:58.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.586+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 >> v1:172.21.15.63:6812/3643207211 conn(0x7f519c036ac0 legacy=0x7f519c038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:58.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.586+0000 7f51bf734640 10 monclient: shutdown 2024-09-16T05:39:58.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.586+0000 7f51bf734640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:58.589 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.586+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 >> v1:172.21.15.6:6790/0 conn(0x7f51b80a1040 legacy=0x7f51b81d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.586+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 shutdown_connections 2024-09-16T05:39:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.586+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 >> 172.21.15.63:0/901308518 conn(0x7f51b80a3eb0 msgr2=0x7f51b80a62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.586+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 shutdown_connections 2024-09-16T05:39:58.590 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.586+0000 7f51bf734640 1 -- 172.21.15.63:0/901308518 wait complete. 2024-09-16T05:39:58.604 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:58.604 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h21 host rack=r1 2024-09-16T05:39:58.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient: get_monmap_and_config 2024-09-16T05:39:58.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient: build_initial_monmap 2024-09-16T05:39:58.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient: monmap: 2024-09-16T05:39:58.729 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:58.729404+0000 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:58.729404+0000 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:58.730 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:58.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 1 Processor -- start 2024-09-16T05:39:58.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 1 -- start start 2024-09-16T05:39:58.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient: init 2024-09-16T05:39:58.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient(hunting): picked mon.noname-c con 0x7f0528134930 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient(hunting): picked mon.noname-a con 0x7f05280a0810 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient(hunting): picked mon.noname-b con 0x7f052806dfa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0528135420 con 0x7f05280a0810 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f052809e830 con 0x7f0528134930 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05280a0dc0 con 0x7f052806dfa0 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f0530194640 10 monclient(hunting): authenticate will time out at 1255.506672s 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052df30640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0528134930 legacy=0x7f0528134d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052e731640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f052806dfa0 0x7f052813cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47190/0 (socket says 172.21.15.63:47190) 2024-09-16T05:39:58.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052e731640 1 -- 172.21.15.63:0/3427694112 learned_addr learned my addr 172.21.15.63:0/3427694112 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052d72f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f05280a0810 0x7f0528139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38024/0 (socket says 172.21.15.63:38024) 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717123182 0 0) 0x7f0528135420 con 0x7f05280a0810 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 10 monclient(hunting): _negotiate 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 10 monclient(hunting): my global_id is 6099 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0510003530 con 0x7f05280a0810 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177574874 0 0) 0x7f05280a0dc0 con 0x7f052806dfa0 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 10 monclient(hunting): _negotiate 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:58.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 10 monclient(hunting): my global_id is 6074 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.726+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0528135420 con 0x7f052806dfa0 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1710890159 0 0) 0x7f0528135420 con 0x7f052806dfa0 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 >> v1:172.21.15.63:6789/0 conn(0x7f0528134930 legacy=0x7f0528134d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 >> v1:172.21.15.6:6789/0 conn(0x7f05280a0810 legacy=0x7f0528139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: _finish_hunting 0 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 1 monclient: found mon.noname-b 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05280a2500 con 0x7f052806dfa0 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: _finish_auth 0 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: _check_auth_tickets 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:58.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0520001600 con 0x7f052806dfa0 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: dump: 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:58.739 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 5 monclient: authenticate success, global_id 6074 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0520001960 con 0x7f052806dfa0 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 1 -- 172.21.15.63:0/3427694112 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f05200048b0 con 0x7f052806dfa0 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 4 set_mon_vals no callback set 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052cf2e640 10 monclient: dump: 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:58.740 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient: shutdown 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- 172.21.15.63:0/3427694112 >> v1:172.21.15.6:6790/0 conn(0x7f052806dfa0 legacy=0x7f052813cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:58.741 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- 172.21.15.63:0/3427694112 shutdown_connections 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- 172.21.15.63:0/3427694112 >> 172.21.15.63:0/3427694112 conn(0x7f05280a1e90 msgr2=0x7f0528075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- 172.21.15.63:0/3427694112 shutdown_connections 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- 172.21.15.63:0/3427694112 wait complete. 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient: build_initial_monmap 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient: monmap: 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:58.742 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 Processor -- start 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- start start 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient: init 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient(hunting): picked mon.noname-b con 0x7f052806dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient(hunting): picked mon.noname-c con 0x7f05280a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient(hunting): picked mon.noname-a con 0x7f0528134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f052813b4e0 con 0x7f0528134930 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f052813e110 con 0x7f052806dfa0 2024-09-16T05:39:58.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052df30640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f052806dfa0 legacy=0x7f052813da00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05280a4240 con 0x7f05280a0810 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f0530194640 10 monclient(hunting): authenticate will time out at 1255.510470s 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052e731640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0528134930 0x7f05281d3810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38026/0 (socket says 172.21.15.63:38026) 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052d72f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f05280a0810 0x7f05281d00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47198/0 (socket says 172.21.15.63:47198) 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f052e731640 1 -- 172.21.15.63:0/703784269 learned_addr learned my addr 172.21.15.63:0/703784269 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2283903750 0 0) 0x7f052813b4e0 con 0x7f0528134930 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): my global_id is 6102 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0514003530 con 0x7f0528134930 2024-09-16T05:39:58.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347258851 0 0) 0x7f05280a4240 con 0x7f05280a0810 2024-09-16T05:39:58.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:39:58.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:58.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:58.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): my global_id is 6077 2024-09-16T05:39:58.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f052813b4e0 con 0x7f05280a0810 2024-09-16T05:39:58.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1033612516 0 0) 0x7f0514003530 con 0x7f0528134930 2024-09-16T05:39:58.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05280a4240 con 0x7f0528134930 2024-09-16T05:39:58.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0520001420 con 0x7f0528134930 2024-09-16T05:39:58.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:58.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): dump: 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:58.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2958027276 0 0) 0x7f052813b4e0 con 0x7f05280a0810 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0514003530 con 0x7f05280a0810 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f051c001b50 con 0x7f05280a0810 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.730+0000 7f05267fc640 10 monclient(hunting): dump: 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:58.749 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 685086826 0 0) 0x7f0514003530 con 0x7f05280a0810 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 >> v1:172.21.15.63:6789/0 conn(0x7f052806dfa0 legacy=0x7f052813da00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 >> v1:172.21.15.6:6789/0 conn(0x7f0528134930 legacy=0x7f05281d3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 10 monclient: _finish_hunting 0 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 monclient: found mon.c 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052809c0f0 con 0x7f05280a0810 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 10 monclient: _finish_auth 0 2024-09-16T05:39:58.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 10 monclient: _check_auth_tickets 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 5 monclient: authenticate success, global_id 6077 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 10 monclient: _renew_subs 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05281391f0 con 0x7f05280a0810 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 10 monclient: _renew_subs 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f052813a090 con 0x7f05280a0810 2024-09-16T05:39:58.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f051c002080 con 0x7f05280a0810 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f051c004d10 con 0x7f05280a0810 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0527fff640 4 set_mon_vals no callback set 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 10 monclient: dump: 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:58.752 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0508005090 con 0x7f05280a0810 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f051c00fb30 con 0x7f05280a0810 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(206..206 src has 20..206) ==== 11588+0+0 (unknown 257310920 0 0) 0x7f051c046260 con 0x7f05280a0810 2024-09-16T05:39:58.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.734+0000 7f052df30640 10 monclient: get_auth_request con 0x7f0514036a20 auth_method 0 2024-09-16T05:39:58.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.746+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f051c015590 con 0x7f05280a0810 2024-09-16T05:39:58.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.746+0000 7f05267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:58.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.746+0000 7f05267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:58.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.922+0000 7f0530194640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:58.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.922+0000 7f0530194640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:58.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.922+0000 7f0530194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:58.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:58.922+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f0508005380 con 0x7f05280a0810 2024-09-16T05:39:59.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.490+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 added bucket h21 type host to location {rack=r1} v207) ==== 168+0+0 (unknown 207027447 0 0) 0x7f051c014e90 con 0x7f05280a0810 2024-09-16T05:39:59.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.490+0000 7f05267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:59.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.490+0000 7f05267fc640 10 monclient: _finish_command 2 = system:0 added bucket h21 type host to location {rack=r1} 2024-09-16T05:39:59.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.594+0000 7f0530194640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:59.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.594+0000 7f0530194640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:59.598 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.594+0000 7f0530194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:59.599 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.594+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f0508005ac0 con 0x7f05280a0810 2024-09-16T05:39:59.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f05267fc640 1 -- 172.21.15.63:0/703784269 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h21' already exists v207) ==== 147+0+0 (unknown 2723828990 0 0) 0x7f051c0191e0 con 0x7f05280a0810 2024-09-16T05:39:59.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f05267fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:59.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f05267fc640 10 monclient: _finish_command 3 = system:0 bucket 'h21' already exists 2024-09-16T05:39:59.601 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h21' already exists 2024-09-16T05:39:59.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0514036a20 legacy=0x7f0514038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:59.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f0530194640 10 monclient: shutdown 2024-09-16T05:39:59.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f0530194640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:59.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 >> v1:172.21.15.6:6790/0 conn(0x7f05280a0810 legacy=0x7f05281d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:59.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 shutdown_connections 2024-09-16T05:39:59.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 >> 172.21.15.63:0/703784269 conn(0x7f05280a1e90 msgr2=0x7f052809d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:59.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 shutdown_connections 2024-09-16T05:39:59.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.598+0000 7f0530194640 1 -- 172.21.15.63:0/703784269 wait complete. 2024-09-16T05:39:59.618 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:39:59.618 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h22 host rack=r1 2024-09-16T05:39:59.742 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.738+0000 7f3cc6506640 10 monclient: get_monmap_and_config 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.738+0000 7f3cc6506640 10 monclient: build_initial_monmap 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.738+0000 7f3cc6506640 10 monclient: monmap: 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:39:59.742828+0000 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:39:59.742828+0000 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:39:59.743 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:59.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.738+0000 7f3cc6506640 1 Processor -- start 2024-09-16T05:39:59.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.738+0000 7f3cc6506640 1 -- start start 2024-09-16T05:39:59.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.738+0000 7f3cc6506640 10 monclient: init 2024-09-16T05:39:59.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient(hunting): picked mon.noname-b con 0x7f3cc00a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient(hunting): picked mon.noname-c con 0x7f3cc00a4d90 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient(hunting): picked mon.noname-a con 0x7f3cc00a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc00a51e0 con 0x7f3cc00a4790 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc00a0d60 con 0x7f3cc00a4d90 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc00a1040 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient(hunting): authenticate will time out at 1256.520645s 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbf7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3cc00a4d90 legacy=0x7f3cc013bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:59.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc4aa3640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3cc00a4790 0x7f3cc013f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38034/0 (socket says 172.21.15.63:38034) 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3cc00a3ca0 0x7f3cc00a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47204/0 (socket says 172.21.15.63:47204) 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc4aa3640 1 -- 172.21.15.63:0/3023133429 learned_addr learned my addr 172.21.15.63:0/3023133429 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200583254 0 0) 0x7f3cc00a1040 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient(hunting): my global_id is 6080 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ca8003530 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4042530489 0 0) 0x7f3cc00a51e0 con 0x7f3cc00a4790 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:39:59.748 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient(hunting): my global_id is 6105 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc00a1040 con 0x7f3cc00a4790 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1773398190 0 0) 0x7f3ca8003530 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 >> v1:172.21.15.63:6789/0 conn(0x7f3cc00a4d90 legacy=0x7f3cc013bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 >> v1:172.21.15.6:6789/0 conn(0x7f3cc00a4790 legacy=0x7f3cc013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 monclient: found mon.noname-b 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc00a0810 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: _finish_auth 0 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3cb00019a0 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: dump: 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:59.750 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 5 monclient: authenticate success, global_id 6080 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3cb0001c50 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 1 -- 172.21.15.63:0/3023133429 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3cb0002020 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:59.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cbeffd640 10 monclient: dump: 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient: get_monmap_and_config success 2024-09-16T05:39:59.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 4 set_mon_vals no callback set 2024-09-16T05:39:59.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient: shutdown 2024-09-16T05:39:59.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:39:59.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 1 -- 172.21.15.63:0/3023133429 >> v1:172.21.15.6:6790/0 conn(0x7f3cc00a3ca0 legacy=0x7f3cc00a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:59.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 1 -- 172.21.15.63:0/3023133429 shutdown_connections 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 1 -- 172.21.15.63:0/3023133429 >> 172.21.15.63:0/3023133429 conn(0x7f3cc0075470 msgr2=0x7f3cc0075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 1 -- 172.21.15.63:0/3023133429 shutdown_connections 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 1 -- 172.21.15.63:0/3023133429 wait complete. 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.742+0000 7f3cc6506640 10 monclient: build_initial_monmap 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: monmap: 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:59.758 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 1 Processor -- start 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 1 -- start start 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: init 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: _reopen_session rank -1 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient(hunting): picked mon.noname-a con 0x7f3cc00a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient(hunting): picked mon.noname-b con 0x7f3cc00a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient(hunting): picked mon.noname-c con 0x7f3cc00a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc0138b10 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc0135dd0 con 0x7f3cc00a4790 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc0138df0 con 0x7f3cc00a4d90 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient(hunting): _renew_subs 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbf7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f3cc00a4790 legacy=0x7f3cc01356c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient(hunting): authenticate will time out at 1256.525266s 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc4aa3640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3cc00a4d90 0x7f3cc01d3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47210/0 (socket says 172.21.15.63:47210) 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3cc00a3ca0 0x7f3cc0134fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38040/0 (socket says 172.21.15.63:38040) 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc4aa3640 1 -- 172.21.15.63:0/2939654618 learned_addr learned my addr 172.21.15.63:0/2939654618 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775456689 0 0) 0x7f3cc0138df0 con 0x7f3cc00a4d90 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): my global_id is 6083 2024-09-16T05:39:59.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cac003530 con 0x7f3cc00a4d90 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1575818457 0 0) 0x7f3cc0138b10 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): my global_id is 6108 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc0138df0 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2673768872 0 0) 0x7f3cac003530 con 0x7f3cc00a4d90 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cc0138b10 con 0x7f3cc00a4d90 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3cb80018c0 con 0x7f3cc00a4d90 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): dump: 2024-09-16T05:39:59.761 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 557348418 0 0) 0x7f3cc0138df0 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cac003530 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3cb0001e00 con 0x7f3cc00a3ca0 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient(hunting): dump: 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:59.763 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2712004061 0 0) 0x7f3cc0138b10 con 0x7f3cc00a4d90 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 >> v1:172.21.15.63:6789/0 conn(0x7f3cc00a4790 legacy=0x7f3cc01356c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 >> v1:172.21.15.6:6789/0 conn(0x7f3cc00a3ca0 legacy=0x7f3cc0134fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient: _finish_hunting 0 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 monclient: found mon.c 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc0138830 con 0x7f3cc00a4d90 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient: _finish_auth 0 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient: _check_auth_tickets 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:39:59.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 5 monclient: authenticate success, global_id 6083 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: _renew_subs 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc01374b0 con 0x7f3cc00a4d90 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: _renew_subs 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3cc013fa70 con 0x7f3cc00a4d90 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3cb80047f0 con 0x7f3cc00a4d90 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3cb8004b40 con 0x7f3cc00a4d90 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbe7fc640 4 set_mon_vals no callback set 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:39:59.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cbcff9640 10 monclient: dump: 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:59.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:59.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.746+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca0005090 con 0x7f3cc00a4d90 2024-09-16T05:39:59.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.750+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3cb800f8e0 con 0x7f3cc00a4d90 2024-09-16T05:39:59.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.750+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(207..207 src has 20..207) ==== 11801+0+0 (unknown 666927163 0 0) 0x7f3cb8046080 con 0x7f3cc00a4d90 2024-09-16T05:39:59.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.750+0000 7f3cbffff640 10 monclient: get_auth_request con 0x7f3cac036d00 auth_method 0 2024-09-16T05:39:59.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.762+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3cb80152f0 con 0x7f3cc00a4d90 2024-09-16T05:39:59.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.762+0000 7f3cbcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:39:59.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.762+0000 7f3cbcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:39:59.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.930+0000 7f3cc6506640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:59.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.930+0000 7f3cc6506640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:39:59.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.930+0000 7f3cc6506640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:39:59.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:39:59.930+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3ca0005380 con 0x7f3cc00a4d90 2024-09-16T05:40:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.490+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 added bucket h22 type host to location {rack=r1} v208) ==== 168+0+0 (unknown 3038253968 0 0) 0x7f3cb8018fa0 con 0x7f3cc00a4d90 2024-09-16T05:40:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.490+0000 7f3cbcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:00.496 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.490+0000 7f3cbcff9640 10 monclient: _finish_command 2 = system:0 added bucket h22 type host to location {rack=r1} 2024-09-16T05:40:00.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.598+0000 7f3cc6506640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:00.600 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.598+0000 7f3cc6506640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:00.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.598+0000 7f3cc6506640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:00.601 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.598+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3ca0005ac0 con 0x7f3cc00a4d90 2024-09-16T05:40:00.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.598+0000 7f3cbcff9640 1 -- 172.21.15.63:0/2939654618 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h22' already exists v208) ==== 147+0+0 (unknown 102639366 0 0) 0x7f3cb80147e0 con 0x7f3cc00a4d90 2024-09-16T05:40:00.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.598+0000 7f3cbcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:00.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.598+0000 7f3cbcff9640 10 monclient: _finish_command 3 = system:0 bucket 'h22' already exists 2024-09-16T05:40:00.602 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h22' already exists 2024-09-16T05:40:00.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.602+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3cac036d00 legacy=0x7f3cac0391c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:00.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.602+0000 7f3cc6506640 10 monclient: shutdown 2024-09-16T05:40:00.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.602+0000 7f3cc6506640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:00.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.602+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 >> v1:172.21.15.6:6790/0 conn(0x7f3cc00a4d90 legacy=0x7f3cc01d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:00.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.602+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 shutdown_connections 2024-09-16T05:40:00.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.602+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 >> 172.21.15.63:0/2939654618 conn(0x7f3cc0075470 msgr2=0x7f3cc00a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:00.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.602+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 shutdown_connections 2024-09-16T05:40:00.605 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.602+0000 7f3cc6506640 1 -- 172.21.15.63:0/2939654618 wait complete. 2024-09-16T05:40:00.620 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:00.620 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h23 host rack=r1 2024-09-16T05:40:00.745 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient: get_monmap_and_config 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient: build_initial_monmap 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient: monmap: 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:00.745935+0000 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:00.745935+0000 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:00.746 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 1 Processor -- start 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 1 -- start start 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient: init 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient(hunting): picked mon.noname-c con 0x7f1cd40a27c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient(hunting): picked mon.noname-b con 0x7f1cd40a32b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient(hunting): picked mon.noname-a con 0x7f1cd413d4d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:00.749 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd40a0050 con 0x7f1cd413d4d0 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd409ed40 con 0x7f1cd40a27c0 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd409fd70 con 0x7f1cd40a32b0 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd96bb640 10 monclient(hunting): authenticate will time out at 1257.522864s 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd2ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f1cd40a27c0 legacy=0x7f1cd40a2ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1ccbfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1cd40a32b0 0x7f1cd413bdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47218/0 (socket says 172.21.15.63:47218) 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1ccbfff640 1 -- 172.21.15.63:0/617659182 learned_addr learned my addr 172.21.15.63:0/617659182 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd37fe640 1 --1- 172.21.15.63:0/617659182 >> v1:172.21.15.6:6789/0 conn(0x7f1cd413d4d0 0x7f1cd413f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38050/0 (socket says 172.21.15.63:38050) 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68859266 0 0) 0x7f1cd409fd70 con 0x7f1cd40a32b0 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:00.750 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 10 monclient(hunting): my global_id is 6086 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cc0003530 con 0x7f1cd40a32b0 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810248440 0 0) 0x7f1ccc0013d0 con 0x7f1cd413d4d0 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 10 monclient(hunting): my global_id is 6111 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.742+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd409fd70 con 0x7f1cd413d4d0 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4030947314 0 0) 0x7f1cd409fd70 con 0x7f1cd413d4d0 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 >> v1:172.21.15.6:6790/0 conn(0x7f1cd40a32b0 legacy=0x7f1cd413bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 >> v1:172.21.15.63:6789/0 conn(0x7f1cd40a27c0 legacy=0x7f1cd40a2ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 1 monclient: found mon.noname-a 2024-09-16T05:40:00.751 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd409de70 con 0x7f1cd413d4d0 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1ccc001a80 con 0x7f1cd413d4d0 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: dump: 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.752 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 5 monclient: authenticate success, global_id 6111 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ccc001ce0 con 0x7f1cd413d4d0 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 1 -- 172.21.15.63:0/617659182 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1ccc004b70 con 0x7f1cd413d4d0 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:00.753 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd27fc640 10 monclient: dump: 2024-09-16T05:40:00.754 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:00.754 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:00.754 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.754 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.754 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:00.754 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:00.755 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:00.755 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:00.755 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:00.755 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:00.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:00.755 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 4 set_mon_vals no callback set 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient: shutdown 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- 172.21.15.63:0/617659182 >> v1:172.21.15.6:6789/0 conn(0x7f1cd413d4d0 legacy=0x7f1cd413f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- 172.21.15.63:0/617659182 shutdown_connections 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- 172.21.15.63:0/617659182 >> 172.21.15.63:0/617659182 conn(0x7f1cd40a3eb0 msgr2=0x7f1cd4075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- 172.21.15.63:0/617659182 shutdown_connections 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- 172.21.15.63:0/617659182 wait complete. 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient: build_initial_monmap 2024-09-16T05:40:00.759 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient: monmap: 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 Processor -- start 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- start start 2024-09-16T05:40:00.760 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient: init 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient(hunting): picked mon.noname-c con 0x7f1cd40a27c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient(hunting): picked mon.noname-a con 0x7f1cd40a32b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient(hunting): picked mon.noname-b con 0x7f1cd413d4d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd409e110 con 0x7f1cd40a32b0 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd40a65f0 con 0x7f1cd413d4d0 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd413bce0 con 0x7f1cd40a27c0 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd96bb640 10 monclient(hunting): authenticate will time out at 1257.527916s 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.746+0000 7f1cd37fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f1cd413d4d0 legacy=0x7f1cd41d3fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccbfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1cd40a32b0 0x7f1cd41d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38058/0 (socket says 172.21.15.63:38058) 2024-09-16T05:40:00.761 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccbfff640 1 -- 172.21.15.63:0/1381596840 learned_addr learned my addr 172.21.15.63:0/1381596840 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1886388730 0 0) 0x7f1cd409e110 con 0x7f1cd40a32b0 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): my global_id is 6114 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cb4003530 con 0x7f1cd40a32b0 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3486242513 0 0) 0x7f1cd413bce0 con 0x7f1cd40a27c0 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): my global_id is 6089 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd409e110 con 0x7f1cd40a27c0 2024-09-16T05:40:00.762 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3879781084 0 0) 0x7f1cb4003530 con 0x7f1cd40a32b0 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cd413bce0 con 0x7f1cd40a32b0 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1cbc001a40 con 0x7f1cd40a32b0 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:00.763 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 921125493 0 0) 0x7f1cd409e110 con 0x7f1cd40a27c0 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cb4003530 con 0x7f1cd40a27c0 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1cc40040d0 con 0x7f1cd40a27c0 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:00.764 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2254469617 0 0) 0x7f1cd413bce0 con 0x7f1cd40a32b0 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 >> v1:172.21.15.6:6790/0 conn(0x7f1cd40a27c0 legacy=0x7f1cd41cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 >> v1:172.21.15.63:6789/0 conn(0x7f1cd413d4d0 legacy=0x7f1cd41d3fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 monclient: found mon.a 2024-09-16T05:40:00.765 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd41378c0 con 0x7f1cd40a32b0 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 5 monclient: authenticate success, global_id 6114 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 10 monclient: _renew_subs 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd413c180 con 0x7f1cd40a32b0 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 10 monclient: _renew_subs 2024-09-16T05:40:00.766 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1cd413ac10 con 0x7f1cd40a32b0 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1cbc001be0 con 0x7f1cd40a32b0 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1cbc004c40 con 0x7f1cd40a32b0 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd1ffb640 4 set_mon_vals no callback set 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 10 monclient: dump: 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:00.767 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb8005090 con 0x7f1cd40a32b0 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1cbc00f9e0 con 0x7f1cd40a32b0 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(208..208 src has 20..208) ==== 12014+0+0 (unknown 1707261992 0 0) 0x7f1cbc0462e0 con 0x7f1cd40a32b0 2024-09-16T05:40:00.768 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.750+0000 7f1cd2ffd640 10 monclient: get_auth_request con 0x7f1cb4036a30 auth_method 0 2024-09-16T05:40:00.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.766+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1cbc015460 con 0x7f1cd40a32b0 2024-09-16T05:40:00.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.766+0000 7f1ccb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:00.769 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.766+0000 7f1ccb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:00.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.938+0000 7f1cd96bb640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:00.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.938+0000 7f1cd96bb640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:00.941 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.938+0000 7f1cd96bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:00.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:00.938+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f1cb8005380 con 0x7f1cd40a32b0 2024-09-16T05:40:01.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.046+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 added bucket h23 type host to location {rack=r1} v209) ==== 168+0+0 (unknown 3372914285 0 0) 0x7f1cbc014d60 con 0x7f1cd40a32b0 2024-09-16T05:40:01.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.046+0000 7f1ccb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:01.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.046+0000 7f1ccb7fe640 10 monclient: _finish_command 2 = system:0 added bucket h23 type host to location {rack=r1} 2024-09-16T05:40:01.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:01.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:01.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:01.153 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f1cb8005be0 con 0x7f1cd40a32b0 2024-09-16T05:40:01.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1ccb7fe640 1 -- 172.21.15.63:0/1381596840 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h23' already exists v209) ==== 147+0+0 (unknown 229129963 0 0) 0x7f1cbc0190b0 con 0x7f1cd40a32b0 2024-09-16T05:40:01.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1ccb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:01.154 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1ccb7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h23' already exists 2024-09-16T05:40:01.154 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h23' already exists 2024-09-16T05:40:01.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1cb4036a30 legacy=0x7f1cb4038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:01.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 10 monclient: shutdown 2024-09-16T05:40:01.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:01.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 >> v1:172.21.15.6:6789/0 conn(0x7f1cd40a32b0 legacy=0x7f1cd41d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:01.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 shutdown_connections 2024-09-16T05:40:01.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 >> 172.21.15.63:0/1381596840 conn(0x7f1cd40a3eb0 msgr2=0x7f1cd409c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:01.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.150+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 shutdown_connections 2024-09-16T05:40:01.157 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.154+0000 7f1cd96bb640 1 -- 172.21.15.63:0/1381596840 wait complete. 2024-09-16T05:40:01.171 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:01.171 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h24 host rack=r1 2024-09-16T05:40:01.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient: get_monmap_and_config 2024-09-16T05:40:01.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient: build_initial_monmap 2024-09-16T05:40:01.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient: monmap: 2024-09-16T05:40:01.289 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:01.289 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:01.290 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:01.289193+0000 2024-09-16T05:40:01.290 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:01.289193+0000 2024-09-16T05:40:01.290 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:01.290 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:01.290 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:01.290 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:01.290 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:01.290 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 1 Processor -- start 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 1 -- start start 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient: init 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient(hunting): picked mon.noname-b con 0x7efe780a1710 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient(hunting): picked mon.noname-c con 0x7efe7809aa60 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient(hunting): picked mon.noname-a con 0x7efe780a2200 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe7809b180 con 0x7efe780a2200 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe7809b740 con 0x7efe7809aa60 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe780730d0 con 0x7efe780a1710 2024-09-16T05:40:01.293 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7f197640 10 monclient(hunting): authenticate will time out at 1258.066969s 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe77fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7efe7809aa60 legacy=0x7efe7813bc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7cf33640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7efe780a1710 0x7efe780a1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47240/0 (socket says 172.21.15.63:47240) 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe7cf33640 1 -- 172.21.15.63:0/3921715494 learned_addr learned my addr 172.21.15.63:0/3921715494 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108572760 0 0) 0x7efe7809b180 con 0x7efe780a2200 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 10 monclient(hunting): my global_id is 6117 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe68003530 con 0x7efe780a2200 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4179389674 0 0) 0x7efe780730d0 con 0x7efe780a1710 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:01.294 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 10 monclient(hunting): my global_id is 6092 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.286+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe7809b180 con 0x7efe780a1710 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 872457070 0 0) 0x7efe68003530 con 0x7efe780a2200 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 >> v1:172.21.15.6:6790/0 conn(0x7efe780a1710 legacy=0x7efe780a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 >> v1:172.21.15.63:6789/0 conn(0x7efe7809aa60 legacy=0x7efe7813bc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 1 monclient: found mon.noname-a 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7806c720 con 0x7efe780a2200 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:01.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe700019a0 con 0x7efe780a2200 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: dump: 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:01.296 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 5 monclient: authenticate success, global_id 6117 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe70001c00 con 0x7efe780a2200 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 1 -- 172.21.15.63:0/3921715494 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe70001fd0 con 0x7efe780a2200 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe777fe640 10 monclient: dump: 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:01.297 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:01.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 4 set_mon_vals no callback set 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient: shutdown 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- 172.21.15.63:0/3921715494 >> v1:172.21.15.6:6789/0 conn(0x7efe780a2200 legacy=0x7efe7813f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- 172.21.15.63:0/3921715494 shutdown_connections 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- 172.21.15.63:0/3921715494 >> 172.21.15.63:0/3921715494 conn(0x7efe78075470 msgr2=0x7efe78075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- 172.21.15.63:0/3921715494 shutdown_connections 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- 172.21.15.63:0/3921715494 wait complete. 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient: build_initial_monmap 2024-09-16T05:40:01.303 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient: monmap: 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 Processor -- start 2024-09-16T05:40:01.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- start start 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient: init 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient(hunting): picked mon.noname-b con 0x7efe7809aa60 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient(hunting): picked mon.noname-a con 0x7efe780a1710 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient(hunting): picked mon.noname-c con 0x7efe780a2200 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7cf33640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7efe7809aa60 legacy=0x7efe7813b2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe7813c0e0 con 0x7efe780a1710 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe78134f30 con 0x7efe7809aa60 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe78134a00 con 0x7efe780a2200 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7f197640 10 monclient(hunting): authenticate will time out at 1258.071076s 2024-09-16T05:40:01.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7d734640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7efe780a2200 0x7efe781d39c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47246/0 (socket says 172.21.15.63:47246) 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe7d734640 1 -- 172.21.15.63:0/2291459692 learned_addr learned my addr 172.21.15.63:0/2291459692 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232529040 0 0) 0x7efe7813c0e0 con 0x7efe780a1710 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 10 monclient(hunting): my global_id is 6120 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe5c003530 con 0x7efe780a1710 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377898284 0 0) 0x7efe78134a00 con 0x7efe780a2200 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:01.306 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 10 monclient(hunting): my global_id is 6095 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.290+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe7813c0e0 con 0x7efe780a2200 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2414648984 0 0) 0x7efe5c003530 con 0x7efe780a1710 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe78134a00 con 0x7efe780a1710 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe640040d0 con 0x7efe780a1710 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient(hunting): dump: 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.307 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 116246720 0 0) 0x7efe7813c0e0 con 0x7efe780a2200 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe5c003530 con 0x7efe780a2200 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe70001f50 con 0x7efe780a2200 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient(hunting): dump: 2024-09-16T05:40:01.308 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2189326940 0 0) 0x7efe78134a00 con 0x7efe780a1710 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 >> v1:172.21.15.6:6790/0 conn(0x7efe780a2200 legacy=0x7efe781d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 >> v1:172.21.15.63:6789/0 conn(0x7efe7809aa60 legacy=0x7efe7813b2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:01.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient: _finish_hunting 0 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 monclient: found mon.a 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7813da40 con 0x7efe780a1710 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient: _finish_auth 0 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient: _check_auth_tickets 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 5 monclient: authenticate success, global_id 6120 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 10 monclient: _renew_subs 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe7813e930 con 0x7efe780a1710 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 10 monclient: _renew_subs 2024-09-16T05:40:01.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efe7813f7c0 con 0x7efe780a1710 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe64004330 con 0x7efe780a1710 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe76ffd640 4 set_mon_vals no callback set 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe64004680 con 0x7efe780a1710 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 10 monclient: dump: 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:01.311 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe58005090 con 0x7efe780a1710 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7efe640048a0 con 0x7efe780a1710 2024-09-16T05:40:01.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(209..209 src has 20..209) ==== 12227+0+0 (unknown 531037174 0 0) 0x7efe64046230 con 0x7efe780a1710 2024-09-16T05:40:01.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.294+0000 7efe7cf33640 10 monclient: get_auth_request con 0x7efe5c036a70 auth_method 0 2024-09-16T05:40:01.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.306+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7efe64015280 con 0x7efe780a1710 2024-09-16T05:40:01.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.306+0000 7efe757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:01.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.306+0000 7efe757fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.478+0000 7efe7f197640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.478+0000 7efe7f197640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.478+0000 7efe7f197640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:01.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:01.478+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7efe58005380 con 0x7efe780a1710 2024-09-16T05:40:02.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.070+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 added bucket h24 type host to location {rack=r1} v210) ==== 168+0+0 (unknown 3709632132 0 0) 0x7efe64018f30 con 0x7efe780a1710 2024-09-16T05:40:02.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.070+0000 7efe757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:02.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.070+0000 7efe757fa640 10 monclient: _finish_command 2 = system:0 added bucket h24 type host to location {rack=r1} 2024-09-16T05:40:02.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.170+0000 7efe7f197640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:02.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.170+0000 7efe7f197640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:02.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.170+0000 7efe7f197640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:02.176 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.170+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7efe58005ac0 con 0x7efe780a1710 2024-09-16T05:40:02.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe757fa640 1 -- 172.21.15.63:0/2291459692 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h24' already exists v210) ==== 147+0+0 (unknown 1829534906 0 0) 0x7efe6401d7f0 con 0x7efe780a1710 2024-09-16T05:40:02.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe757fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:02.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe757fa640 10 monclient: _finish_command 3 = system:0 bucket 'h24' already exists 2024-09-16T05:40:02.177 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h24' already exists 2024-09-16T05:40:02.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 >> v1:172.21.15.63:6812/3643207211 conn(0x7efe5c036a70 legacy=0x7efe5c038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:02.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe7f197640 10 monclient: shutdown 2024-09-16T05:40:02.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe7f197640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:02.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 >> v1:172.21.15.6:6789/0 conn(0x7efe780a1710 legacy=0x7efe7813b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:02.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 shutdown_connections 2024-09-16T05:40:02.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 >> 172.21.15.63:0/2291459692 conn(0x7efe78075470 msgr2=0x7efe781361e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:02.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 shutdown_connections 2024-09-16T05:40:02.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.174+0000 7efe7f197640 1 -- 172.21.15.63:0/2291459692 wait complete. 2024-09-16T05:40:02.195 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:02.195 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h25 host rack=r1 2024-09-16T05:40:02.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient: get_monmap_and_config 2024-09-16T05:40:02.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient: build_initial_monmap 2024-09-16T05:40:02.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient: monmap: 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:02.320909+0000 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:02.320909+0000 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:02.322 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:02.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 1 Processor -- start 2024-09-16T05:40:02.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 1 -- start start 2024-09-16T05:40:02.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient: init 2024-09-16T05:40:02.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:02.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:02.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient(hunting): picked mon.noname-a con 0x7f8b501339b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:02.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient(hunting): picked mon.noname-c con 0x7f8b5009fc50 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:02.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient(hunting): picked mon.noname-b con 0x7f8b5013d140 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:02.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b500a0710 con 0x7f8b501339b0 2024-09-16T05:40:02.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b500a09f0 con 0x7f8b5009fc50 2024-09-16T05:40:02.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b500a0cd0 con 0x7f8b5013d140 2024-09-16T05:40:02.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:02.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4e59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8b5009fc50 legacy=0x7f8b5013ba20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:02.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient(hunting): authenticate will time out at 1259.098052s 2024-09-16T05:40:02.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8b501339b0 0x7f8b50133d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38078/0 (socket says 172.21.15.63:38078) 2024-09-16T05:40:02.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4ed9d640 1 -- 172.21.15.63:0/2050498842 learned_addr learned my addr 172.21.15.63:0/2050498842 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:02.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937339125 0 0) 0x7f8b500a0710 con 0x7f8b501339b0 2024-09-16T05:40:02.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:02.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:02.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:02.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient(hunting): my global_id is 6123 2024-09-16T05:40:02.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b38003530 con 0x7f8b501339b0 2024-09-16T05:40:02.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4144064779 0 0) 0x7f8b500a0cd0 con 0x7f8b5013d140 2024-09-16T05:40:02.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:02.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:02.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:02.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient(hunting): my global_id is 6098 2024-09-16T05:40:02.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b500a0710 con 0x7f8b5013d140 2024-09-16T05:40:02.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2216422270 0 0) 0x7f8b38003530 con 0x7f8b501339b0 2024-09-16T05:40:02.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 >> v1:172.21.15.6:6790/0 conn(0x7f8b5013d140 legacy=0x7f8b5013f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:02.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 >> v1:172.21.15.63:6789/0 conn(0x7f8b5009fc50 legacy=0x7f8b5013ba20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:02.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:40:02.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 monclient: found mon.noname-a 2024-09-16T05:40:02.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:02.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:02.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b500a15b0 con 0x7f8b501339b0 2024-09-16T05:40:02.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: _finish_auth 0 2024-09-16T05:40:02.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:40:02.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:02.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b40001620 con 0x7f8b501339b0 2024-09-16T05:40:02.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:02.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:02.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: dump: 2024-09-16T05:40:02.342 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:02.342 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:02.342 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.342 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.343 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:02.343 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:02.343 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:02.343 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:02.343 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:02.343 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:02.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 5 monclient: authenticate success, global_id 6123 2024-09-16T05:40:02.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:02.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b40001940 con 0x7f8b501339b0 2024-09-16T05:40:02.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:02.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 1 -- 172.21.15.63:0/2050498842 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b40001d10 con 0x7f8b501339b0 2024-09-16T05:40:02.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:02.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:02.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b4dd9b640 10 monclient: dump: 2024-09-16T05:40:02.344 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:02.345 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:02.345 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.345 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.345 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:02.345 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:02.345 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:02.345 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:02.345 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.318+0000 7f8b55093640 4 set_mon_vals no callback set 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient: shutdown 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- 172.21.15.63:0/2050498842 >> v1:172.21.15.6:6789/0 conn(0x7f8b501339b0 legacy=0x7f8b50133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- 172.21.15.63:0/2050498842 shutdown_connections 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- 172.21.15.63:0/2050498842 >> 172.21.15.63:0/2050498842 conn(0x7f8b500a1e90 msgr2=0x7f8b50075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:02.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- 172.21.15.63:0/2050498842 shutdown_connections 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- 172.21.15.63:0/2050498842 wait complete. 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient: build_initial_monmap 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient: monmap: 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:02.347 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 Processor -- start 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- start start 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient: init 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient(hunting): picked mon.noname-c con 0x7f8b5009fc50 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:02.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient(hunting): picked mon.noname-b con 0x7f8b501339b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:02.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient(hunting): picked mon.noname-a con 0x7f8b5013d140 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:02.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b50133480 con 0x7f8b5013d140 2024-09-16T05:40:02.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b5012f840 con 0x7f8b501339b0 2024-09-16T05:40:02.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b501329b0 con 0x7f8b5009fc50 2024-09-16T05:40:02.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:02.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b4e59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8b501339b0 legacy=0x7f8b5012f130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:02.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b55093640 10 monclient(hunting): authenticate will time out at 1259.102658s 2024-09-16T05:40:02.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b4ed9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8b5009fc50 0x7f8b5012ea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47266/0 (socket says 172.21.15.63:47266) 2024-09-16T05:40:02.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b4ed9d640 1 -- 172.21.15.63:0/1843172303 learned_addr learned my addr 172.21.15.63:0/1843172303 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:02.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 34633759 0 0) 0x7f8b500a0710 con 0x7f8b5013d140 2024-09-16T05:40:02.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:02.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:02.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:02.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 10 monclient(hunting): my global_id is 6126 2024-09-16T05:40:02.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b3c003530 con 0x7f8b5013d140 2024-09-16T05:40:02.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729296474 0 0) 0x7f8b501329b0 con 0x7f8b5009fc50 2024-09-16T05:40:02.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:02.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:02.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:02.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 10 monclient(hunting): my global_id is 6101 2024-09-16T05:40:02.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.322+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b500a0710 con 0x7f8b5009fc50 2024-09-16T05:40:02.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3326599082 0 0) 0x7f8b500a0710 con 0x7f8b5009fc50 2024-09-16T05:40:02.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b501329b0 con 0x7f8b5009fc50 2024-09-16T05:40:02.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b40001440 con 0x7f8b5009fc50 2024-09-16T05:40:02.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:02.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:02.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient(hunting): dump: 2024-09-16T05:40:02.353 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:02.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1145253633 0 0) 0x7f8b3c003530 con 0x7f8b5013d140 2024-09-16T05:40:02.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b500a0710 con 0x7f8b5013d140 2024-09-16T05:40:02.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b48001c40 con 0x7f8b5013d140 2024-09-16T05:40:02.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:02.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:02.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient(hunting): dump: 2024-09-16T05:40:02.355 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3877351344 0 0) 0x7f8b501329b0 con 0x7f8b5009fc50 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 >> v1:172.21.15.63:6789/0 conn(0x7f8b501339b0 legacy=0x7f8b5012f130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 >> v1:172.21.15.6:6789/0 conn(0x7f8b5013d140 legacy=0x7f8b501d3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:02.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 monclient: found mon.c 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5013c9f0 con 0x7f8b5009fc50 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 5 monclient: authenticate success, global_id 6101 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 10 monclient: _renew_subs 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b50140920 con 0x7f8b5009fc50 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 10 monclient: _renew_subs 2024-09-16T05:40:02.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b501323f0 con 0x7f8b5009fc50 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b40001940 con 0x7f8b5009fc50 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b40004e70 con 0x7f8b5009fc50 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b4d59a640 4 set_mon_vals no callback set 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 10 monclient: dump: 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:02.358 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b30005090 con 0x7f8b5009fc50 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8b4000fc10 con 0x7f8b5009fc50 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(210..210 src has 20..210) ==== 12440+0+0 (unknown 1241609890 0 0) 0x7f8b40046490 con 0x7f8b5009fc50 2024-09-16T05:40:02.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.326+0000 7f8b4e59c640 10 monclient: get_auth_request con 0x7f8b3c036a70 auth_method 0 2024-09-16T05:40:02.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.338+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8b40015590 con 0x7f8b5009fc50 2024-09-16T05:40:02.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.338+0000 7f8b377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:02.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.338+0000 7f8b377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:02.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.506+0000 7f8b55093640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:02.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.506+0000 7f8b55093640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:02.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.506+0000 7f8b55093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:02.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:02.506+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f8b30005380 con 0x7f8b5009fc50 2024-09-16T05:40:03.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.078+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 added bucket h25 type host to location {rack=r1} v211) ==== 168+0+0 (unknown 2702113657 0 0) 0x7f8b40014e90 con 0x7f8b5009fc50 2024-09-16T05:40:03.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.078+0000 7f8b377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:03.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.078+0000 7f8b377fe640 10 monclient: _finish_command 2 = system:0 added bucket h25 type host to location {rack=r1} 2024-09-16T05:40:03.185 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.182+0000 7f8b55093640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:03.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.182+0000 7f8b55093640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:03.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.182+0000 7f8b55093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:03.186 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.182+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f8b30005ac0 con 0x7f8b5009fc50 2024-09-16T05:40:03.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.182+0000 7f8b377fe640 1 -- 172.21.15.63:0/1843172303 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h25' already exists v211) ==== 147+0+0 (unknown 1723491671 0 0) 0x7f8b40040740 con 0x7f8b5009fc50 2024-09-16T05:40:03.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.182+0000 7f8b377fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:03.188 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.182+0000 7f8b377fe640 10 monclient: _finish_command 3 = system:0 bucket 'h25' already exists 2024-09-16T05:40:03.188 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h25' already exists 2024-09-16T05:40:03.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.186+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8b3c036a70 legacy=0x7f8b3c038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:03.190 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.186+0000 7f8b55093640 10 monclient: shutdown 2024-09-16T05:40:03.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.186+0000 7f8b55093640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:03.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.186+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 >> v1:172.21.15.6:6790/0 conn(0x7f8b5009fc50 legacy=0x7f8b5012ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:03.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.186+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 shutdown_connections 2024-09-16T05:40:03.191 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.186+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 >> 172.21.15.63:0/1843172303 conn(0x7f8b500a1e90 msgr2=0x7f8b50072140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:03.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.186+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 shutdown_connections 2024-09-16T05:40:03.192 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.186+0000 7f8b55093640 1 -- 172.21.15.63:0/1843172303 wait complete. 2024-09-16T05:40:03.205 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:03.207 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h26 host rack=r1 2024-09-16T05:40:03.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.326+0000 7f9669171640 10 monclient: get_monmap_and_config 2024-09-16T05:40:03.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.326+0000 7f9669171640 10 monclient: build_initial_monmap 2024-09-16T05:40:03.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.326+0000 7f9669171640 10 monclient: monmap: 2024-09-16T05:40:03.331 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:03.331 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:03.332 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:03.330870+0000 2024-09-16T05:40:03.332 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:03.330870+0000 2024-09-16T05:40:03.332 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:03.332 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:03.332 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:03.333 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:03.333 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:03.333 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:03.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.326+0000 7f9669171640 1 Processor -- start 2024-09-16T05:40:03.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.326+0000 7f9669171640 1 -- start start 2024-09-16T05:40:03.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.326+0000 7f9669171640 10 monclient: init 2024-09-16T05:40:03.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:03.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:03.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient(hunting): picked mon.noname-c con 0x7f96640a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:03.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient(hunting): picked mon.noname-b con 0x7f96640a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:03.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient(hunting): picked mon.noname-a con 0x7f96640a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:03.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96640a51e0 con 0x7f96640a4790 2024-09-16T05:40:03.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96640a0d60 con 0x7f96640a3ca0 2024-09-16T05:40:03.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96640a1040 con 0x7f96640a4d90 2024-09-16T05:40:03.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:03.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient(hunting): authenticate will time out at 1260.108559s 2024-09-16T05:40:03.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9662d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f96640a3ca0 legacy=0x7f96640a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f966359e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f96640a4790 0x7f966413f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38084/0 (socket says 172.21.15.63:38084) 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f966359e640 1 -- 172.21.15.63:0/4013992517 learned_addr learned my addr 172.21.15.63:0/4013992517 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 805362674 0 0) 0x7f96640a1040 con 0x7f96640a4d90 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient(hunting): my global_id is 6104 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f964c003530 con 0x7f96640a4d90 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2397535857 0 0) 0x7f96640a51e0 con 0x7f96640a4790 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:03.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:03.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:03.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient(hunting): my global_id is 6129 2024-09-16T05:40:03.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96640a1040 con 0x7f96640a4790 2024-09-16T05:40:03.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2674183286 0 0) 0x7f964c003530 con 0x7f96640a4d90 2024-09-16T05:40:03.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 >> v1:172.21.15.63:6789/0 conn(0x7f96640a3ca0 legacy=0x7f96640a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:03.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 >> v1:172.21.15.6:6789/0 conn(0x7f96640a4790 legacy=0x7f966413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:03.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:40:03.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 monclient: found mon.noname-b 2024-09-16T05:40:03.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:03.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:03.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96640a0810 con 0x7f96640a4d90 2024-09-16T05:40:03.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: _finish_auth 0 2024-09-16T05:40:03.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:40:03.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:03.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9658001620 con 0x7f96640a4d90 2024-09-16T05:40:03.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:03.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:03.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: dump: 2024-09-16T05:40:03.352 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:03.352 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:03.352 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.352 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.352 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:03.353 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:03.353 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:03.353 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:03.353 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:03.353 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:03.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 5 monclient: authenticate success, global_id 6104 2024-09-16T05:40:03.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:03.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9658001880 con 0x7f96640a4d90 2024-09-16T05:40:03.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:03.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 1 -- 172.21.15.63:0/4013992517 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9658001bd0 con 0x7f96640a4d90 2024-09-16T05:40:03.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:03.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:03.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9661d9b640 10 monclient: dump: 2024-09-16T05:40:03.355 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:03.355 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:03.355 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.355 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.355 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:03.356 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:03.356 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:03.356 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:03.356 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:03.356 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:03.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:03.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 4 set_mon_vals no callback set 2024-09-16T05:40:03.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient: shutdown 2024-09-16T05:40:03.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:03.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 -- 172.21.15.63:0/4013992517 >> v1:172.21.15.6:6790/0 conn(0x7f96640a4d90 legacy=0x7f966413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:03.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 -- 172.21.15.63:0/4013992517 shutdown_connections 2024-09-16T05:40:03.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 -- 172.21.15.63:0/4013992517 >> 172.21.15.63:0/4013992517 conn(0x7f9664075470 msgr2=0x7f9664075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:03.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 -- 172.21.15.63:0/4013992517 shutdown_connections 2024-09-16T05:40:03.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 -- 172.21.15.63:0/4013992517 wait complete. 2024-09-16T05:40:03.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient: build_initial_monmap 2024-09-16T05:40:03.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 10 monclient: monmap: 2024-09-16T05:40:03.359 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:03.359 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:03.359 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:03.359 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:03.359 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:03.359 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:03.360 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:03.360 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:03.360 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:03.360 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:03.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.330+0000 7f9669171640 1 Processor -- start 2024-09-16T05:40:03.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 1 -- start start 2024-09-16T05:40:03.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: init 2024-09-16T05:40:03.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient(hunting): picked mon.noname-a con 0x7f96640a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient(hunting): picked mon.noname-b con 0x7f96640a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient(hunting): picked mon.noname-c con 0x7f96640a4d90 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9664138ac0 con 0x7f96640a3ca0 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9664138da0 con 0x7f96640a4790 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96640a8670 con 0x7f96640a4d90 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f966259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f96640a4790 legacy=0x7f9664135690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient(hunting): authenticate will time out at 1260.112565s 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f966359e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f96640a4d90 0x7f9664135da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:47292/0 (socket says 172.21.15.63:47292) 2024-09-16T05:40:03.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9662d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f96640a3ca0 0x7f9664134f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38092/0 (socket says 172.21.15.63:38092) 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f966359e640 1 -- 172.21.15.63:0/1636720890 learned_addr learned my addr 172.21.15.63:0/1636720890 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779104754 0 0) 0x7f96640a8670 con 0x7f96640a4d90 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): my global_id is 6107 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9650003530 con 0x7f96640a4d90 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266114866 0 0) 0x7f9664138ac0 con 0x7f96640a3ca0 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:03.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): my global_id is 6132 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96640a8670 con 0x7f96640a3ca0 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3248064823 0 0) 0x7f9650003530 con 0x7f96640a4d90 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9664138ac0 con 0x7f96640a4d90 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f965c001800 con 0x7f96640a4d90 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:03.363 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 343590122 0 0) 0x7f96640a8670 con 0x7f96640a3ca0 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9650003530 con 0x7f96640a3ca0 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9654004450 con 0x7f96640a3ca0 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:03.364 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1009608592 0 0) 0x7f9664138ac0 con 0x7f96640a4d90 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 >> v1:172.21.15.63:6789/0 conn(0x7f96640a4790 legacy=0x7f9664135690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 >> v1:172.21.15.6:6789/0 conn(0x7f96640a3ca0 legacy=0x7f9664134f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:03.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 monclient: found mon.c 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96641387e0 con 0x7f96640a4d90 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 5 monclient: authenticate success, global_id 6107 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: _renew_subs 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9664137260 con 0x7f96640a4d90 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: _renew_subs 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:03.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f96641d4b20 con 0x7f96640a4d90 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f965c0047f0 con 0x7f96640a4d90 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f965c004b40 con 0x7f96640a4d90 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 10 monclient: dump: 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:03.367 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f966159a640 4 set_mon_vals no callback set 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9644005090 con 0x7f96640a4d90 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.334+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f965c00f8e0 con 0x7f96640a4d90 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.338+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(211..211 src has 20..211) ==== 12653+0+0 (unknown 1734669865 0 0) 0x7f965c046520 con 0x7f96640a4d90 2024-09-16T05:40:03.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.338+0000 7f9662d9d640 10 monclient: get_auth_request con 0x7f96500369d0 auth_method 0 2024-09-16T05:40:03.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.350+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f965c015420 con 0x7f96640a4d90 2024-09-16T05:40:03.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.350+0000 7f964b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:03.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.350+0000 7f964b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:03.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.514+0000 7f9669171640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:03.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.514+0000 7f9669171640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:03.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.514+0000 7f9669171640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:03.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:03.514+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f9644005380 con 0x7f96640a4d90 2024-09-16T05:40:04.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.094+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 added bucket h26 type host to location {rack=r1} v212) ==== 168+0+0 (unknown 1086222997 0 0) 0x7f965c014d20 con 0x7f96640a4d90 2024-09-16T05:40:04.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.094+0000 7f964b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:04.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.094+0000 7f964b7fe640 10 monclient: _finish_command 2 = system:0 added bucket h26 type host to location {rack=r1} 2024-09-16T05:40:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.198+0000 7f9669171640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.198+0000 7f9669171640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.198+0000 7f9669171640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:04.203 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.198+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f9644005be0 con 0x7f96640a4d90 2024-09-16T05:40:04.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f964b7fe640 1 -- 172.21.15.63:0/1636720890 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h26' already exists v212) ==== 147+0+0 (unknown 684081330 0 0) 0x7f965c019070 con 0x7f96640a4d90 2024-09-16T05:40:04.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f964b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:04.205 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f964b7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h26' already exists 2024-09-16T05:40:04.205 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h26' already exists 2024-09-16T05:40:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 >> v1:172.21.15.63:6812/3643207211 conn(0x7f96500369d0 legacy=0x7f9650038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f9669171640 10 monclient: shutdown 2024-09-16T05:40:04.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f9669171640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 >> v1:172.21.15.6:6790/0 conn(0x7f96640a4d90 legacy=0x7f9664135da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 shutdown_connections 2024-09-16T05:40:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 >> 172.21.15.63:0/1636720890 conn(0x7f9664075470 msgr2=0x7f96640a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 shutdown_connections 2024-09-16T05:40:04.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.202+0000 7f9669171640 1 -- 172.21.15.63:0/1636720890 wait complete. 2024-09-16T05:40:04.223 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:04.223 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h27 host rack=r1 2024-09-16T05:40:04.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient: get_monmap_and_config 2024-09-16T05:40:04.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient: build_initial_monmap 2024-09-16T05:40:04.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient: monmap: 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:04.337314+0000 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:04.337314+0000 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:04.338 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:04.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 1 Processor -- start 2024-09-16T05:40:04.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 1 -- start start 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient: init 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient(hunting): picked mon.noname-b con 0x7f0bec0a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient(hunting): picked mon.noname-a con 0x7f0bec0a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient(hunting): picked mon.noname-c con 0x7f0bec0a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bec0a53b0 con 0x7f0bec0a07a0 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bec0a0290 con 0x7f0bec0a4600 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bec0a0dc0 con 0x7f0bec0a3b10 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bf10b0640 10 monclient(hunting): authenticate will time out at 1261.115050s 2024-09-16T05:40:04.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0beb59e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0bec0a4600 legacy=0x7f0bec13f550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:04.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bead9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0bec0a3b10 0x7f0bec0a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58894/0 (socket says 172.21.15.63:58894) 2024-09-16T05:40:04.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bead9d640 1 -- 172.21.15.63:0/1414595121 learned_addr learned my addr 172.21.15.63:0/1414595121 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:04.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0be3fff640 1 --1- 172.21.15.63:0/1414595121 >> v1:172.21.15.6:6789/0 conn(0x7f0bec0a07a0 0x7f0bec13be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44098/0 (socket says 172.21.15.63:44098) 2024-09-16T05:40:04.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925437766 0 0) 0x7f0bec0a0dc0 con 0x7f0bec0a3b10 2024-09-16T05:40:04.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 10 monclient(hunting): _negotiate 2024-09-16T05:40:04.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:04.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:04.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 10 monclient(hunting): my global_id is 6110 2024-09-16T05:40:04.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bd8003530 con 0x7f0bec0a3b10 2024-09-16T05:40:04.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2275707041 0 0) 0x7f0bec0a53b0 con 0x7f0bec0a07a0 2024-09-16T05:40:04.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 10 monclient(hunting): _negotiate 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 10 monclient(hunting): my global_id is 6135 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.334+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bec0a0dc0 con 0x7f0bec0a07a0 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 915315301 0 0) 0x7f0bd8003530 con 0x7f0bec0a3b10 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 >> v1:172.21.15.63:6789/0 conn(0x7f0bec0a4600 legacy=0x7f0bec13f550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 >> v1:172.21.15.6:6789/0 conn(0x7f0bec0a07a0 legacy=0x7f0bec13be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: _finish_hunting 0 2024-09-16T05:40:04.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 1 monclient: found mon.noname-b 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bec0a10a0 con 0x7f0bec0a3b10 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: _finish_auth 0 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: _check_auth_tickets 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bd4001620 con 0x7f0bec0a3b10 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:04.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: dump: 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:04.346 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:04.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 5 monclient: authenticate success, global_id 6110 2024-09-16T05:40:04.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:04.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bd4001850 con 0x7f0bec0a3b10 2024-09-16T05:40:04.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:04.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 1 -- 172.21.15.63:0/1414595121 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bd4001c60 con 0x7f0bec0a3b10 2024-09-16T05:40:04.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:04.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:04.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bea59c640 10 monclient: dump: 2024-09-16T05:40:04.348 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:04.348 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:04.348 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.348 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.349 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:04.349 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:04.349 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:04.349 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:04.349 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:04.349 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:04.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:04.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 4 set_mon_vals no callback set 2024-09-16T05:40:04.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient: shutdown 2024-09-16T05:40:04.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:04.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- 172.21.15.63:0/1414595121 >> v1:172.21.15.6:6790/0 conn(0x7f0bec0a3b10 legacy=0x7f0bec0a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:04.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- 172.21.15.63:0/1414595121 shutdown_connections 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- 172.21.15.63:0/1414595121 >> 172.21.15.63:0/1414595121 conn(0x7f0bec075470 msgr2=0x7f0bec075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- 172.21.15.63:0/1414595121 shutdown_connections 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- 172.21.15.63:0/1414595121 wait complete. 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient: build_initial_monmap 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient: monmap: 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:04.351 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:04.352 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:04.352 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:04.352 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:04.352 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:04.352 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:04.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 Processor -- start 2024-09-16T05:40:04.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- start start 2024-09-16T05:40:04.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient: init 2024-09-16T05:40:04.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:04.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:04.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient(hunting): picked mon.noname-a con 0x7f0bec0a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:04.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient(hunting): picked mon.noname-c con 0x7f0bec0a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:04.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient(hunting): picked mon.noname-b con 0x7f0bec0a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:04.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bec124270 con 0x7f0bec0a07a0 2024-09-16T05:40:04.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bec125c80 con 0x7f0bec0a4600 2024-09-16T05:40:04.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bec0a8940 con 0x7f0bec0a3b10 2024-09-16T05:40:04.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:04.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bf10b0640 10 monclient(hunting): authenticate will time out at 1261.119580s 2024-09-16T05:40:04.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0beb59e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0bec0a4600 legacy=0x7f0bec1d3b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:04.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bead9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0bec0a07a0 0x7f0bec122030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44110/0 (socket says 172.21.15.63:44110) 2024-09-16T05:40:04.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.338+0000 7f0bead9d640 1 -- 172.21.15.63:0/2024073313 learned_addr learned my addr 172.21.15.63:0/2024073313 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:04.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473091483 0 0) 0x7f0bec0a8940 con 0x7f0bec0a3b10 2024-09-16T05:40:04.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:04.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:04.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:04.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): my global_id is 6113 2024-09-16T05:40:04.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bcc003530 con 0x7f0bec0a3b10 2024-09-16T05:40:04.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1571483946 0 0) 0x7f0bec124270 con 0x7f0bec0a07a0 2024-09-16T05:40:04.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:04.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:04.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:04.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): my global_id is 6138 2024-09-16T05:40:04.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bec0a8940 con 0x7f0bec0a07a0 2024-09-16T05:40:04.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3087202634 0 0) 0x7f0bcc003530 con 0x7f0bec0a3b10 2024-09-16T05:40:04.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0bec124270 con 0x7f0bec0a3b10 2024-09-16T05:40:04.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bdc001950 con 0x7f0bec0a3b10 2024-09-16T05:40:04.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:04.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:04.361 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): dump: 2024-09-16T05:40:04.361 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:04.361 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:04.361 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.361 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.361 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:04.361 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:04.361 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:04.362 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:04.362 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:04.362 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:04.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:04.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2433234593 0 0) 0x7f0bec0a8940 con 0x7f0bec0a07a0 2024-09-16T05:40:04.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0bcc003530 con 0x7f0bec0a07a0 2024-09-16T05:40:04.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bd4001ba0 con 0x7f0bec0a07a0 2024-09-16T05:40:04.362 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient(hunting): dump: 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:04.363 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:04.364 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:04.364 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:04.364 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:04.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3789635038 0 0) 0x7f0bec124270 con 0x7f0bec0a3b10 2024-09-16T05:40:04.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 >> v1:172.21.15.63:6789/0 conn(0x7f0bec0a4600 legacy=0x7f0bec1d3b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:04.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 >> v1:172.21.15.6:6789/0 conn(0x7f0bec0a07a0 legacy=0x7f0bec122030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:04.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:04.364 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 monclient: found mon.c 2024-09-16T05:40:04.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:04.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:04.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bec0a83d0 con 0x7f0bec0a3b10 2024-09-16T05:40:04.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:04.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:04.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:04.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 5 monclient: authenticate success, global_id 6113 2024-09-16T05:40:04.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 10 monclient: _renew_subs 2024-09-16T05:40:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0bec0a0500 con 0x7f0bec0a3b10 2024-09-16T05:40:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 10 monclient: _renew_subs 2024-09-16T05:40:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0bec1259a0 con 0x7f0bec0a3b10 2024-09-16T05:40:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bdc0047f0 con 0x7f0bec0a3b10 2024-09-16T05:40:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bdc004b40 con 0x7f0bec0a3b10 2024-09-16T05:40:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 10 monclient: dump: 2024-09-16T05:40:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:04.368 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:04.368 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.368 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:04.368 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:04.368 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:04.368 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:04.368 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:04.368 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be9d9b640 4 set_mon_vals no callback set 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd0005090 con 0x7f0bec0a3b10 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0bdc00f8e0 con 0x7f0bec0a3b10 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(212..212 src has 20..212) ==== 12866+0+0 (unknown 4169091318 0 0) 0x7f0bdc046570 con 0x7f0bec0a3b10 2024-09-16T05:40:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.342+0000 7f0bead9d640 10 monclient: get_auth_request con 0x7f0bcc0369d0 auth_method 0 2024-09-16T05:40:04.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.354+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0bdc0153a0 con 0x7f0bec0a3b10 2024-09-16T05:40:04.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.354+0000 7f0be37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:04.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.354+0000 7f0be37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:04.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.518+0000 7f0bf10b0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:04.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.518+0000 7f0bf10b0640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:04.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.518+0000 7f0bf10b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:04.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:04.518+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f0bd0005380 con 0x7f0bec0a3b10 2024-09-16T05:40:05.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.102+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 added bucket h27 type host to location {rack=r1} v213) ==== 168+0+0 (unknown 1017968488 0 0) 0x7f0bdc014ca0 con 0x7f0bec0a3b10 2024-09-16T05:40:05.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.102+0000 7f0be37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:05.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.102+0000 7f0be37fe640 10 monclient: _finish_command 2 = system:0 added bucket h27 type host to location {rack=r1} 2024-09-16T05:40:05.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.202+0000 7f0bf10b0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:05.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.202+0000 7f0bf10b0640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:05.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.202+0000 7f0bf10b0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:05.208 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.202+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f0bd0005ac0 con 0x7f0bec0a3b10 2024-09-16T05:40:05.209 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0be37fe640 1 -- 172.21.15.63:0/2024073313 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h27' already exists v213) ==== 147+0+0 (unknown 594565471 0 0) 0x7f0bdc018ff0 con 0x7f0bec0a3b10 2024-09-16T05:40:05.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0be37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:05.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0be37fe640 10 monclient: _finish_command 3 = system:0 bucket 'h27' already exists 2024-09-16T05:40:05.210 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h27' already exists 2024-09-16T05:40:05.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0bcc0369d0 legacy=0x7f0bcc038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:05.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0bf10b0640 10 monclient: shutdown 2024-09-16T05:40:05.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0bf10b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:05.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 >> v1:172.21.15.6:6790/0 conn(0x7f0bec0a3b10 legacy=0x7f0bec122740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:05.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 shutdown_connections 2024-09-16T05:40:05.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 >> 172.21.15.63:0/2024073313 conn(0x7f0bec075470 msgr2=0x7f0bec1bcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:05.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 shutdown_connections 2024-09-16T05:40:05.212 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.206+0000 7f0bf10b0640 1 -- 172.21.15.63:0/2024073313 wait complete. 2024-09-16T05:40:05.227 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:05.227 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h28 host rack=r1 2024-09-16T05:40:05.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient: get_monmap_and_config 2024-09-16T05:40:05.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient: build_initial_monmap 2024-09-16T05:40:05.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient: monmap: 2024-09-16T05:40:05.351 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:05.351 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:05.351 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:05.350594+0000 2024-09-16T05:40:05.351 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:05.350594+0000 2024-09-16T05:40:05.352 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:05.352 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:05.352 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:05.352 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:05.352 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:05.353 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:05.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 1 Processor -- start 2024-09-16T05:40:05.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 1 -- start start 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient: init 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient(hunting): picked mon.noname-a con 0x7fc04c134930 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient(hunting): picked mon.noname-b con 0x7fc04c0a0810 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient(hunting): picked mon.noname-c con 0x7fc04c06dfa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c135420 con 0x7fc04c134930 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c09e830 con 0x7fc04c06dfa0 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c0a0dc0 con 0x7fc04c0a0810 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc0540b1640 10 monclient(hunting): authenticate will time out at 1262.127811s 2024-09-16T05:40:05.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.346+0000 7fc05264e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc04c06dfa0 legacy=0x7fc04c13cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc051e4d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc04c134930 0x7fc04c134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44118/0 (socket says 172.21.15.63:44118) 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc051e4d640 1 -- 172.21.15.63:0/23197197 learned_addr learned my addr 172.21.15.63:0/23197197 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc05164c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc04c0a0810 0x7fc04c139820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58914/0 (socket says 172.21.15.63:58914) 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1657249849 0 0) 0x7fc04c0a0dc0 con 0x7fc04c0a0810 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient(hunting): my global_id is 6116 2024-09-16T05:40:05.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc038003530 con 0x7fc04c0a0810 2024-09-16T05:40:05.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3918747847 0 0) 0x7fc04c135420 con 0x7fc04c134930 2024-09-16T05:40:05.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:05.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:05.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:05.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient(hunting): my global_id is 6141 2024-09-16T05:40:05.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc04c0a0dc0 con 0x7fc04c134930 2024-09-16T05:40:05.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3919408289 0 0) 0x7fc038003530 con 0x7fc04c0a0810 2024-09-16T05:40:05.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 >> v1:172.21.15.63:6789/0 conn(0x7fc04c06dfa0 legacy=0x7fc04c13cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 >> v1:172.21.15.6:6789/0 conn(0x7fc04c134930 legacy=0x7fc04c134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: _finish_hunting 0 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 monclient: found mon.noname-b 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04c0a2500 con 0x7fc04c0a0810 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: _finish_auth 0 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: _check_auth_tickets 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc048001550 con 0x7fc04c0a0810 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:05.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:05.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: dump: 2024-09-16T05:40:05.370 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:05.370 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:05.370 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.370 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.370 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:05.370 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:05.370 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:05.371 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:05.371 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:05.371 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:05.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 5 monclient: authenticate success, global_id 6116 2024-09-16T05:40:05.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:05.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc0480017b0 con 0x7fc04c0a0810 2024-09-16T05:40:05.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:05.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 1 -- 172.21.15.63:0/23197197 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc048001b80 con 0x7fc04c0a0810 2024-09-16T05:40:05.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:05.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:05.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 4 set_mon_vals no callback set 2024-09-16T05:40:05.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:05.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc050e4b640 10 monclient: dump: 2024-09-16T05:40:05.373 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:05.373 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:05.373 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.373 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.374 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:05.374 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:05.374 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:05.374 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:05.374 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:05.374 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:05.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient: shutdown 2024-09-16T05:40:05.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:05.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- 172.21.15.63:0/23197197 >> v1:172.21.15.6:6790/0 conn(0x7fc04c0a0810 legacy=0x7fc04c139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:05.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- 172.21.15.63:0/23197197 shutdown_connections 2024-09-16T05:40:05.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- 172.21.15.63:0/23197197 >> 172.21.15.63:0/23197197 conn(0x7fc04c0a1e90 msgr2=0x7fc04c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:05.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- 172.21.15.63:0/23197197 shutdown_connections 2024-09-16T05:40:05.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- 172.21.15.63:0/23197197 wait complete. 2024-09-16T05:40:05.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient: build_initial_monmap 2024-09-16T05:40:05.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient: monmap: 2024-09-16T05:40:05.376 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:05.376 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 Processor -- start 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- start start 2024-09-16T05:40:05.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient: init 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient(hunting): picked mon.noname-a con 0x7fc04c06dfa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient(hunting): picked mon.noname-b con 0x7fc04c0a0810 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient(hunting): picked mon.noname-c con 0x7fc04c134930 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c13de50 con 0x7fc04c06dfa0 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c1cc5b0 con 0x7fc04c0a0810 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04c1308f0 con 0x7fc04c134930 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc05164c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc04c0a0810 legacy=0x7fc04c1cbea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.350+0000 7fc0540b1640 10 monclient(hunting): authenticate will time out at 1262.131846s 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc051e4d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc04c06dfa0 0x7fc04c1cb790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44126/0 (socket says 172.21.15.63:44126) 2024-09-16T05:40:05.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc051e4d640 1 -- 172.21.15.63:0/4202955014 learned_addr learned my addr 172.21.15.63:0/4202955014 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3328951926 0 0) 0x7fc04c1308f0 con 0x7fc04c134930 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): my global_id is 6119 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc030003530 con 0x7fc04c134930 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370032279 0 0) 0x7fc04c0a0dc0 con 0x7fc04c06dfa0 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): my global_id is 6144 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc04c1308f0 con 0x7fc04c06dfa0 2024-09-16T05:40:05.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2603745393 0 0) 0x7fc030003530 con 0x7fc04c134930 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc04c0a0dc0 con 0x7fc04c134930 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc044003f40 con 0x7fc04c134930 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): dump: 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:05.380 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 747927830 0 0) 0x7fc04c1308f0 con 0x7fc04c06dfa0 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc030003530 con 0x7fc04c06dfa0 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc040001c80 con 0x7fc04c06dfa0 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient(hunting): dump: 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:05.381 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1567979842 0 0) 0x7fc04c0a0dc0 con 0x7fc04c134930 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 >> v1:172.21.15.63:6789/0 conn(0x7fc04c0a0810 legacy=0x7fc04c1cbea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:05.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 >> v1:172.21.15.6:6789/0 conn(0x7fc04c06dfa0 legacy=0x7fc04c1cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 monclient: found mon.c 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04c072110 con 0x7fc04c134930 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 5 monclient: authenticate success, global_id 6119 2024-09-16T05:40:05.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 10 monclient: _renew_subs 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc04c09c0f0 con 0x7fc04c134930 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 10 monclient: _renew_subs 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc04c132460 con 0x7fc04c134930 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc044004450 con 0x7fc04c134930 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03ffff640 4 set_mon_vals no callback set 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc0440047a0 con 0x7fc04c134930 2024-09-16T05:40:05.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 10 monclient: dump: 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:05.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:05.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:05.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:05.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc034005090 con 0x7fc04c134930 2024-09-16T05:40:05.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc044005990 con 0x7fc04c134930 2024-09-16T05:40:05.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(213..213 src has 20..213) ==== 13079+0+0 (unknown 648782810 0 0) 0x7fc044046370 con 0x7fc04c134930 2024-09-16T05:40:05.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.354+0000 7fc051e4d640 10 monclient: get_auth_request con 0x7fc030036aa0 auth_method 0 2024-09-16T05:40:05.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.366+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc044015150 con 0x7fc04c134930 2024-09-16T05:40:05.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.366+0000 7fc03e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:05.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.366+0000 7fc03e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:05.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.526+0000 7fc0540b1640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:05.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.526+0000 7fc0540b1640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:05.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.526+0000 7fc0540b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:05.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:05.526+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fc034005380 con 0x7fc04c134930 2024-09-16T05:40:06.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.118+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 added bucket h28 type host to location {rack=r1} v214) ==== 168+0+0 (unknown 219456684 0 0) 0x7fc044014a50 con 0x7fc04c134930 2024-09-16T05:40:06.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.118+0000 7fc03e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:06.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.118+0000 7fc03e7fc640 10 monclient: _finish_command 2 = system:0 added bucket h28 type host to location {rack=r1} 2024-09-16T05:40:06.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.226+0000 7fc0540b1640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:06.230 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.226+0000 7fc0540b1640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:06.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.226+0000 7fc0540b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:06.231 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.226+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fc034005ac0 con 0x7fc04c134930 2024-09-16T05:40:06.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc03e7fc640 1 -- 172.21.15.63:0/4202955014 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h28' already exists v214) ==== 147+0+0 (unknown 3140012994 0 0) 0x7fc04401d600 con 0x7fc04c134930 2024-09-16T05:40:06.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc03e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:06.232 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc03e7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h28' already exists 2024-09-16T05:40:06.232 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h28' already exists 2024-09-16T05:40:06.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc030036aa0 legacy=0x7fc030038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:06.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc0540b1640 10 monclient: shutdown 2024-09-16T05:40:06.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc0540b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:06.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 >> v1:172.21.15.6:6790/0 conn(0x7fc04c134930 legacy=0x7fc04c1d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:06.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 shutdown_connections 2024-09-16T05:40:06.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 >> 172.21.15.63:0/4202955014 conn(0x7fc04c0a1e90 msgr2=0x7fc04c12d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:06.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 shutdown_connections 2024-09-16T05:40:06.235 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.230+0000 7fc0540b1640 1 -- 172.21.15.63:0/4202955014 wait complete. 2024-09-16T05:40:06.250 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:06.250 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h29 host rack=r1 2024-09-16T05:40:06.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient: get_monmap_and_config 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient: build_initial_monmap 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient: monmap: 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:06.373721+0000 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:06.373721+0000 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:06.374 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 1 Processor -- start 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 1 -- start start 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient: init 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient(hunting): picked mon.noname-b con 0x7f94880a22a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient(hunting): picked mon.noname-a con 0x7f94880731d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient(hunting): picked mon.noname-c con 0x7f94881395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f948809c580 con 0x7f94880731d0 2024-09-16T05:40:06.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94880a1300 con 0x7f94881395a0 2024-09-16T05:40:06.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f948809d0f0 con 0x7f94880a22a0 2024-09-16T05:40:06.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:06.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f949036e640 10 monclient(hunting): authenticate will time out at 1263.150952s 2024-09-16T05:40:06.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948e90b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f94881395a0 legacy=0x7f948813b990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:06.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d909640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f94880731d0 0x7f948809fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44128/0 (socket says 172.21.15.63:44128) 2024-09-16T05:40:06.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d909640 1 -- 172.21.15.63:0/1128552726 learned_addr learned my addr 172.21.15.63:0/1128552726 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948e10a640 1 --1- 172.21.15.63:0/1128552726 >> v1:172.21.15.6:6790/0 conn(0x7f94880a22a0 0x7f948809f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58924/0 (socket says 172.21.15.63:58924) 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1756352602 0 0) 0x7f948809c580 con 0x7f94880731d0 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 10 monclient(hunting): _negotiate 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 10 monclient(hunting): my global_id is 6147 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9474003450 con 0x7f94880731d0 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 380528988 0 0) 0x7f948809d0f0 con 0x7f94880a22a0 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 10 monclient(hunting): _negotiate 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:06.379 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 10 monclient(hunting): my global_id is 6122 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.370+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f948809c580 con 0x7f94880a22a0 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3810100382 0 0) 0x7f9474003450 con 0x7f94880731d0 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 >> v1:172.21.15.6:6790/0 conn(0x7f94880a22a0 legacy=0x7f948809f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 >> v1:172.21.15.63:6789/0 conn(0x7f94881395a0 legacy=0x7f948813b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: _finish_hunting 0 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 1 monclient: found mon.noname-a 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948809d3d0 con 0x7f94880731d0 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: _finish_auth 0 2024-09-16T05:40:06.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: _check_auth_tickets 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9478001620 con 0x7f94880731d0 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: dump: 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:06.381 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 5 monclient: authenticate success, global_id 6147 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9478001880 con 0x7f94880731d0 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 1 -- 172.21.15.63:0/1128552726 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9478001c50 con 0x7f94880731d0 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948d108640 10 monclient: dump: 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:06.382 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:06.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 4 set_mon_vals no callback set 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient: shutdown 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- 172.21.15.63:0/1128552726 >> v1:172.21.15.6:6789/0 conn(0x7f94880731d0 legacy=0x7f948809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- 172.21.15.63:0/1128552726 shutdown_connections 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- 172.21.15.63:0/1128552726 >> 172.21.15.63:0/1128552726 conn(0x7f94880a1e90 msgr2=0x7f9488075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- 172.21.15.63:0/1128552726 shutdown_connections 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- 172.21.15.63:0/1128552726 wait complete. 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient: build_initial_monmap 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient: monmap: 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:06.388 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 Processor -- start 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- start start 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient: init 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:06.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient(hunting): picked mon.noname-c con 0x7f94880731d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient(hunting): picked mon.noname-a con 0x7f94880a22a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient(hunting): picked mon.noname-b con 0x7f94881395a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94880a42e0 con 0x7f94880a22a0 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f948811c220 con 0x7f94881395a0 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9488072a00 con 0x7f94880731d0 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f949036e640 10 monclient(hunting): authenticate will time out at 1263.155170s 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948e90b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f94881395a0 legacy=0x7f94881cefe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948e10a640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f94880731d0 0x7f94881c6c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58936/0 (socket says 172.21.15.63:58936) 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f948e10a640 1 -- 172.21.15.63:0/1261130888 learned_addr learned my addr 172.21.15.63:0/1261130888 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:06.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1563871232 0 0) 0x7f948809d3d0 con 0x7f94880a22a0 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f947effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f947effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f947effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.374+0000 7f947effd640 10 monclient(hunting): my global_id is 6150 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f946c003530 con 0x7f94880a22a0 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1321387086 0 0) 0x7f9488072a00 con 0x7f94880731d0 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): my global_id is 6125 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f948809d3d0 con 0x7f94880731d0 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2212391837 0 0) 0x7f946c003530 con 0x7f94880a22a0 2024-09-16T05:40:06.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9488072a00 con 0x7f94880a22a0 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2775260968 0 0) 0x7f948809d3d0 con 0x7f94880731d0 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f946c003530 con 0x7f94880731d0 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9478001bd0 con 0x7f94880a22a0 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): dump: 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:06.392 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9480001bd0 con 0x7f94880731d0 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient(hunting): dump: 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.393 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1561735767 0 0) 0x7f9488072a00 con 0x7f94880a22a0 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 >> v1:172.21.15.6:6790/0 conn(0x7f94880731d0 legacy=0x7f94881c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 >> v1:172.21.15.63:6789/0 conn(0x7f94881395a0 legacy=0x7f94881cefe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 monclient: found mon.a 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:06.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9488071c80 con 0x7f94880a22a0 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient: _finish_auth 0 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 5 monclient: authenticate success, global_id 6150 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 10 monclient: _renew_subs 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f948811bc60 con 0x7f94880a22a0 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 10 monclient: _renew_subs 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9488132430 con 0x7f94880a22a0 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9478001880 con 0x7f94880a22a0 2024-09-16T05:40:06.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9478004da0 con 0x7f94880a22a0 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f948c907640 4 set_mon_vals no callback set 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 10 monclient: dump: 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:06.396 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9470005090 con 0x7f94880a22a0 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f947800fb40 con 0x7f94880a22a0 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(214..214 src has 20..214) ==== 13292+0+0 (unknown 1640441057 0 0) 0x7f9478046680 con 0x7f94880a22a0 2024-09-16T05:40:06.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.378+0000 7f948e10a640 10 monclient: get_auth_request con 0x7f946c036a10 auth_method 0 2024-09-16T05:40:06.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.390+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f94880a42e0 con 0x7f94880a22a0 2024-09-16T05:40:06.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.390+0000 7f947effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:06.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.390+0000 7f947effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:06.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.558+0000 7f949036e640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:06.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.558+0000 7f949036e640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:06.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.558+0000 7f949036e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:06.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:06.558+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f9470005380 con 0x7f94880a22a0 2024-09-16T05:40:07.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.118+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 added bucket h29 type host to location {rack=r1} v215) ==== 168+0+0 (unknown 1896227153 0 0) 0x7f94780154a0 con 0x7f94880a22a0 2024-09-16T05:40:07.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.118+0000 7f947effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:07.120 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.118+0000 7f947effd640 10 monclient: _finish_command 2 = system:0 added bucket h29 type host to location {rack=r1} 2024-09-16T05:40:07.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.222+0000 7f949036e640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:07.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.222+0000 7f949036e640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:07.225 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.222+0000 7f949036e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:07.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.222+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f9470005ac0 con 0x7f94880a22a0 2024-09-16T05:40:07.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.222+0000 7f947effd640 1 -- 172.21.15.63:0/1261130888 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h29' already exists v215) ==== 147+0+0 (unknown 2963198511 0 0) 0x7f9478014d60 con 0x7f94880a22a0 2024-09-16T05:40:07.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.222+0000 7f947effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:07.226 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.222+0000 7f947effd640 10 monclient: _finish_command 3 = system:0 bucket 'h29' already exists 2024-09-16T05:40:07.226 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h29' already exists 2024-09-16T05:40:07.228 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.226+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 >> v1:172.21.15.63:6812/3643207211 conn(0x7f946c036a10 legacy=0x7f946c038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.226+0000 7f949036e640 10 monclient: shutdown 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.226+0000 7f949036e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.226+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 >> v1:172.21.15.6:6789/0 conn(0x7f94880a22a0 legacy=0x7f9488131b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.226+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 shutdown_connections 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.226+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 >> 172.21.15.63:0/1261130888 conn(0x7f94880a1e90 msgr2=0x7f94880a2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.226+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 shutdown_connections 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.226+0000 7f949036e640 1 -- 172.21.15.63:0/1261130888 wait complete. 2024-09-16T05:40:07.243 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:07.243 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h30 host rack=r1 2024-09-16T05:40:07.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient: get_monmap_and_config 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient: build_initial_monmap 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient: monmap: 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:07.365828+0000 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:07.365828+0000 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:07.366 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:07.367 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:07.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 1 Processor -- start 2024-09-16T05:40:07.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 1 -- start start 2024-09-16T05:40:07.368 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient: init 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient(hunting): picked mon.noname-c con 0x7f74280a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient(hunting): picked mon.noname-b con 0x7f74280a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient(hunting): picked mon.noname-a con 0x7f74280a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74280a53b0 con 0x7f74280a4600 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74280a0290 con 0x7f74280a3b10 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f742e029640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f74280a3b10 legacy=0x7f74280a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74280a0dc0 con 0x7f74280a07a0 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:07.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.362+0000 7f743028d640 10 monclient(hunting): authenticate will time out at 1264.143818s 2024-09-16T05:40:07.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d828640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f74280a07a0 0x7f742813be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58940/0 (socket says 172.21.15.63:58940) 2024-09-16T05:40:07.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742e82a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f74280a4600 0x7f742813f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44152/0 (socket says 172.21.15.63:44152) 2024-09-16T05:40:07.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d828640 1 -- 172.21.15.63:0/1306642681 learned_addr learned my addr 172.21.15.63:0/1306642681 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:07.371 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942990156 0 0) 0x7f74280a53b0 con 0x7f74280a4600 2024-09-16T05:40:07.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient(hunting): _negotiate 2024-09-16T05:40:07.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:07.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:07.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient(hunting): my global_id is 6153 2024-09-16T05:40:07.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7414003530 con 0x7f74280a4600 2024-09-16T05:40:07.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 539182665 0 0) 0x7f74280a0dc0 con 0x7f74280a07a0 2024-09-16T05:40:07.372 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient(hunting): _negotiate 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient(hunting): my global_id is 6128 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74280a53b0 con 0x7f74280a07a0 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4003608603 0 0) 0x7f74280a53b0 con 0x7f74280a07a0 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 >> v1:172.21.15.63:6789/0 conn(0x7f74280a3b10 legacy=0x7f74280a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 >> v1:172.21.15.6:6789/0 conn(0x7f74280a4600 legacy=0x7f742813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: _finish_hunting 0 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 monclient: found mon.noname-b 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:07.373 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74280a10a0 con 0x7f74280a07a0 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: _finish_auth 0 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: _check_auth_tickets 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7418001580 con 0x7f74280a07a0 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: dump: 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:07.374 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 5 monclient: authenticate success, global_id 6128 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7418001870 con 0x7f74280a07a0 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 1 -- 172.21.15.63:0/1306642681 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7418001c80 con 0x7f74280a07a0 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f742d027640 10 monclient: dump: 2024-09-16T05:40:07.375 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 4 set_mon_vals no callback set 2024-09-16T05:40:07.376 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 10 monclient: shutdown 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 1 -- 172.21.15.63:0/1306642681 >> v1:172.21.15.6:6790/0 conn(0x7f74280a07a0 legacy=0x7f742813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 1 -- 172.21.15.63:0/1306642681 shutdown_connections 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 1 -- 172.21.15.63:0/1306642681 >> 172.21.15.63:0/1306642681 conn(0x7f7428075470 msgr2=0x7f7428075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 1 -- 172.21.15.63:0/1306642681 shutdown_connections 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 1 -- 172.21.15.63:0/1306642681 wait complete. 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 10 monclient: build_initial_monmap 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 10 monclient: monmap: 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:07.377 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:07.378 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:07.378 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:07.378 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:07.378 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:07.378 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:07.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 1 Processor -- start 2024-09-16T05:40:07.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 1 -- start start 2024-09-16T05:40:07.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.366+0000 7f743028d640 10 monclient: init 2024-09-16T05:40:07.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:07.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:07.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient(hunting): picked mon.noname-b con 0x7f74280a07a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:07.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient(hunting): picked mon.noname-c con 0x7f74280a3b10 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient(hunting): picked mon.noname-a con 0x7f74280a4600 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7428127460 con 0x7f74280a4600 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7428127740 con 0x7f74280a07a0 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74280a8200 con 0x7f74280a3b10 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f742e029640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f74280a07a0 legacy=0x7f7428136d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient(hunting): authenticate will time out at 1264.148759s 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f742d828640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f74280a3b10 0x7f74281d0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58956/0 (socket says 172.21.15.63:58956) 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f742d828640 1 -- 172.21.15.63:0/3134416407 learned_addr learned my addr 172.21.15.63:0/3134416407 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286307256 0 0) 0x7f74280a8200 con 0x7f74280a3b10 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:07.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): my global_id is 6131 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f740c003560 con 0x7f74280a3b10 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8441756 0 0) 0x7f7428127460 con 0x7f74280a4600 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): my global_id is 6156 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74280a8200 con 0x7f74280a4600 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2205391800 0 0) 0x7f740c003560 con 0x7f74280a3b10 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7428127460 con 0x7f74280a3b10 2024-09-16T05:40:07.384 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4204424055 0 0) 0x7f74280a8200 con 0x7f74280a4600 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f740c003560 con 0x7f74280a4600 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f74180013a0 con 0x7f74280a3b10 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): dump: 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:07.385 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7420001b50 con 0x7f74280a4600 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient(hunting): dump: 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.386 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2861336173 0 0) 0x7f740c003560 con 0x7f74280a4600 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 >> v1:172.21.15.6:6790/0 conn(0x7f74280a3b10 legacy=0x7f74281d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 >> v1:172.21.15.63:6789/0 conn(0x7f74280a07a0 legacy=0x7f7428136d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 monclient: found mon.a 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:07.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74280a0500 con 0x7f74280a4600 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient: _finish_auth 0 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 5 monclient: authenticate success, global_id 6156 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient: _renew_subs 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7428126470 con 0x7f74280a4600 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient: _renew_subs 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f74281376c0 con 0x7f74280a4600 2024-09-16T05:40:07.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7420004b70 con 0x7f74280a4600 2024-09-16T05:40:07.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:07.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7420004ec0 con 0x7f74280a4600 2024-09-16T05:40:07.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:07.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f742c826640 4 set_mon_vals no callback set 2024-09-16T05:40:07.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:07.392 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f741effd640 10 monclient: dump: 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:07.393 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:07.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:07.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:07.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:07.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.370+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7410005090 con 0x7f74280a4600 2024-09-16T05:40:07.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.374+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f742000fce0 con 0x7f74280a4600 2024-09-16T05:40:07.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.374+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(215..215 src has 20..215) ==== 13505+0+0 (unknown 4102374493 0 0) 0x7f7420046aa0 con 0x7f74280a4600 2024-09-16T05:40:07.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.374+0000 7f742e029640 10 monclient: get_auth_request con 0x7f740c036b70 auth_method 0 2024-09-16T05:40:07.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.386+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7420015710 con 0x7f74280a4600 2024-09-16T05:40:07.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.386+0000 7f741effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:07.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.386+0000 7f741effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:07.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.550+0000 7f743028d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:07.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.550+0000 7f743028d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:07.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.550+0000 7f743028d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:07.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:07.550+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f7410005380 con 0x7f74280a4600 2024-09-16T05:40:08.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.142+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 added bucket h30 type host to location {rack=r1} v216) ==== 168+0+0 (unknown 3866985571 0 0) 0x7f7420015010 con 0x7f74280a4600 2024-09-16T05:40:08.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.142+0000 7f741effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:08.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.142+0000 7f741effd640 10 monclient: _finish_command 2 = system:0 added bucket h30 type host to location {rack=r1} 2024-09-16T05:40:08.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.242+0000 7f743028d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:08.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.242+0000 7f743028d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:08.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.242+0000 7f743028d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:08.246 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.242+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f7410005ac0 con 0x7f74280a4600 2024-09-16T05:40:08.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.242+0000 7f741effd640 1 -- 172.21.15.63:0/3134416407 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h30' already exists v216) ==== 147+0+0 (unknown 241038673 0 0) 0x7f742001db60 con 0x7f74280a4600 2024-09-16T05:40:08.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.242+0000 7f741effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:08.247 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.242+0000 7f741effd640 10 monclient: _finish_command 3 = system:0 bucket 'h30' already exists 2024-09-16T05:40:08.247 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h30' already exists 2024-09-16T05:40:08.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.246+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 >> v1:172.21.15.63:6812/3643207211 conn(0x7f740c036b70 legacy=0x7f740c039030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:08.249 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.246+0000 7f743028d640 10 monclient: shutdown 2024-09-16T05:40:08.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.246+0000 7f743028d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:08.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.246+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 >> v1:172.21.15.6:6789/0 conn(0x7f74280a4600 legacy=0x7f74281d3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:08.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.246+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 shutdown_connections 2024-09-16T05:40:08.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.246+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 >> 172.21.15.63:0/3134416407 conn(0x7f7428075470 msgr2=0x7f74280a7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:08.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.246+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 shutdown_connections 2024-09-16T05:40:08.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.246+0000 7f743028d640 1 -- 172.21.15.63:0/3134416407 wait complete. 2024-09-16T05:40:08.264 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:08.264 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h31 host rack=r1 2024-09-16T05:40:08.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient: get_monmap_and_config 2024-09-16T05:40:08.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient: build_initial_monmap 2024-09-16T05:40:08.389 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient: monmap: 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:08.389644+0000 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:08.389644+0000 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:08.390 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:08.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 1 Processor -- start 2024-09-16T05:40:08.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 1 -- start start 2024-09-16T05:40:08.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient: init 2024-09-16T05:40:08.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:08.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:08.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient(hunting): picked mon.noname-b con 0x7f973809c300 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:08.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient(hunting): picked mon.noname-a con 0x7f97380a09c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:08.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient(hunting): picked mon.noname-c con 0x7f973809d3a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:08.393 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97380a0ed0 con 0x7f97380a09c0 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973809d950 con 0x7f973809d3a0 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973809ebd0 con 0x7f973809c300 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973e4ca640 10 monclient(hunting): authenticate will time out at 1265.167474s 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f973ca67640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f973809d3a0 legacy=0x7f973813c150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f9737fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f973809c300 0x7f973809c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58960/0 (socket says 172.21.15.63:58960) 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f972ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f97380a09c0 0x7f9738138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44172/0 (socket says 172.21.15.63:44172) 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.386+0000 7f9737fff640 1 -- 172.21.15.63:0/251520476 learned_addr learned my addr 172.21.15.63:0/251520476 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3711403636 0 0) 0x7f97380a0ed0 con 0x7f97380a09c0 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:08.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient(hunting): my global_id is 6159 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9724005740 con 0x7f97380a09c0 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 432313752 0 0) 0x7f973809ebd0 con 0x7f973809c300 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient(hunting): my global_id is 6134 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97380a0ed0 con 0x7f973809c300 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2538644298 0 0) 0x7f9724005740 con 0x7f97380a09c0 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 >> v1:172.21.15.6:6790/0 conn(0x7f973809c300 legacy=0x7f973809c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 >> v1:172.21.15.63:6789/0 conn(0x7f973809d3a0 legacy=0x7f973813c150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:08.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 monclient: found mon.noname-a 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97380a2470 con 0x7f97380a09c0 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9720001620 con 0x7f97380a09c0 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: dump: 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:08.396 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 5 monclient: authenticate success, global_id 6159 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9720001880 con 0x7f97380a09c0 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:08.397 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 1 -- 172.21.15.63:0/251520476 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9720001c50 con 0x7f97380a09c0 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f97377fe640 10 monclient: dump: 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:08.398 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:08.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:08.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 4 set_mon_vals no callback set 2024-09-16T05:40:08.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient: shutdown 2024-09-16T05:40:08.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:08.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- 172.21.15.63:0/251520476 >> v1:172.21.15.6:6789/0 conn(0x7f97380a09c0 legacy=0x7f9738138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:08.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- 172.21.15.63:0/251520476 shutdown_connections 2024-09-16T05:40:08.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- 172.21.15.63:0/251520476 >> 172.21.15.63:0/251520476 conn(0x7f97380a1e90 msgr2=0x7f9738075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:08.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- 172.21.15.63:0/251520476 shutdown_connections 2024-09-16T05:40:08.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- 172.21.15.63:0/251520476 wait complete. 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient: build_initial_monmap 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient: monmap: 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:08.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 Processor -- start 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- start start 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient: init 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient(hunting): picked mon.noname-a con 0x7f973809c300 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient(hunting): picked mon.noname-c con 0x7f973809d3a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient(hunting): picked mon.noname-b con 0x7f97381c6d20 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9738121270 con 0x7f973809c300 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9738121550 con 0x7f97381c6d20 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97380a4590 con 0x7f973809d3a0 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973ca67640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f97381c6d20 legacy=0x7f97381cf790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f973e4ca640 10 monclient(hunting): authenticate will time out at 1265.171540s 2024-09-16T05:40:08.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f972ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f973809d3a0 0x7f97381cc060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58974/0 (socket says 172.21.15.63:58974) 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f9737fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f973809c300 0x7f9738131b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44186/0 (socket says 172.21.15.63:44186) 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.390+0000 7f972ffff640 1 -- 172.21.15.63:0/3807480330 learned_addr learned my addr 172.21.15.63:0/3807480330 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2935329266 0 0) 0x7f9738121270 con 0x7f973809c300 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): my global_id is 6162 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9718003530 con 0x7f973809c300 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 378873043 0 0) 0x7f97380a4590 con 0x7f973809d3a0 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:08.406 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): my global_id is 6137 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9738121270 con 0x7f973809d3a0 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1516859213 0 0) 0x7f9718003530 con 0x7f973809c300 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97380a4590 con 0x7f973809c300 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9728001760 con 0x7f973809c300 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): dump: 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:08.407 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4075773208 0 0) 0x7f9738121270 con 0x7f973809d3a0 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9718003530 con 0x7f973809d3a0 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9720001c10 con 0x7f973809d3a0 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient(hunting): dump: 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.408 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2587200270 0 0) 0x7f97380a4590 con 0x7f973809c300 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 >> v1:172.21.15.6:6790/0 conn(0x7f973809d3a0 legacy=0x7f97381cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 >> v1:172.21.15.63:6789/0 conn(0x7f97381c6d20 legacy=0x7f97381cf790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient: _finish_hunting 0 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 monclient: found mon.a 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:08.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973809c0f0 con 0x7f973809c300 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient: _finish_auth 0 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient: _check_auth_tickets 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 5 monclient: authenticate success, global_id 6162 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 10 monclient: _renew_subs 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f973811f260 con 0x7f973809c300 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 10 monclient: _renew_subs 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:08.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f97381d0eb0 con 0x7f973809c300 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9728001c90 con 0x7f973809c300 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9728004990 con 0x7f973809c300 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97367fc640 4 set_mon_vals no callback set 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 10 monclient: dump: 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:08.411 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f971c005090 con 0x7f973809c300 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9728004bf0 con 0x7f973809c300 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(216..216 src has 20..216) ==== 13718+0+0 (unknown 1312355113 0 0) 0x7f9728046710 con 0x7f973809c300 2024-09-16T05:40:08.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.394+0000 7f972ffff640 10 monclient: get_auth_request con 0x7f9718036ac0 auth_method 0 2024-09-16T05:40:08.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.406+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9728015250 con 0x7f973809c300 2024-09-16T05:40:08.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.406+0000 7f97357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:08.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.406+0000 7f97357fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:08.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.574+0000 7f973e4ca640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:08.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.574+0000 7f973e4ca640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:08.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.574+0000 7f973e4ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:08.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:08.574+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f971c005380 con 0x7f973809c300 2024-09-16T05:40:09.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.146+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 added bucket h31 type host to location {rack=r1} v217) ==== 168+0+0 (unknown 2590966174 0 0) 0x7f9728014b50 con 0x7f973809c300 2024-09-16T05:40:09.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.146+0000 7f97357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:09.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.146+0000 7f97357fa640 10 monclient: _finish_command 2 = system:0 added bucket h31 type host to location {rack=r1} 2024-09-16T05:40:09.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.246+0000 7f973e4ca640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:09.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.246+0000 7f973e4ca640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:09.250 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.246+0000 7f973e4ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:09.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.246+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f971c005ac0 con 0x7f973809c300 2024-09-16T05:40:09.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.246+0000 7f97357fa640 1 -- 172.21.15.63:0/3807480330 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h31' already exists v217) ==== 147+0+0 (unknown 99347644 0 0) 0x7f972801d700 con 0x7f973809c300 2024-09-16T05:40:09.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.246+0000 7f97357fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:09.251 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.246+0000 7f97357fa640 10 monclient: _finish_command 3 = system:0 bucket 'h31' already exists 2024-09-16T05:40:09.251 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h31' already exists 2024-09-16T05:40:09.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.250+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9718036ac0 legacy=0x7f9718038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:09.253 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.250+0000 7f973e4ca640 10 monclient: shutdown 2024-09-16T05:40:09.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.250+0000 7f973e4ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:09.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.250+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 >> v1:172.21.15.6:6789/0 conn(0x7f973809c300 legacy=0x7f9738131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:09.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.250+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 shutdown_connections 2024-09-16T05:40:09.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.250+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 >> 172.21.15.63:0/3807480330 conn(0x7f97380a1e90 msgr2=0x7f97380a2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:09.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.250+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 shutdown_connections 2024-09-16T05:40:09.254 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.250+0000 7f973e4ca640 1 -- 172.21.15.63:0/3807480330 wait complete. 2024-09-16T05:40:09.269 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:09.269 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush add-bucket h32 host rack=r1 2024-09-16T05:40:09.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 10 monclient: get_monmap_and_config 2024-09-16T05:40:09.394 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 10 monclient: build_initial_monmap 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 10 monclient: monmap: 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:09.394738+0000 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:09.394738+0000 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:09.395 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:09.398 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 1 Processor -- start 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 1 -- start start 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 10 monclient: init 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6c0a3b10 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.390+0000 7f4b717ec640 10 monclient(hunting): picked mon.noname-a con 0x7f4b6c0a07a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 10 monclient(hunting): picked mon.noname-b con 0x7f4b6c0a4600 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c0a53b0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b6affd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4b6c0a3b10 legacy=0x7f4b6c0a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:09.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c0a0290 con 0x7f4b6c0a3b10 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c0a0dc0 con 0x7f4b6c0a4600 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 10 monclient(hunting): authenticate will time out at 1266.172131s 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b6b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4b6c0a4600 0x7f4b6c13f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58976/0 (socket says 172.21.15.63:58976) 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b6a7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b6c0a07a0 0x7f4b6c13be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44192/0 (socket says 172.21.15.63:44192) 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b6b7fe640 1 -- 172.21.15.63:0/3384631993 learned_addr learned my addr 172.21.15.63:0/3384631993 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2407479889 0 0) 0x7f4b6c0a0dc0 con 0x7f4b6c0a4600 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient(hunting): my global_id is 6140 2024-09-16T05:40:09.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b54003530 con 0x7f4b6c0a4600 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001012765 0 0) 0x7f4b6c0a53b0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient(hunting): my global_id is 6165 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b6c0a0dc0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3152267328 0 0) 0x7f4b54003530 con 0x7f4b6c0a4600 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 >> v1:172.21.15.63:6789/0 conn(0x7f4b6c0a3b10 legacy=0x7f4b6c0a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 >> v1:172.21.15.6:6789/0 conn(0x7f4b6c0a07a0 legacy=0x7f4b6c13be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 monclient: found mon.noname-b 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:09.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c0a10a0 con 0x7f4b6c0a4600 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b640019a0 con 0x7f4b6c0a4600 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: dump: 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.402 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 5 monclient: authenticate success, global_id 6140 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b64001c50 con 0x7f4b6c0a4600 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 1 -- 172.21.15.63:0/3384631993 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b64002020 con 0x7f4b6c0a4600 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:09.403 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b69ffb640 10 monclient: dump: 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:09.404 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 4 set_mon_vals no callback set 2024-09-16T05:40:09.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 10 monclient: shutdown 2024-09-16T05:40:09.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:09.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- 172.21.15.63:0/3384631993 >> v1:172.21.15.6:6790/0 conn(0x7f4b6c0a4600 legacy=0x7f4b6c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:09.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- 172.21.15.63:0/3384631993 shutdown_connections 2024-09-16T05:40:09.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- 172.21.15.63:0/3384631993 >> 172.21.15.63:0/3384631993 conn(0x7f4b6c075470 msgr2=0x7f4b6c075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:09.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- 172.21.15.63:0/3384631993 shutdown_connections 2024-09-16T05:40:09.405 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- 172.21.15.63:0/3384631993 wait complete. 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 10 monclient: build_initial_monmap 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 10 monclient: monmap: 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:09.409 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 Processor -- start 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 1 -- start start 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.394+0000 7f4b717ec640 10 monclient: init 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6c0a07a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient(hunting): picked mon.noname-a con 0x7f4b6c0a3b10 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient(hunting): picked mon.noname-b con 0x7f4b6c0a4600 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c0a0500 con 0x7f4b6c0a3b10 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c125460 con 0x7f4b6c0a4600 2024-09-16T05:40:09.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6c0a8cd0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient(hunting): authenticate will time out at 1266.176545s 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b6b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f4b6c0a4600 legacy=0x7f4b6c1d3480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b6a7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b6c0a3b10 0x7f4b6c137050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44206/0 (socket says 172.21.15.63:44206) 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b6affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4b6c0a07a0 0x7f4b6c1cb100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58986/0 (socket says 172.21.15.63:58986) 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b6a7fc640 1 -- 172.21.15.63:0/3301841740 learned_addr learned my addr 172.21.15.63:0/3301841740 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724497711 0 0) 0x7f4b6c0a8cd0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): my global_id is 6143 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b58003530 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.411 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1836059219 0 0) 0x7f4b6c0a0dc0 con 0x7f4b6c0a3b10 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): my global_id is 6168 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b6c0a8cd0 con 0x7f4b6c0a3b10 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1368289801 0 0) 0x7f4b58003530 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b6c0a0dc0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b5c004000 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): dump: 2024-09-16T05:40:09.412 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2949249569 0 0) 0x7f4b6c0a8cd0 con 0x7f4b6c0a3b10 2024-09-16T05:40:09.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b58003530 con 0x7f4b6c0a3b10 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b600018b0 con 0x7f4b6c0a3b10 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient(hunting): dump: 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:09.414 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3946634922 0 0) 0x7f4b6c0a0dc0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 >> v1:172.21.15.63:6789/0 conn(0x7f4b6c0a4600 legacy=0x7f4b6c1d3480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 >> v1:172.21.15.6:6789/0 conn(0x7f4b6c0a3b10 legacy=0x7f4b6c137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 monclient: found mon.c 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c124010 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:09.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 5 monclient: authenticate success, global_id 6143 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient: _renew_subs 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c137760 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient: _renew_subs 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b6c1d4d70 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b5c004260 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:09.416 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b5c0045b0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 10 monclient: dump: 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:09.417 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b697fa640 4 set_mon_vals no callback set 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b4c005090 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.398+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4b5c0047d0 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.402+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(217..217 src has 20..217) ==== 13931+0+0 (unknown 1624570287 0 0) 0x7f4b5c046820 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.402+0000 7f4b6a7fc640 10 monclient: get_auth_request con 0x7f4b58036ac0 auth_method 0 2024-09-16T05:40:09.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.414+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4b5c015220 con 0x7f4b6c0a07a0 2024-09-16T05:40:09.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.414+0000 7f4b537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:09.419 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.414+0000 7f4b537fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:09.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.582+0000 7f4b717ec640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:09.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.582+0000 7f4b717ec640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:09.587 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.582+0000 7f4b717ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:09.588 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:09.582+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f4b4c005380 con 0x7f4b6c0a07a0 2024-09-16T05:40:10.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.174+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 added bucket h32 type host to location {rack=r1} v218) ==== 168+0+0 (unknown 509120409 0 0) 0x7f4b5c014b20 con 0x7f4b6c0a07a0 2024-09-16T05:40:10.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.174+0000 7f4b537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:10.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.174+0000 7f4b537fe640 10 monclient: _finish_command 2 = system:0 added bucket h32 type host to location {rack=r1} 2024-09-16T05:40:10.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.278+0000 7f4b717ec640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:10.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.278+0000 7f4b717ec640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:10.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.278+0000 7f4b717ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:10.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.278+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f4b4c005ac0 con 0x7f4b6c0a07a0 2024-09-16T05:40:10.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.278+0000 7f4b537fe640 1 -- 172.21.15.63:0/3301841740 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h32' already exists v218) ==== 147+0+0 (unknown 422692491 0 0) 0x7f4b5c018e70 con 0x7f4b6c0a07a0 2024-09-16T05:40:10.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.278+0000 7f4b537fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T05:40:10.284 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.278+0000 7f4b537fe640 10 monclient: _finish_command 3 = system:0 bucket 'h32' already exists 2024-09-16T05:40:10.284 INFO:tasks.workunit.client.0.smithi063.stderr:bucket 'h32' already exists 2024-09-16T05:40:10.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.282+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4b58036ac0 legacy=0x7f4b58038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:10.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.282+0000 7f4b717ec640 10 monclient: shutdown 2024-09-16T05:40:10.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.282+0000 7f4b717ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:10.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.282+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 >> v1:172.21.15.6:6790/0 conn(0x7f4b6c0a07a0 legacy=0x7f4b6c1cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:10.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.282+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 shutdown_connections 2024-09-16T05:40:10.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.282+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 >> 172.21.15.63:0/3301841740 conn(0x7f4b6c075470 msgr2=0x7f4b6c1bcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:10.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.282+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 shutdown_connections 2024-09-16T05:40:10.287 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.282+0000 7f4b717ec640 1 -- 172.21.15.63:0/3301841740 wait complete. 2024-09-16T05:40:10.302 INFO:tasks.workunit.client.0.smithi063.stderr:++ seq 1 32 2024-09-16T05:40:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:10.303 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h1 2024-09-16T05:40:10.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.430+0000 7fd0247d5640 10 monclient: get_monmap_and_config 2024-09-16T05:40:10.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.430+0000 7fd0247d5640 10 monclient: build_initial_monmap 2024-09-16T05:40:10.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.430+0000 7fd0247d5640 10 monclient: monmap: 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:10.434603+0000 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:10.434603+0000 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:10.435 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:10.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.430+0000 7fd0247d5640 1 Processor -- start 2024-09-16T05:40:10.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.430+0000 7fd0247d5640 1 -- start start 2024-09-16T05:40:10.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.430+0000 7fd0247d5640 10 monclient: init 2024-09-16T05:40:10.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:10.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:10.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c072750 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:10.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient(hunting): picked mon.noname-b con 0x7fd01c072b30 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:10.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c139e50 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:10.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c1341a0 con 0x7fd01c072750 2024-09-16T05:40:10.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c0706c0 con 0x7fd01c139e50 2024-09-16T05:40:10.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c0709a0 con 0x7fd01c072b30 2024-09-16T05:40:10.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:10.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient(hunting): authenticate will time out at 1267.212488s 2024-09-16T05:40:10.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd022d72640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd01c139e50 legacy=0x7fd01c13c240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:10.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd021d70640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd01c072b30 0x7fd01c138730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58994/0 (socket says 172.21.15.63:58994) 2024-09-16T05:40:10.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd021d70640 1 -- 172.21.15.63:0/2026312020 learned_addr learned my addr 172.21.15.63:0/2026312020 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:10.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131907665 0 0) 0x7fd01c0709a0 con 0x7fd01c072b30 2024-09-16T05:40:10.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient(hunting): _negotiate 2024-09-16T05:40:10.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient(hunting): my global_id is 6146 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd008003530 con 0x7fd01c072b30 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1203214709 0 0) 0x7fd01c1341a0 con 0x7fd01c072750 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient(hunting): _negotiate 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient(hunting): my global_id is 6171 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd01c0709a0 con 0x7fd01c072750 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2193098165 0 0) 0x7fd008003530 con 0x7fd01c072b30 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 >> v1:172.21.15.63:6789/0 conn(0x7fd01c139e50 legacy=0x7fd01c13c240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:10.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 >> v1:172.21.15.6:6789/0 conn(0x7fd01c072750 legacy=0x7fd01c133a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: _finish_hunting 0 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 monclient: found mon.noname-b 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c06d1e0 con 0x7fd01c072b30 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: _finish_auth 0 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: _check_auth_tickets 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 5 monclient: authenticate success, global_id 6146 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd00c001550 con 0x7fd01c072b30 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:10.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: dump: 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:10.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd00c001870 con 0x7fd01c072b30 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 1 -- 172.21.15.63:0/2026312020 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd00c001c40 con 0x7fd01c072b30 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd02156f640 10 monclient: dump: 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:10.445 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 4 set_mon_vals no callback set 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient: shutdown 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- 172.21.15.63:0/2026312020 >> v1:172.21.15.6:6790/0 conn(0x7fd01c072b30 legacy=0x7fd01c138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- 172.21.15.63:0/2026312020 shutdown_connections 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- 172.21.15.63:0/2026312020 >> 172.21.15.63:0/2026312020 conn(0x7fd01c0a2b80 msgr2=0x7fd01c06bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- 172.21.15.63:0/2026312020 shutdown_connections 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- 172.21.15.63:0/2026312020 wait complete. 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient: build_initial_monmap 2024-09-16T05:40:10.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient: monmap: 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 Processor -- start 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 1 -- start start 2024-09-16T05:40:10.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.434+0000 7fd0247d5640 10 monclient: init 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c072750 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient(hunting): picked mon.noname-b con 0x7fd01c072b30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c139e50 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c11d6c0 con 0x7fd01c139e50 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c11d9a0 con 0x7fd01c072b30 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c073550 con 0x7fd01c072750 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient(hunting): authenticate will time out at 1267.216517s 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd021d70640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd01c072b30 legacy=0x7fd01c1d0be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd022571640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd01c072750 0x7fd01c1cbc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59004/0 (socket says 172.21.15.63:59004) 2024-09-16T05:40:10.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd022571640 1 -- 172.21.15.63:0/442675153 learned_addr learned my addr 172.21.15.63:0/442675153 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd022d72640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd01c139e50 0x7fd01c1d4310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44226/0 (socket says 172.21.15.63:44226) 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179525818 0 0) 0x7fd01c073550 con 0x7fd01c072750 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): my global_id is 6149 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd000003530 con 0x7fd01c072750 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1186625963 0 0) 0x7fd01c11d6c0 con 0x7fd01c139e50 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): my global_id is 6174 2024-09-16T05:40:10.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd01c073550 con 0x7fd01c139e50 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 953547153 0 0) 0x7fd01c073550 con 0x7fd01c139e50 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd01c11d6c0 con 0x7fd01c139e50 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd014004280 con 0x7fd01c139e50 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.452 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2742992897 0 0) 0x7fd000003530 con 0x7fd01c072750 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd01c073550 con 0x7fd01c072750 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd0180018a0 con 0x7fd01c072750 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:10.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 843269017 0 0) 0x7fd01c073550 con 0x7fd01c072750 2024-09-16T05:40:10.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 >> v1:172.21.15.63:6789/0 conn(0x7fd01c072b30 legacy=0x7fd01c1d0be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 >> v1:172.21.15.6:6789/0 conn(0x7fd01c139e50 legacy=0x7fd01c1d4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 monclient: found mon.c 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c13d5b0 con 0x7fd01c072750 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 5 monclient: authenticate success, global_id 6149 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient: _renew_subs 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:10.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd01c13d260 con 0x7fd01c072750 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient: _renew_subs 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd01c11d3e0 con 0x7fd01c072750 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd018003620 con 0x7fd01c072750 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd018003970 con 0x7fd01c072750 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd013fff640 4 set_mon_vals no callback set 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd012ffd640 10 monclient: dump: 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:10.456 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.438+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd004005090 con 0x7fd01c072750 2024-09-16T05:40:10.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.442+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd018003b10 con 0x7fd01c072750 2024-09-16T05:40:10.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.442+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(218..218 src has 20..218) ==== 14144+0+0 (unknown 4098116787 0 0) 0x7fd018046a30 con 0x7fd01c072750 2024-09-16T05:40:10.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.442+0000 7fd021d70640 10 monclient: get_auth_request con 0x7fd000036a30 auth_method 0 2024-09-16T05:40:10.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.454+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd018015360 con 0x7fd01c072750 2024-09-16T05:40:10.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.454+0000 7fd012ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:10.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.454+0000 7fd012ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.626+0000 7fd0247d5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T05:40:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.626+0000 7fd0247d5640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T05:40:10.631 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.626+0000 7fd0247d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:10.632 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:10.626+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) -- 0x7fd004005380 con 0x7fd01c072750 2024-09-16T05:40:11.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.058+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 removed item id -19 name 'h1' from crush map v219) ==== 118+0+0 (unknown 3171800231 0 0) 0x7fd018014c60 con 0x7fd01c072750 2024-09-16T05:40:11.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.058+0000 7fd012ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T05:40:11.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.058+0000 7fd012ffd640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'h1' from crush map 2024-09-16T05:40:11.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.166+0000 7fd0247d5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T05:40:11.169 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.166+0000 7fd0247d5640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T05:40:11.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.166+0000 7fd0247d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:11.170 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.166+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) -- 0x7fd0040010a0 con 0x7fd01c072750 2024-09-16T05:40:11.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.174+0000 7fd012ffd640 1 -- 172.21.15.63:0/442675153 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 device 'h1' does not appear in the crush map v219) ==== 118+0+0 (unknown 2176747362 0 0) 0x7fd018018fb0 con 0x7fd01c072750 2024-09-16T05:40:11.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.174+0000 7fd012ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T05:40:11.180 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.174+0000 7fd012ffd640 10 monclient: _finish_command 3 = system:0 device 'h1' does not appear in the crush map 2024-09-16T05:40:11.181 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h1' does not appear in the crush map 2024-09-16T05:40:11.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.178+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd000036a30 legacy=0x7fd000038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:11.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.178+0000 7fd0247d5640 10 monclient: shutdown 2024-09-16T05:40:11.182 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.178+0000 7fd0247d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:11.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.178+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 >> v1:172.21.15.6:6790/0 conn(0x7fd01c072750 legacy=0x7fd01c1cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:11.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.178+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 shutdown_connections 2024-09-16T05:40:11.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.178+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 >> 172.21.15.63:0/442675153 conn(0x7fd01c0a2b80 msgr2=0x7fd01c0a44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:11.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.178+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 shutdown_connections 2024-09-16T05:40:11.183 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.178+0000 7fd0247d5640 1 -- 172.21.15.63:0/442675153 wait complete. 2024-09-16T05:40:11.198 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:11.198 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h2 2024-09-16T05:40:11.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.314+0000 7f6d881dc640 10 monclient: get_monmap_and_config 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.314+0000 7f6d881dc640 10 monclient: build_initial_monmap 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.314+0000 7f6d881dc640 10 monclient: monmap: 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:11.318904+0000 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:11.318904+0000 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:11.319 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:11.320 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:11.320 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.314+0000 7f6d881dc640 1 Processor -- start 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.314+0000 7f6d881dc640 1 -- start start 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.314+0000 7f6d881dc640 10 monclient: init 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient(hunting): picked mon.noname-c con 0x7f6d800a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient(hunting): picked mon.noname-a con 0x7f6d800a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient(hunting): picked mon.noname-b con 0x7f6d800a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:11.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d800a4fd0 con 0x7f6d800a08a0 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d800a5b40 con 0x7f6d800a3ca0 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d800a1a70 con 0x7f6d800a4790 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient(hunting): authenticate will time out at 1268.096340s 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d85f78640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6d800a3ca0 legacy=0x7f6d800a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d85777640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6d800a08a0 0x7f6d8013beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44236/0 (socket says 172.21.15.63:44236) 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d85777640 1 -- 172.21.15.63:0/766173593 learned_addr learned my addr 172.21.15.63:0/766173593 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d86779640 1 --1- 172.21.15.63:0/766173593 >> v1:172.21.15.6:6790/0 conn(0x7f6d800a4790 0x7f6d8013f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59018/0 (socket says 172.21.15.63:59018) 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225464724 0 0) 0x7f6d800a4fd0 con 0x7f6d800a08a0 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient(hunting): _negotiate 2024-09-16T05:40:11.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient(hunting): my global_id is 6177 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d6c003530 con 0x7f6d800a08a0 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2571932693 0 0) 0x7f6d800a1a70 con 0x7f6d800a4790 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient(hunting): _negotiate 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient(hunting): my global_id is 6152 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d800a4fd0 con 0x7f6d800a4790 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4001696743 0 0) 0x7f6d6c003530 con 0x7f6d800a08a0 2024-09-16T05:40:11.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 >> v1:172.21.15.6:6790/0 conn(0x7f6d800a4790 legacy=0x7f6d8013f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 >> v1:172.21.15.63:6789/0 conn(0x7f6d800a3ca0 legacy=0x7f6d800a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: _finish_hunting 0 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 monclient: found mon.noname-a 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d800a53e0 con 0x7f6d800a08a0 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: _finish_auth 0 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: _check_auth_tickets 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d70001620 con 0x7f6d800a08a0 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:11.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: dump: 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 5 monclient: authenticate success, global_id 6177 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:11.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d700018d0 con 0x7f6d800a08a0 2024-09-16T05:40:11.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:11.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 1 -- 172.21.15.63:0/766173593 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d70001ca0 con 0x7f6d800a08a0 2024-09-16T05:40:11.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:11.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 4 set_mon_vals no callback set 2024-09-16T05:40:11.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d84f76640 10 monclient: dump: 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:11.330 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient: shutdown 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- 172.21.15.63:0/766173593 >> v1:172.21.15.6:6789/0 conn(0x7f6d800a08a0 legacy=0x7f6d8013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- 172.21.15.63:0/766173593 shutdown_connections 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- 172.21.15.63:0/766173593 >> 172.21.15.63:0/766173593 conn(0x7f6d800a67c0 msgr2=0x7f6d80075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- 172.21.15.63:0/766173593 shutdown_connections 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- 172.21.15.63:0/766173593 wait complete. 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient: build_initial_monmap 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient: monmap: 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:11.331 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 Processor -- start 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 1 -- start start 2024-09-16T05:40:11.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.318+0000 7f6d881dc640 10 monclient: init 2024-09-16T05:40:11.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:11.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:11.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient(hunting): picked mon.noname-c con 0x7f6d800a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:11.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient(hunting): picked mon.noname-a con 0x7f6d800a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:11.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient(hunting): picked mon.noname-b con 0x7f6d800a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d801274f0 con 0x7f6d800a3ca0 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d801277d0 con 0x7f6d800a4790 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d800a8170 con 0x7f6d800a08a0 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d86779640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6d800a4790 legacy=0x7f6d801d39e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient(hunting): authenticate will time out at 1268.100438s 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d85f78640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6d800a08a0 0x7f6d80136de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59028/0 (socket says 172.21.15.63:59028) 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d85f78640 1 -- 172.21.15.63:0/3424788956 learned_addr learned my addr 172.21.15.63:0/3424788956 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d85777640 1 --1- 172.21.15.63:0/3424788956 >> v1:172.21.15.6:6789/0 conn(0x7f6d800a3ca0 0x7f6d801d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44252/0 (socket says 172.21.15.63:44252) 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336147417 0 0) 0x7f6d801274f0 con 0x7f6d800a3ca0 2024-09-16T05:40:11.337 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): my global_id is 6180 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d64003530 con 0x7f6d800a3ca0 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2056416337 0 0) 0x7f6d800a8170 con 0x7f6d800a08a0 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): my global_id is 6155 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d801274f0 con 0x7f6d800a08a0 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4262047698 0 0) 0x7f6d64003530 con 0x7f6d800a3ca0 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d800a8170 con 0x7f6d800a3ca0 2024-09-16T05:40:11.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d70001c20 con 0x7f6d800a3ca0 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): dump: 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:11.339 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3956099916 0 0) 0x7f6d801274f0 con 0x7f6d800a08a0 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d64003530 con 0x7f6d800a08a0 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d7c004090 con 0x7f6d800a08a0 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient(hunting): dump: 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:11.340 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3776289948 0 0) 0x7f6d800a8170 con 0x7f6d800a3ca0 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 >> v1:172.21.15.6:6790/0 conn(0x7f6d800a08a0 legacy=0x7f6d80136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 >> v1:172.21.15.63:6789/0 conn(0x7f6d800a4790 legacy=0x7f6d801d39e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:11.341 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 monclient: found mon.a 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d800a0500 con 0x7f6d800a3ca0 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 5 monclient: authenticate success, global_id 6180 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient: _renew_subs 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d80126500 con 0x7f6d800a3ca0 2024-09-16T05:40:11.342 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient: _renew_subs 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d80137750 con 0x7f6d800a3ca0 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d700018d0 con 0x7f6d800a3ca0 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d70004c50 con 0x7f6d800a3ca0 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d77fff640 4 set_mon_vals no callback set 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 10 monclient: dump: 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:11.343 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d68005090 con 0x7f6d800a3ca0 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6d7000f9f0 con 0x7f6d800a3ca0 2024-09-16T05:40:11.344 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(219..219 src has 20..219) ==== 13934+0+0 (unknown 2913708703 0 0) 0x7f6d70046a10 con 0x7f6d800a3ca0 2024-09-16T05:40:11.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.322+0000 7f6d85f78640 10 monclient: get_auth_request con 0x7f6d64036aa0 auth_method 0 2024-09-16T05:40:11.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.338+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6d70015470 con 0x7f6d800a3ca0 2024-09-16T05:40:11.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.338+0000 7f6d767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:11.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.338+0000 7f6d767fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:11.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.502+0000 7f6d881dc640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T05:40:11.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.506+0000 7f6d881dc640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T05:40:11.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.506+0000 7f6d881dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:11.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:11.506+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) -- 0x7f6d68005380 con 0x7f6d800a3ca0 2024-09-16T05:40:12.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.178+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 removed item id -22 name 'h2' from crush map v220) ==== 118+0+0 (unknown 3400328333 0 0) 0x7f6d70014d70 con 0x7f6d800a3ca0 2024-09-16T05:40:12.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.178+0000 7f6d767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T05:40:12.179 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.178+0000 7f6d767fc640 10 monclient: _finish_command 2 = system:0 removed item id -22 name 'h2' from crush map 2024-09-16T05:40:12.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.282+0000 7f6d881dc640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T05:40:12.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.282+0000 7f6d881dc640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T05:40:12.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.282+0000 7f6d881dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:12.285 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.282+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) -- 0x7f6d680010a0 con 0x7f6d800a3ca0 2024-09-16T05:40:12.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.282+0000 7f6d767fc640 1 -- 172.21.15.63:0/3424788956 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 device 'h2' does not appear in the crush map v220) ==== 118+0+0 (unknown 3481878774 0 0) 0x7f6d7001d920 con 0x7f6d800a3ca0 2024-09-16T05:40:12.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.282+0000 7f6d767fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T05:40:12.286 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.282+0000 7f6d767fc640 10 monclient: _finish_command 3 = system:0 device 'h2' does not appear in the crush map 2024-09-16T05:40:12.286 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h2' does not appear in the crush map 2024-09-16T05:40:12.288 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.286+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6d64036aa0 legacy=0x7f6d64038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:12.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.286+0000 7f6d881dc640 10 monclient: shutdown 2024-09-16T05:40:12.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.286+0000 7f6d881dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:12.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.286+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 >> v1:172.21.15.6:6789/0 conn(0x7f6d800a3ca0 legacy=0x7f6d801d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:12.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.286+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 shutdown_connections 2024-09-16T05:40:12.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.286+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 >> 172.21.15.63:0/3424788956 conn(0x7f6d800a67c0 msgr2=0x7f6d800a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:12.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.286+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 shutdown_connections 2024-09-16T05:40:12.289 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.286+0000 7f6d881dc640 1 -- 172.21.15.63:0/3424788956 wait complete. 2024-09-16T05:40:12.303 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:12.303 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h3 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient: get_monmap_and_config 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient: build_initial_monmap 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient: monmap: 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:12.417882+0000 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:12.417882+0000 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:12.418 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:12.419 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:12.419 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:12.421 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 1 Processor -- start 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 1 -- start start 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient: init 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient(hunting): picked mon.noname-c con 0x7f97780a12b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient(hunting): picked mon.noname-a con 0x7f9778072fa0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient(hunting): picked mon.noname-b con 0x7f97781395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97780a05b0 con 0x7f9778072fa0 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f977809c510 con 0x7f97780a12b0 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f977809ce10 con 0x7f97781395a0 2024-09-16T05:40:12.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977dc07640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f97780a12b0 legacy=0x7f977809f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977fe6b640 10 monclient(hunting): authenticate will time out at 1269.194866s 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977d406640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9778072fa0 0x7f977809fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44256/0 (socket says 172.21.15.63:44256) 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977d406640 1 -- 172.21.15.63:0/2242331822 learned_addr learned my addr 172.21.15.63:0/2242331822 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1374059181 0 0) 0x7f97780a05b0 con 0x7f9778072fa0 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 10 monclient(hunting): _negotiate 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 10 monclient(hunting): my global_id is 6183 2024-09-16T05:40:12.423 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9764003450 con 0x7f9778072fa0 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092096865 0 0) 0x7f977809ce10 con 0x7f97781395a0 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 10 monclient(hunting): _negotiate 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 10 monclient(hunting): my global_id is 6158 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.414+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97780a05b0 con 0x7f97781395a0 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3260228833 0 0) 0x7f9764003450 con 0x7f9778072fa0 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 >> v1:172.21.15.6:6790/0 conn(0x7f97781395a0 legacy=0x7f977813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 >> v1:172.21.15.63:6789/0 conn(0x7f97780a12b0 legacy=0x7f977809f790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: _finish_hunting 0 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 1 monclient: found mon.noname-a 2024-09-16T05:40:12.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977809d0f0 con 0x7f9778072fa0 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: _finish_auth 0 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: _check_auth_tickets 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9768001620 con 0x7f9778072fa0 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: dump: 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.425 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 5 monclient: authenticate success, global_id 6183 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97680018d0 con 0x7f9778072fa0 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 1 -- 172.21.15.63:0/2242331822 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9768001ca0 con 0x7f9778072fa0 2024-09-16T05:40:12.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 4 set_mon_vals no callback set 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977cc05640 10 monclient: dump: 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:12.427 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient: shutdown 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- 172.21.15.63:0/2242331822 >> v1:172.21.15.6:6789/0 conn(0x7f9778072fa0 legacy=0x7f977809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- 172.21.15.63:0/2242331822 shutdown_connections 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- 172.21.15.63:0/2242331822 >> 172.21.15.63:0/2242331822 conn(0x7f97780a1c90 msgr2=0x7f97780a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- 172.21.15.63:0/2242331822 shutdown_connections 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- 172.21.15.63:0/2242331822 wait complete. 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient: build_initial_monmap 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient: monmap: 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:12.428 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:12.429 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:12.429 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:12.429 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:12.429 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:12.429 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:12.429 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:12.429 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:12.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 Processor -- start 2024-09-16T05:40:12.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- start start 2024-09-16T05:40:12.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient: init 2024-09-16T05:40:12.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:12.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:12.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient(hunting): picked mon.noname-b con 0x7f9778072fa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:12.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient(hunting): picked mon.noname-a con 0x7f97780a12b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:12.433 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient(hunting): picked mon.noname-c con 0x7f97781395a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f977811f010 con 0x7f97780a12b0 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97780a3f20 con 0x7f9778072fa0 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977dc07640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f9778072fa0 legacy=0x7f9778134940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f977811ed30 con 0x7f97781395a0 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977fe6b640 10 monclient(hunting): authenticate will time out at 1269.198611s 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977d406640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f97780a12b0 0x7f97781cbdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44258/0 (socket says 172.21.15.63:44258) 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977d406640 1 -- 172.21.15.63:0/685638384 learned_addr learned my addr 172.21.15.63:0/685638384 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f977e408640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f97781395a0 0x7f97781cf4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59034/0 (socket says 172.21.15.63:59034) 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1042207709 0 0) 0x7f977811f010 con 0x7f97780a12b0 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:12.434 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): my global_id is 6186 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f975c003530 con 0x7f97780a12b0 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635612904 0 0) 0x7f977811ed30 con 0x7f97781395a0 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): my global_id is 6161 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f977811f010 con 0x7f97781395a0 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2685551380 0 0) 0x7f975c003530 con 0x7f97780a12b0 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f977811ed30 con 0x7f97780a12b0 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9768001c60 con 0x7f97780a12b0 2024-09-16T05:40:12.435 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.418+0000 7f976e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient(hunting): dump: 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:12.436 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 403099844 0 0) 0x7f977811f010 con 0x7f97781395a0 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f975c003530 con 0x7f97781395a0 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9770001bc0 con 0x7f97781395a0 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient(hunting): dump: 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:12.437 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1930974243 0 0) 0x7f977811ed30 con 0x7f97780a12b0 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 >> v1:172.21.15.6:6790/0 conn(0x7f97781395a0 legacy=0x7f97781cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 >> v1:172.21.15.63:6789/0 conn(0x7f9778072fa0 legacy=0x7f9778134940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 monclient: found mon.a 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977811f2f0 con 0x7f97780a12b0 2024-09-16T05:40:12.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 5 monclient: authenticate success, global_id 6186 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 10 monclient: _renew_subs 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f977811d8e0 con 0x7f97780a12b0 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 10 monclient: _renew_subs 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f97781c6a90 con 0x7f97780a12b0 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97680011c0 con 0x7f97780a12b0 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9768004af0 con 0x7f97780a12b0 2024-09-16T05:40:12.439 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 10 monclient: dump: 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:12.440 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:12.441 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:12.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976ffff640 4 set_mon_vals no callback set 2024-09-16T05:40:12.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:12.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:12.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:12.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9760005090 con 0x7f97780a12b0 2024-09-16T05:40:12.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9768004d10 con 0x7f97780a12b0 2024-09-16T05:40:12.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(220..220 src has 20..220) ==== 13724+0+0 (unknown 4229414947 0 0) 0x7f97680468d0 con 0x7f97780a12b0 2024-09-16T05:40:12.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.422+0000 7f977dc07640 10 monclient: get_auth_request con 0x7f975c036ae0 auth_method 0 2024-09-16T05:40:12.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.434+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9768018ff0 con 0x7f97780a12b0 2024-09-16T05:40:12.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.434+0000 7f976e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:12.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.434+0000 7f976e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:12.602 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.598+0000 7f977fe6b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T05:40:12.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.598+0000 7f977fe6b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T05:40:12.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.598+0000 7f977fe6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:12.603 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:12.598+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) -- 0x7f9760005380 con 0x7f97780a12b0 2024-09-16T05:40:13.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.194+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 removed item id -25 name 'h3' from crush map v221) ==== 118+0+0 (unknown 1060063834 0 0) 0x7f976801d8b0 con 0x7f97780a12b0 2024-09-16T05:40:13.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.194+0000 7f976e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T05:40:13.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.194+0000 7f976e7fc640 10 monclient: _finish_command 2 = system:0 removed item id -25 name 'h3' from crush map 2024-09-16T05:40:13.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T05:40:13.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T05:40:13.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:13.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) -- 0x7f97600010a0 con 0x7f97780a12b0 2024-09-16T05:40:13.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f976e7fc640 1 -- 172.21.15.63:0/685638384 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 device 'h3' does not appear in the crush map v221) ==== 118+0+0 (unknown 216406514 0 0) 0x7f9768040430 con 0x7f97780a12b0 2024-09-16T05:40:13.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f976e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T05:40:13.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f976e7fc640 10 monclient: _finish_command 3 = system:0 device 'h3' does not appear in the crush map 2024-09-16T05:40:13.298 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h3' does not appear in the crush map 2024-09-16T05:40:13.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 >> v1:172.21.15.63:6812/3643207211 conn(0x7f975c036ae0 legacy=0x7f975c038fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:13.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 10 monclient: shutdown 2024-09-16T05:40:13.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:13.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 >> v1:172.21.15.6:6789/0 conn(0x7f97780a12b0 legacy=0x7f97781cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:13.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 shutdown_connections 2024-09-16T05:40:13.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.294+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 >> 172.21.15.63:0/685638384 conn(0x7f97780a1c90 msgr2=0x7f97780a42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:13.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.298+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 shutdown_connections 2024-09-16T05:40:13.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.298+0000 7f977fe6b640 1 -- 172.21.15.63:0/685638384 wait complete. 2024-09-16T05:40:13.314 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:13.314 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h4 2024-09-16T05:40:13.437 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient: get_monmap_and_config 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient: build_initial_monmap 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient: monmap: 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:13.437733+0000 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:13.437733+0000 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:13.438 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:13.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 1 Processor -- start 2024-09-16T05:40:13.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 1 -- start start 2024-09-16T05:40:13.441 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient: init 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5c0a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5c09b450 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient(hunting): picked mon.noname-a con 0x7f2a5c09b8a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5c09e970 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5c09c370 con 0x7f2a5c0a08a0 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5c09d620 con 0x7f2a5c09b450 2024-09-16T05:40:13.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:13.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a61eb4640 10 monclient(hunting): authenticate will time out at 1270.214733s 2024-09-16T05:40:13.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2a5c0a08a0 legacy=0x7f2a5c09ad40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:13.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2a5c09b450 0x7f2a5c1389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:59042/0 (socket says 172.21.15.63:59042) 2024-09-16T05:40:13.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5affd640 1 -- 172.21.15.63:0/3704795620 learned_addr learned my addr 172.21.15.63:0/3704795620 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:13.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950714245 0 0) 0x7f2a5c09e970 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.443 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 10 monclient(hunting): my global_id is 6189 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a44003530 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2083343902 0 0) 0x7f2a5c09d620 con 0x7f2a5c09b450 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 10 monclient(hunting): my global_id is 6164 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.434+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a5c09e970 con 0x7f2a5c09b450 2024-09-16T05:40:13.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4160832676 0 0) 0x7f2a44003530 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 >> v1:172.21.15.6:6790/0 conn(0x7f2a5c09b450 legacy=0x7f2a5c1389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 >> v1:172.21.15.63:6789/0 conn(0x7f2a5c0a08a0 legacy=0x7f2a5c09ad40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 1 monclient: found mon.noname-a 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c09d900 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a540019a0 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:13.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: dump: 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:13.446 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 5 monclient: authenticate success, global_id 6189 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a54001c00 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 1 -- 172.21.15.63:0/3704795620 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a54001f50 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 4 set_mon_vals no callback set 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5a7fc640 10 monclient: dump: 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:13.447 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient: shutdown 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- 172.21.15.63:0/3704795620 >> v1:172.21.15.6:6789/0 conn(0x7f2a5c09b8a0 legacy=0x7f2a5c13c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:13.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- 172.21.15.63:0/3704795620 shutdown_connections 2024-09-16T05:40:13.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- 172.21.15.63:0/3704795620 >> 172.21.15.63:0/3704795620 conn(0x7f2a5c0a1c90 msgr2=0x7f2a5c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:13.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- 172.21.15.63:0/3704795620 shutdown_connections 2024-09-16T05:40:13.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- 172.21.15.63:0/3704795620 wait complete. 2024-09-16T05:40:13.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient: build_initial_monmap 2024-09-16T05:40:13.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient: monmap: 2024-09-16T05:40:13.452 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 Processor -- start 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- start start 2024-09-16T05:40:13.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient: init 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient(hunting): picked mon.noname-a con 0x7f2a5c09b450 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5c09b8a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5c0a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5c132520 con 0x7f2a5c09b450 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5c121590 con 0x7f2a5c09b8a0 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5c09c0f0 con 0x7f2a5c0a08a0 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a61eb4640 10 monclient(hunting): authenticate will time out at 1270.218682s 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5affd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f2a5c09b8a0 legacy=0x7f2a5c1cc0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:13.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2a5c09b450 0x7f2a5c131bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:44280/0 (socket says 172.21.15.63:44280) 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a5b7fe640 1 -- 172.21.15.63:0/1401439468 learned_addr learned my addr 172.21.15.63:0/1401439468 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684262004 0 0) 0x7f2a4c003aa0 con 0x7f2a5c09b450 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 10 monclient(hunting): my global_id is 6192 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a48003560 con 0x7f2a5c09b450 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 258740138 0 0) 0x7f2a5c09c0f0 con 0x7f2a5c0a08a0 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:13.455 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 10 monclient(hunting): my global_id is 6167 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.438+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a4c003aa0 con 0x7f2a5c0a08a0 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2686309624 0 0) 0x7f2a48003560 con 0x7f2a5c09b450 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a5c09c0f0 con 0x7f2a5c09b450 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a4c004330 con 0x7f2a5c09b450 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient(hunting): dump: 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.456 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2973961522 0 0) 0x7f2a4c003aa0 con 0x7f2a5c0a08a0 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a48003560 con 0x7f2a5c0a08a0 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a540018e0 con 0x7f2a5c0a08a0 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:13.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient(hunting): dump: 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3458349994 0 0) 0x7f2a5c09c0f0 con 0x7f2a5c09b450 2024-09-16T05:40:13.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 >> v1:172.21.15.6:6790/0 conn(0x7f2a5c0a08a0 legacy=0x7f2a5c1cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 >> v1:172.21.15.63:6789/0 conn(0x7f2a5c09b8a0 legacy=0x7f2a5c1cc0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 monclient: found mon.a 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c11eaa0 con 0x7f2a5c09b450 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient: _finish_auth 0 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 5 monclient: authenticate success, global_id 6192 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 10 monclient: _renew_subs 2024-09-16T05:40:13.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5c11fe00 con 0x7f2a5c09b450 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 10 monclient: _renew_subs 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a5c1322c0 con 0x7f2a5c09b450 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a4c004590 con 0x7f2a5c09b450 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a4c004960 con 0x7f2a5c09b450 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a597fa640 4 set_mon_vals no callback set 2024-09-16T05:40:13.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 10 monclient: dump: 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:13.461 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c005090 con 0x7f2a5c09b450 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2a4c004b00 con 0x7f2a5c09b450 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(221..221 src has 20..221) ==== 13514+0+0 (unknown 1646904892 0 0) 0x7f2a4c046910 con 0x7f2a5c09b450 2024-09-16T05:40:13.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.442+0000 7f2a5affd640 10 monclient: get_auth_request con 0x7f2a48036aa0 auth_method 0 2024-09-16T05:40:13.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.454+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2a5c132520 con 0x7f2a5c09b450 2024-09-16T05:40:13.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.454+0000 7f2a43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:13.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.454+0000 7f2a43fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:13.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.622+0000 7f2a61eb4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T05:40:13.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.622+0000 7f2a61eb4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T05:40:13.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.622+0000 7f2a61eb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:13.626 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:13.622+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) -- 0x7f2a3c005380 con 0x7f2a5c09b450 2024-09-16T05:40:14.197 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.194+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 removed item id -28 name 'h4' from crush map v222) ==== 118+0+0 (unknown 3292075316 0 0) 0x7f2a4c015510 con 0x7f2a5c09b450 2024-09-16T05:40:14.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.194+0000 7f2a43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T05:40:14.198 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.194+0000 7f2a43fff640 10 monclient: _finish_command 2 = system:0 removed item id -28 name 'h4' from crush map 2024-09-16T05:40:14.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.298+0000 7f2a61eb4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T05:40:14.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.298+0000 7f2a61eb4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T05:40:14.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.298+0000 7f2a61eb4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:14.301 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.298+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) -- 0x7f2a3c0010a0 con 0x7f2a5c09b450 2024-09-16T05:40:14.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.298+0000 7f2a43fff640 1 -- 172.21.15.63:0/1401439468 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 device 'h4' does not appear in the crush map v222) ==== 118+0+0 (unknown 1096294757 0 0) 0x7f2a4c01d9c0 con 0x7f2a5c09b450 2024-09-16T05:40:14.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.298+0000 7f2a43fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T05:40:14.302 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.298+0000 7f2a43fff640 10 monclient: _finish_command 3 = system:0 device 'h4' does not appear in the crush map 2024-09-16T05:40:14.302 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h4' does not appear in the crush map 2024-09-16T05:40:14.304 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.302+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2a48036aa0 legacy=0x7f2a48038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:14.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.302+0000 7f2a61eb4640 10 monclient: shutdown 2024-09-16T05:40:14.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.302+0000 7f2a61eb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:14.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.302+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 >> v1:172.21.15.6:6789/0 conn(0x7f2a5c09b450 legacy=0x7f2a5c131bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:14.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.302+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 shutdown_connections 2024-09-16T05:40:14.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.302+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 >> 172.21.15.63:0/1401439468 conn(0x7f2a5c0a1c90 msgr2=0x7f2a5c075410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:14.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.302+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 shutdown_connections 2024-09-16T05:40:14.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.302+0000 7f2a61eb4640 1 -- 172.21.15.63:0/1401439468 wait complete. 2024-09-16T05:40:14.320 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:14.320 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h5 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient: get_monmap_and_config 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient: build_initial_monmap 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient: monmap: 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:14.441992+0000 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:14.441992+0000 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:14.442 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:14.443 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:14.443 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 1 Processor -- start 2024-09-16T05:40:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 1 -- start start 2024-09-16T05:40:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient: init 2024-09-16T05:40:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient(hunting): picked mon.noname-c con 0x7f704c0a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient(hunting): picked mon.noname-a con 0x7f704c0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:14.444 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient(hunting): picked mon.noname-b con 0x7f704c0a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704c0a4fd0 con 0x7f704c0a08a0 2024-09-16T05:40:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704c0a5b40 con 0x7f704c0a3ca0 2024-09-16T05:40:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704c0a1a70 con 0x7f704c0a4790 2024-09-16T05:40:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f70508a3640 10 monclient(hunting): authenticate will time out at 1271.219715s 2024-09-16T05:40:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.438+0000 7f704a59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f704c0a3ca0 legacy=0x7f704c0a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f7049d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f704c0a08a0 0x7f704c13beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47626/0 (socket says 172.21.15.63:47626) 2024-09-16T05:40:14.445 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f7049d9b640 1 -- 172.21.15.63:0/3872587032 learned_addr learned my addr 172.21.15.63:0/3872587032 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2224606391 0 0) 0x7f704c0a1a70 con 0x7f704c0a4790 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient(hunting): my global_id is 6170 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7034003530 con 0x7f704c0a4790 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217080885 0 0) 0x7f704c0a4fd0 con 0x7f704c0a08a0 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:14.447 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient(hunting): my global_id is 6195 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f704c0a1a70 con 0x7f704c0a08a0 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1653756964 0 0) 0x7f704c0a1a70 con 0x7f704c0a08a0 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 >> v1:172.21.15.6:6790/0 conn(0x7f704c0a4790 legacy=0x7f704c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 >> v1:172.21.15.63:6789/0 conn(0x7f704c0a3ca0 legacy=0x7f704c0a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 monclient: found mon.noname-a 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704c0a53e0 con 0x7f704c0a08a0 2024-09-16T05:40:14.448 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: _finish_auth 0 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7040001530 con 0x7f704c0a08a0 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: dump: 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:14.449 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 5 monclient: authenticate success, global_id 6195 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7040001820 con 0x7f704c0a08a0 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 1 -- 172.21.15.63:0/3872587032 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7040001c30 con 0x7f704c0a08a0 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 4 set_mon_vals no callback set 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:14.450 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f704959a640 10 monclient: dump: 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 10 monclient: shutdown 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:14.451 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 1 -- 172.21.15.63:0/3872587032 >> v1:172.21.15.6:6789/0 conn(0x7f704c0a08a0 legacy=0x7f704c13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 1 -- 172.21.15.63:0/3872587032 shutdown_connections 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 1 -- 172.21.15.63:0/3872587032 >> 172.21.15.63:0/3872587032 conn(0x7f704c0a67c0 msgr2=0x7f704c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 1 -- 172.21.15.63:0/3872587032 shutdown_connections 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 1 -- 172.21.15.63:0/3872587032 wait complete. 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 10 monclient: build_initial_monmap 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 10 monclient: monmap: 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:14.452 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:14.453 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:14.453 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:14.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 1 Processor -- start 2024-09-16T05:40:14.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 1 -- start start 2024-09-16T05:40:14.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.442+0000 7f70508a3640 10 monclient: init 2024-09-16T05:40:14.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:14.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:14.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient(hunting): picked mon.noname-b con 0x7f704c0a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:14.456 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient(hunting): picked mon.noname-c con 0x7f704c0a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient(hunting): picked mon.noname-a con 0x7f704c0a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704c125b30 con 0x7f704c0a4790 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f704a59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f704c0a08a0 legacy=0x7f704c136d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704c127180 con 0x7f704c0a08a0 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704c0a8500 con 0x7f704c0a3ca0 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient(hunting): authenticate will time out at 1271.224175s 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f7049d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f704c0a3ca0 0x7f704c1d00c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48334/0 (socket says 172.21.15.63:48334) 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f7049d9b640 1 -- 172.21.15.63:0/3288778455 learned_addr learned my addr 172.21.15.63:0/3288778455 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348620091 0 0) 0x7f704c125b30 con 0x7f704c0a4790 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:14.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): my global_id is 6198 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f702c003530 con 0x7f704c0a4790 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1007607885 0 0) 0x7f704c0a8500 con 0x7f704c0a3ca0 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): my global_id is 6173 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f704c125b30 con 0x7f704c0a3ca0 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1560691870 0 0) 0x7f702c003530 con 0x7f704c0a4790 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f704c0a8500 con 0x7f704c0a4790 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70440018e0 con 0x7f704c0a4790 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:14.458 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): dump: 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 65667193 0 0) 0x7f704c125b30 con 0x7f704c0a3ca0 2024-09-16T05:40:14.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f702c003530 con 0x7f704c0a3ca0 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7040001530 con 0x7f704c0a3ca0 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient(hunting): dump: 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:14.460 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3611269796 0 0) 0x7f704c0a8500 con 0x7f704c0a4790 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 >> v1:172.21.15.6:6790/0 conn(0x7f704c0a3ca0 legacy=0x7f704c1d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 >> v1:172.21.15.63:6789/0 conn(0x7f704c0a08a0 legacy=0x7f704c136d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 monclient: found mon.a 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704c0a7fb0 con 0x7f704c0a4790 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient: _finish_auth 0 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 5 monclient: authenticate success, global_id 6198 2024-09-16T05:40:14.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient: _renew_subs 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f704c0a0500 con 0x7f704c0a4790 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient: _renew_subs 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f704c137460 con 0x7f704c0a4790 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7044003620 con 0x7f704c0a4790 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7044003970 con 0x7f704c0a4790 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f7048d99640 4 set_mon_vals no callback set 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 10 monclient: dump: 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:14.462 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7030005090 con 0x7f704c0a4790 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f704400e710 con 0x7f704c0a4790 2024-09-16T05:40:14.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(222..222 src has 20..222) ==== 13304+0+0 (unknown 656399620 0 0) 0x7f70440454f0 con 0x7f704c0a4790 2024-09-16T05:40:14.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.446+0000 7f704a59c640 10 monclient: get_auth_request con 0x7f702c036a20 auth_method 0 2024-09-16T05:40:14.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.462+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7044014170 con 0x7f704c0a4790 2024-09-16T05:40:14.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.462+0000 7f703affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:14.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.462+0000 7f703affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:14.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.630+0000 7f70508a3640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T05:40:14.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.630+0000 7f70508a3640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T05:40:14.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.630+0000 7f70508a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:14.633 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:14.630+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) -- 0x7f7030005380 con 0x7f704c0a4790 2024-09-16T05:40:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.210+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 removed item id -31 name 'h5' from crush map v223) ==== 118+0+0 (unknown 2767218704 0 0) 0x7f7044013a70 con 0x7f704c0a4790 2024-09-16T05:40:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.210+0000 7f703affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T05:40:15.213 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.210+0000 7f703affd640 10 monclient: _finish_command 2 = system:0 removed item id -31 name 'h5' from crush map 2024-09-16T05:40:15.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.314+0000 7f70508a3640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T05:40:15.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.314+0000 7f70508a3640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T05:40:15.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.314+0000 7f70508a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:15.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.314+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) -- 0x7f70300010a0 con 0x7f704c0a4790 2024-09-16T05:40:15.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.314+0000 7f703affd640 1 -- 172.21.15.63:0/3288778455 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 device 'h5' does not appear in the crush map v223) ==== 118+0+0 (unknown 2184673377 0 0) 0x7f7044017dc0 con 0x7f704c0a4790 2024-09-16T05:40:15.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.314+0000 7f703affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T05:40:15.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.314+0000 7f703affd640 10 monclient: _finish_command 3 = system:0 device 'h5' does not appear in the crush map 2024-09-16T05:40:15.319 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h5' does not appear in the crush map 2024-09-16T05:40:15.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.318+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 >> v1:172.21.15.63:6812/3643207211 conn(0x7f702c036a20 legacy=0x7f702c038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:15.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.318+0000 7f70508a3640 10 monclient: shutdown 2024-09-16T05:40:15.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.318+0000 7f70508a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:15.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.318+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 >> v1:172.21.15.6:6789/0 conn(0x7f704c0a4790 legacy=0x7f704c1d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:15.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.318+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 shutdown_connections 2024-09-16T05:40:15.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.318+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 >> 172.21.15.63:0/3288778455 conn(0x7f704c0a67c0 msgr2=0x7f704c0a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:15.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.318+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 shutdown_connections 2024-09-16T05:40:15.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.318+0000 7f70508a3640 1 -- 172.21.15.63:0/3288778455 wait complete. 2024-09-16T05:40:15.336 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:15.336 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h6 2024-09-16T05:40:15.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient: get_monmap_and_config 2024-09-16T05:40:15.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient: build_initial_monmap 2024-09-16T05:40:15.454 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient: monmap: 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:15.454668+0000 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:15.454668+0000 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:15.455 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 1 Processor -- start 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 1 -- start start 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient: init 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient(hunting): picked mon.noname-c con 0x7fbb040a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient(hunting): picked mon.noname-a con 0x7fbb040a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient(hunting): picked mon.noname-b con 0x7fbb040a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb040a4fd0 con 0x7fbb040a08a0 2024-09-16T05:40:15.459 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb040a5b40 con 0x7fbb040a3ca0 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb040a1a70 con 0x7fbb040a4790 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0c49f640 10 monclient(hunting): authenticate will time out at 1272.231687s 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.450+0000 7fbb0a23b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbb040a3ca0 legacy=0x7fbb040a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0aa3c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbb040a4790 0x7fbb0413f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48338/0 (socket says 172.21.15.63:48338) 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09a3a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbb040a08a0 0x7fbb0413beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47638/0 (socket says 172.21.15.63:47638) 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0aa3c640 1 -- 172.21.15.63:0/1652700361 learned_addr learned my addr 172.21.15.63:0/1652700361 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237891297 0 0) 0x7fbb040a4fd0 con 0x7fbb040a08a0 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient(hunting): _negotiate 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient(hunting): my global_id is 6201 2024-09-16T05:40:15.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaf0003530 con 0x7fbb040a08a0 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932655251 0 0) 0x7fbb040a1a70 con 0x7fbb040a4790 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient(hunting): _negotiate 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient(hunting): my global_id is 6176 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb040a4fd0 con 0x7fbb040a4790 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2027625615 0 0) 0x7fbb040a4fd0 con 0x7fbb040a4790 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 >> v1:172.21.15.63:6789/0 conn(0x7fbb040a3ca0 legacy=0x7fbb040a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 >> v1:172.21.15.6:6789/0 conn(0x7fbb040a08a0 legacy=0x7fbb0413beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: _finish_hunting 0 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 monclient: found mon.noname-b 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:15.461 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb040a53e0 con 0x7fbb040a4790 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: _finish_auth 0 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: _check_auth_tickets 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbafc001980 con 0x7fbb040a4790 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: dump: 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.462 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 5 monclient: authenticate success, global_id 6176 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbafc001be0 con 0x7fbb040a4790 2024-09-16T05:40:15.463 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:15.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 1 -- 172.21.15.63:0/1652700361 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbafc001f30 con 0x7fbb040a4790 2024-09-16T05:40:15.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:15.464 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:15.465 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb09239640 10 monclient: dump: 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:15.466 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 4 set_mon_vals no callback set 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient: shutdown 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- 172.21.15.63:0/1652700361 >> v1:172.21.15.6:6790/0 conn(0x7fbb040a4790 legacy=0x7fbb0413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- 172.21.15.63:0/1652700361 shutdown_connections 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- 172.21.15.63:0/1652700361 >> 172.21.15.63:0/1652700361 conn(0x7fbb040a67c0 msgr2=0x7fbb04075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- 172.21.15.63:0/1652700361 shutdown_connections 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- 172.21.15.63:0/1652700361 wait complete. 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient: build_initial_monmap 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient: monmap: 2024-09-16T05:40:15.467 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 Processor -- start 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- start start 2024-09-16T05:40:15.468 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient: init 2024-09-16T05:40:15.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:15.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:15.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient(hunting): picked mon.noname-a con 0x7fbb040a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:15.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient(hunting): picked mon.noname-c con 0x7fbb040a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:15.472 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient(hunting): picked mon.noname-b con 0x7fbb040a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:15.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb040a0500 con 0x7fbb040a08a0 2024-09-16T05:40:15.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb041256e0 con 0x7fbb040a4790 2024-09-16T05:40:15.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb040a7f00 con 0x7fbb040a3ca0 2024-09-16T05:40:15.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:15.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0c49f640 10 monclient(hunting): authenticate will time out at 1272.235670s 2024-09-16T05:40:15.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0aa3c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fbb040a4790 legacy=0x7fbb041d31e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:15.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0a23b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbb040a08a0 0x7fbb041cae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47642/0 (socket says 172.21.15.63:47642) 2024-09-16T05:40:15.473 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.454+0000 7fbb0a23b640 1 -- 172.21.15.63:0/245419225 learned_addr learned my addr 172.21.15.63:0/245419225 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 251401776 0 0) 0x7fbb00003aa0 con 0x7fbb040a08a0 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): my global_id is 6204 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae8003530 con 0x7fbb040a08a0 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2648168817 0 0) 0x7fbb040a7f00 con 0x7fbb040a3ca0 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:15.474 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): my global_id is 6179 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb00003aa0 con 0x7fbb040a3ca0 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 382947601 0 0) 0x7fbae8003530 con 0x7fbb040a08a0 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb040a7f00 con 0x7fbb040a08a0 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb00004330 con 0x7fbb040a08a0 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): dump: 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:15.475 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2226303551 0 0) 0x7fbb00003aa0 con 0x7fbb040a3ca0 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbae8003530 con 0x7fbb040a3ca0 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbaf4001970 con 0x7fbb040a3ca0 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:15.476 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:15.477 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient(hunting): dump: 2024-09-16T05:40:15.477 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:15.477 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:15.477 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.477 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.477 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:15.477 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:15.477 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:15.478 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:15.478 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:15.478 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:15.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3830993600 0 0) 0x7fbae8003530 con 0x7fbb040a3ca0 2024-09-16T05:40:15.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 >> v1:172.21.15.63:6789/0 conn(0x7fbb040a4790 legacy=0x7fbb041d31e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:15.478 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 >> v1:172.21.15.6:6789/0 conn(0x7fbb040a08a0 legacy=0x7fbb041cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 monclient: found mon.c 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb041233f0 con 0x7fbb040a3ca0 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 5 monclient: authenticate success, global_id 6179 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 10 monclient: _renew_subs 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:15.479 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb04125400 con 0x7fbb040a3ca0 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 10 monclient: _renew_subs 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb04137420 con 0x7fbb040a3ca0 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbaf4001c70 con 0x7fbb040a3ca0 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbaf4004c40 con 0x7fbb040a3ca0 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb08a38640 4 set_mon_vals no callback set 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 10 monclient: dump: 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:15.480 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:15.481 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.481 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:15.481 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:15.481 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:15.481 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:15.481 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaec005090 con 0x7fbb040a3ca0 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbaf400f9e0 con 0x7fbb040a3ca0 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(223..223 src has 20..223) ==== 13094+0+0 (unknown 3344427135 0 0) 0x7fbaf40466d0 con 0x7fbb040a3ca0 2024-09-16T05:40:15.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.458+0000 7fbb0a23b640 10 monclient: get_auth_request con 0x7fbae8036980 auth_method 0 2024-09-16T05:40:15.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.470+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbaf4015480 con 0x7fbb040a3ca0 2024-09-16T05:40:15.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.470+0000 7fbafaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:15.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.470+0000 7fbafaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:15.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.634+0000 7fbb0c49f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T05:40:15.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.634+0000 7fbb0c49f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T05:40:15.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.634+0000 7fbb0c49f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:15.637 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:15.634+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) -- 0x7fbaec005380 con 0x7fbb040a3ca0 2024-09-16T05:40:16.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.233+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 removed item id -34 name 'h6' from crush map v224) ==== 118+0+0 (unknown 4241156527 0 0) 0x7fbaf4014d80 con 0x7fbb040a3ca0 2024-09-16T05:40:16.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.233+0000 7fbafaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T05:40:16.240 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.233+0000 7fbafaffd640 10 monclient: _finish_command 2 = system:0 removed item id -34 name 'h6' from crush map 2024-09-16T05:40:16.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.337+0000 7fbb0c49f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T05:40:16.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.337+0000 7fbb0c49f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T05:40:16.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.337+0000 7fbb0c49f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:16.343 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.337+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) -- 0x7fbaec0010a0 con 0x7fbb040a3ca0 2024-09-16T05:40:16.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.341+0000 7fbafaffd640 1 -- 172.21.15.63:0/245419225 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 device 'h6' does not appear in the crush map v224) ==== 118+0+0 (unknown 2586816971 0 0) 0x7fbaf401d930 con 0x7fbb040a3ca0 2024-09-16T05:40:16.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.341+0000 7fbafaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T05:40:16.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.341+0000 7fbafaffd640 10 monclient: _finish_command 3 = system:0 device 'h6' does not appear in the crush map 2024-09-16T05:40:16.346 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h6' does not appear in the crush map 2024-09-16T05:40:16.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.341+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbae8036980 legacy=0x7fbae8038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:16.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.341+0000 7fbb0c49f640 10 monclient: shutdown 2024-09-16T05:40:16.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.341+0000 7fbb0c49f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:16.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.341+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 >> v1:172.21.15.6:6790/0 conn(0x7fbb040a3ca0 legacy=0x7fbb04136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:16.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.345+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 shutdown_connections 2024-09-16T05:40:16.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.345+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 >> 172.21.15.63:0/245419225 conn(0x7fbb040a67c0 msgr2=0x7fbb040a87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:16.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.345+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 shutdown_connections 2024-09-16T05:40:16.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.345+0000 7fbb0c49f640 1 -- 172.21.15.63:0/245419225 wait complete. 2024-09-16T05:40:16.363 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:16.363 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h7 2024-09-16T05:40:16.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient: get_monmap_and_config 2024-09-16T05:40:16.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient: build_initial_monmap 2024-09-16T05:40:16.482 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient: monmap: 2024-09-16T05:40:16.482 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:16.482 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:16.483 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:16.482518+0000 2024-09-16T05:40:16.483 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:16.482518+0000 2024-09-16T05:40:16.483 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:16.483 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:16.483 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:16.483 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:16.483 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:16.483 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:16.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 1 Processor -- start 2024-09-16T05:40:16.486 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 1 -- start start 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient: init 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient(hunting): picked mon.noname-c con 0x7f92bc0a12b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient(hunting): picked mon.noname-b con 0x7f92bc072fa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient(hunting): picked mon.noname-a con 0x7f92bc1395a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92bc0a05b0 con 0x7f92bc1395a0 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92bc09c510 con 0x7f92bc0a12b0 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92bc09ce10 con 0x7f92bc072fa0 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92c0952640 10 monclient(hunting): authenticate will time out at 1273.259388s 2024-09-16T05:40:16.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92ba59c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f92bc0a12b0 legacy=0x7f92bc09f790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92b9d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f92bc072fa0 0x7f92bc09fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48358/0 (socket says 172.21.15.63:48358) 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.477+0000 7f92b9d9b640 1 -- 172.21.15.63:0/519213802 learned_addr learned my addr 172.21.15.63:0/519213802 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3720916924 0 0) 0x7f92bc0a05b0 con 0x7f92bc1395a0 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient(hunting): my global_id is 6207 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f929c004640 con 0x7f92bc1395a0 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1875422496 0 0) 0x7f92bc09ce10 con 0x7f92bc072fa0 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:16.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient(hunting): my global_id is 6182 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92bc0a05b0 con 0x7f92bc072fa0 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 314160175 0 0) 0x7f929c004640 con 0x7f92bc1395a0 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 >> v1:172.21.15.6:6790/0 conn(0x7f92bc072fa0 legacy=0x7f92bc09fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 >> v1:172.21.15.63:6789/0 conn(0x7f92bc0a12b0 legacy=0x7f92bc09f790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 monclient: found mon.noname-a 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92bc09d0f0 con 0x7f92bc1395a0 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: _finish_auth 0 2024-09-16T05:40:16.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92b4001620 con 0x7f92bc1395a0 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: dump: 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:16.490 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 5 monclient: authenticate success, global_id 6207 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92b40018d0 con 0x7f92bc1395a0 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 1 -- 172.21.15.63:0/519213802 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92b4001ca0 con 0x7f92bc1395a0 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 4 set_mon_vals no callback set 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:16.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b959a640 10 monclient: dump: 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:16.492 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient: shutdown 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- 172.21.15.63:0/519213802 >> v1:172.21.15.6:6789/0 conn(0x7f92bc1395a0 legacy=0x7f92bc13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- 172.21.15.63:0/519213802 shutdown_connections 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- 172.21.15.63:0/519213802 >> 172.21.15.63:0/519213802 conn(0x7f92bc0a1c90 msgr2=0x7f92bc0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- 172.21.15.63:0/519213802 shutdown_connections 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- 172.21.15.63:0/519213802 wait complete. 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient: build_initial_monmap 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient: monmap: 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:16.493 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 Processor -- start 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- start start 2024-09-16T05:40:16.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient: init 2024-09-16T05:40:16.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:16.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:16.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient(hunting): picked mon.noname-c con 0x7f92bc072fa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:16.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient(hunting): picked mon.noname-b con 0x7f92bc0a12b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:16.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient(hunting): picked mon.noname-a con 0x7f92bc1395a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92bc123d50 con 0x7f92bc1395a0 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92bc126ec0 con 0x7f92bc0a12b0 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92bc0a3e90 con 0x7f92bc072fa0 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92c0952640 10 monclient(hunting): authenticate will time out at 1273.263257s 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92b9d9b640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f92bc0a12b0 legacy=0x7f92bc123640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92ba59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f92bc072fa0 0x7f92bc122f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48368/0 (socket says 172.21.15.63:48368) 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.481+0000 7f92ba59c640 1 -- 172.21.15.63:0/2846380058 learned_addr learned my addr 172.21.15.63:0/2846380058 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 672097301 0 0) 0x7f92bc123d50 con 0x7f92bc1395a0 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): my global_id is 6210 2024-09-16T05:40:16.499 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92a0003530 con 0x7f92bc1395a0 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3693347801 0 0) 0x7f92bc0a3e90 con 0x7f92bc072fa0 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): my global_id is 6185 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92bc123d50 con 0x7f92bc072fa0 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3014213273 0 0) 0x7f92a0003530 con 0x7f92bc1395a0 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92bc0a3e90 con 0x7f92bc1395a0 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92b4001a80 con 0x7f92bc1395a0 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:16.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2005951326 0 0) 0x7f92bc123d50 con 0x7f92bc072fa0 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92a0003530 con 0x7f92bc072fa0 2024-09-16T05:40:16.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92a4004160 con 0x7f92bc072fa0 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:16.502 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2904127762 0 0) 0x7f92bc0a3e90 con 0x7f92bc1395a0 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 >> v1:172.21.15.6:6790/0 conn(0x7f92bc072fa0 legacy=0x7f92bc122f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 >> v1:172.21.15.63:6789/0 conn(0x7f92bc0a12b0 legacy=0x7f92bc123640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 monclient: found mon.a 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92bc09fe70 con 0x7f92bc1395a0 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 5 monclient: authenticate success, global_id 6210 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 10 monclient: _renew_subs 2024-09-16T05:40:16.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92bc126620 con 0x7f92bc1395a0 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 10 monclient: _renew_subs 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f92bc126be0 con 0x7f92bc1395a0 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92b40018d0 con 0x7f92bc1395a0 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b8d99640 4 set_mon_vals no callback set 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92b4003b60 con 0x7f92bc1395a0 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 10 monclient: dump: 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:16.504 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9294005090 con 0x7f92bc1395a0 2024-09-16T05:40:16.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f92b400e900 con 0x7f92bc1395a0 2024-09-16T05:40:16.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(224..224 src has 20..224) ==== 12884+0+0 (unknown 3582513721 0 0) 0x7f92b4045290 con 0x7f92bc1395a0 2024-09-16T05:40:16.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.485+0000 7f92ba59c640 10 monclient: get_auth_request con 0x7f92a0036c60 auth_method 0 2024-09-16T05:40:16.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.497+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f92b4014130 con 0x7f92bc1395a0 2024-09-16T05:40:16.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.497+0000 7f92b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:16.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.497+0000 7f92b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:16.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.665+0000 7f92c0952640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T05:40:16.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.665+0000 7f92c0952640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T05:40:16.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.665+0000 7f92c0952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:16.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:16.665+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) -- 0x7f9294005380 con 0x7f92bc1395a0 2024-09-16T05:40:17.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.233+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 removed item id -37 name 'h7' from crush map v225) ==== 118+0+0 (unknown 4242561783 0 0) 0x7f92b4013a30 con 0x7f92bc1395a0 2024-09-16T05:40:17.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.233+0000 7f92b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T05:40:17.238 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.233+0000 7f92b2ffd640 10 monclient: _finish_command 2 = system:0 removed item id -37 name 'h7' from crush map 2024-09-16T05:40:17.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T05:40:17.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T05:40:17.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:17.345 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) -- 0x7f92940010a0 con 0x7f92bc1395a0 2024-09-16T05:40:17.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92b2ffd640 1 -- 172.21.15.63:0/2846380058 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 device 'h7' does not appear in the crush map v225) ==== 118+0+0 (unknown 1497426127 0 0) 0x7f92b401c5e0 con 0x7f92bc1395a0 2024-09-16T05:40:17.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92b2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T05:40:17.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92b2ffd640 10 monclient: _finish_command 3 = system:0 device 'h7' does not appear in the crush map 2024-09-16T05:40:17.346 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h7' does not appear in the crush map 2024-09-16T05:40:17.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 >> v1:172.21.15.63:6812/3643207211 conn(0x7f92a0036c60 legacy=0x7f92a0039120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:17.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 10 monclient: shutdown 2024-09-16T05:40:17.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:17.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 >> v1:172.21.15.6:6789/0 conn(0x7f92bc1395a0 legacy=0x7f92bc1d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:17.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 shutdown_connections 2024-09-16T05:40:17.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 >> 172.21.15.63:0/2846380058 conn(0x7f92bc0a1c90 msgr2=0x7f92bc072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:17.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 shutdown_connections 2024-09-16T05:40:17.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.341+0000 7f92c0952640 1 -- 172.21.15.63:0/2846380058 wait complete. 2024-09-16T05:40:17.362 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:17.363 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h8 2024-09-16T05:40:17.483 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.477+0000 7fd21cb79640 10 monclient: get_monmap_and_config 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.477+0000 7fd21cb79640 10 monclient: build_initial_monmap 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.477+0000 7fd21cb79640 10 monclient: monmap: 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:17.483736+0000 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:17.483736+0000 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:17.484 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:17.485 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:17.485 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:17.487 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.477+0000 7fd21cb79640 1 Processor -- start 2024-09-16T05:40:17.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- start start 2024-09-16T05:40:17.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient: init 2024-09-16T05:40:17.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:17.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:17.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient(hunting): picked mon.noname-b con 0x7fd2180a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:17.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient(hunting): picked mon.noname-a con 0x7fd21809b450 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:17.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient(hunting): picked mon.noname-c con 0x7fd21809b8a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:17.488 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd21809e970 con 0x7fd21809b450 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd21809c370 con 0x7fd21809b8a0 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd21809d620 con 0x7fd2180a08a0 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient(hunting): authenticate will time out at 1274.260737s 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd216d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd21809b8a0 legacy=0x7fd21813c0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd2180a08a0 0x7fd21809ad40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48378/0 (socket says 172.21.15.63:48378) 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21659c640 1 -- 172.21.15.63:0/3655160477 learned_addr learned my addr 172.21.15.63:0/3655160477 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd20dd9b640 1 --1- 172.21.15.63:0/3655160477 >> v1:172.21.15.6:6789/0 conn(0x7fd21809b450 0x7fd2181389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47666/0 (socket says 172.21.15.63:47666) 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 514268989 0 0) 0x7fd21809d620 con 0x7fd2180a08a0 2024-09-16T05:40:17.489 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient(hunting): my global_id is 6188 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd204003530 con 0x7fd2180a08a0 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573800026 0 0) 0x7fd21809e970 con 0x7fd21809b450 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient(hunting): my global_id is 6213 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd21809d620 con 0x7fd21809b450 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3349779230 0 0) 0x7fd21809d620 con 0x7fd21809b450 2024-09-16T05:40:17.490 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 >> v1:172.21.15.6:6790/0 conn(0x7fd2180a08a0 legacy=0x7fd21809ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 >> v1:172.21.15.63:6789/0 conn(0x7fd21809b8a0 legacy=0x7fd21813c0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 monclient: found mon.noname-a 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21809d900 con 0x7fd21809b450 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: _finish_auth 0 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd200001600 con 0x7fd21809b450 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:17.491 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: dump: 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 5 monclient: authenticate success, global_id 6213 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:17.492 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd200001860 con 0x7fd21809b450 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 1 -- 172.21.15.63:0/3655160477 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd200001c30 con 0x7fd21809b450 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 4 set_mon_vals no callback set 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd215d9b640 10 monclient: dump: 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.493 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient: shutdown 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- 172.21.15.63:0/3655160477 >> v1:172.21.15.6:6789/0 conn(0x7fd21809b450 legacy=0x7fd2181389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- 172.21.15.63:0/3655160477 shutdown_connections 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- 172.21.15.63:0/3655160477 >> 172.21.15.63:0/3655160477 conn(0x7fd2180a1c90 msgr2=0x7fd2180a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- 172.21.15.63:0/3655160477 shutdown_connections 2024-09-16T05:40:17.494 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 1 -- 172.21.15.63:0/3655160477 wait complete. 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient: build_initial_monmap 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.481+0000 7fd21cb79640 10 monclient: monmap: 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:17.495 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:17.496 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:17.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 1 Processor -- start 2024-09-16T05:40:17.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 1 -- start start 2024-09-16T05:40:17.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: init 2024-09-16T05:40:17.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:17.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:17.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient(hunting): picked mon.noname-c con 0x7fd21809b450 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:17.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient(hunting): picked mon.noname-a con 0x7fd21809b8a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:17.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient(hunting): picked mon.noname-b con 0x7fd2180a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd21811d230 con 0x7fd21809b8a0 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd21811c200 con 0x7fd2180a08a0 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd21811e960 con 0x7fd21809b450 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd216d9d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fd2180a08a0 legacy=0x7fd2181d3890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient(hunting): authenticate will time out at 1274.264817s 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd21809b8a0 0x7fd21811baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47670/0 (socket says 172.21.15.63:47670) 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20dd9b640 1 -- 172.21.15.63:0/1307600829 learned_addr learned my addr 172.21.15.63:0/1307600829 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452972428 0 0) 0x7fd21811d230 con 0x7fd21809b8a0 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:17.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): my global_id is 6216 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1f8003530 con 0x7fd21809b8a0 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3233417345 0 0) 0x7fd21811e960 con 0x7fd21809b450 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): my global_id is 6191 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd21811d230 con 0x7fd21809b450 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3665626505 0 0) 0x7fd1f8003530 con 0x7fd21809b8a0 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd21811e960 con 0x7fd21809b8a0 2024-09-16T05:40:17.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd200001580 con 0x7fd21809b8a0 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:17.503 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2013383967 0 0) 0x7fd21811d230 con 0x7fd21809b450 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1f8003530 con 0x7fd21809b450 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd208001a40 con 0x7fd21809b450 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:17.504 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 997880917 0 0) 0x7fd21811e960 con 0x7fd21809b8a0 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 >> v1:172.21.15.6:6790/0 conn(0x7fd21809b450 legacy=0x7fd21811b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 >> v1:172.21.15.63:6789/0 conn(0x7fd2180a08a0 legacy=0x7fd2181d3890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:17.505 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 monclient: found mon.a 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21813cbd0 con 0x7fd21809b8a0 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 5 monclient: authenticate success, global_id 6216 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: _renew_subs 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd218134570 con 0x7fd21809b8a0 2024-09-16T05:40:17.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: _renew_subs 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd21811e3a0 con 0x7fd21809b8a0 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd200001720 con 0x7fd21809b8a0 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21559a640 4 set_mon_vals no callback set 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd200004aa0 con 0x7fd21809b8a0 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 10 monclient: dump: 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.507 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1fc005090 con 0x7fd21809b8a0 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.485+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd200004d00 con 0x7fd21809b8a0 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.489+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(225..225 src has 20..225) ==== 12674+0+0 (unknown 3877273820 0 0) 0x7fd200046470 con 0x7fd21809b8a0 2024-09-16T05:40:17.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.489+0000 7fd21659c640 10 monclient: get_auth_request con 0x7fd1f80369b0 auth_method 0 2024-09-16T05:40:17.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.501+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd2000153c0 con 0x7fd21809b8a0 2024-09-16T05:40:17.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.501+0000 7fd20f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:17.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.501+0000 7fd20f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:17.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.661+0000 7fd21cb79640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T05:40:17.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.661+0000 7fd21cb79640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T05:40:17.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.661+0000 7fd21cb79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:17.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:17.661+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) -- 0x7fd1fc005380 con 0x7fd21809b8a0 2024-09-16T05:40:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.253+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 removed item id -40 name 'h8' from crush map v226) ==== 118+0+0 (unknown 2085914414 0 0) 0x7fd200014cc0 con 0x7fd21809b8a0 2024-09-16T05:40:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.253+0000 7fd20f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T05:40:18.260 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.253+0000 7fd20f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -40 name 'h8' from crush map 2024-09-16T05:40:18.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.361+0000 7fd21cb79640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T05:40:18.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.361+0000 7fd21cb79640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T05:40:18.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.361+0000 7fd21cb79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:18.365 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.361+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) -- 0x7fd1fc0010a0 con 0x7fd21809b8a0 2024-09-16T05:40:18.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.361+0000 7fd20f7fe640 1 -- 172.21.15.63:0/1307600829 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 device 'h8' does not appear in the crush map v226) ==== 118+0+0 (unknown 255433356 0 0) 0x7fd20001d870 con 0x7fd21809b8a0 2024-09-16T05:40:18.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.361+0000 7fd20f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T05:40:18.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.361+0000 7fd20f7fe640 10 monclient: _finish_command 3 = system:0 device 'h8' does not appear in the crush map 2024-09-16T05:40:18.366 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h8' does not appear in the crush map 2024-09-16T05:40:18.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.365+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd1f80369b0 legacy=0x7fd1f8038e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:18.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.365+0000 7fd21cb79640 10 monclient: shutdown 2024-09-16T05:40:18.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.365+0000 7fd21cb79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:18.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.365+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 >> v1:172.21.15.6:6789/0 conn(0x7fd21809b8a0 legacy=0x7fd21811baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:18.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.365+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 shutdown_connections 2024-09-16T05:40:18.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.365+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 >> 172.21.15.63:0/1307600829 conn(0x7fd2180a1c90 msgr2=0x7fd21813b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:18.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.365+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 shutdown_connections 2024-09-16T05:40:18.370 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.365+0000 7fd21cb79640 1 -- 172.21.15.63:0/1307600829 wait complete. 2024-09-16T05:40:18.383 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:18.383 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h9 2024-09-16T05:40:18.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient: get_monmap_and_config 2024-09-16T05:40:18.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient: build_initial_monmap 2024-09-16T05:40:18.506 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient: monmap: 2024-09-16T05:40:18.506 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:18.506 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:18.506 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:18.506050+0000 2024-09-16T05:40:18.506 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:18.506050+0000 2024-09-16T05:40:18.506 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:18.507 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:18.507 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:18.507 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:18.507 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:18.507 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 1 Processor -- start 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 1 -- start start 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient: init 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient(hunting): picked mon.noname-c con 0x7f34a80a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient(hunting): picked mon.noname-a con 0x7f34a80a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient(hunting): picked mon.noname-b con 0x7f34a80a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a80a4fd0 con 0x7f34a80a08a0 2024-09-16T05:40:18.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a80a5b40 con 0x7f34a80a3ca0 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a80a1a70 con 0x7f34a80a4790 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34af621640 10 monclient(hunting): authenticate will time out at 1275.283092s 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34ad3bd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f34a80a3ca0 legacy=0x7f34a80a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34acbbc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f34a80a08a0 0x7f34a813beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47674/0 (socket says 172.21.15.63:47674) 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.501+0000 7f34acbbc640 1 -- 172.21.15.63:0/1350216260 learned_addr learned my addr 172.21.15.63:0/1350216260 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039700315 0 0) 0x7f34a80a1a70 con 0x7f34a80a4790 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:18.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient(hunting): my global_id is 6194 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3494003530 con 0x7f34a80a4790 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 627543498 0 0) 0x7f34a80a4fd0 con 0x7f34a80a08a0 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient(hunting): my global_id is 6219 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34a80a1a70 con 0x7f34a80a08a0 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2259455546 0 0) 0x7f3494003530 con 0x7f34a80a4790 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 >> v1:172.21.15.63:6789/0 conn(0x7f34a80a3ca0 legacy=0x7f34a80a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:18.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 >> v1:172.21.15.6:6789/0 conn(0x7f34a80a08a0 legacy=0x7f34a813beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 monclient: found mon.noname-b 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a80a53e0 con 0x7f34a80a4790 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: _finish_auth 0 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f34a00018d0 con 0x7f34a80a4790 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: dump: 2024-09-16T05:40:18.518 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 5 monclient: authenticate success, global_id 6194 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f34a0001b30 con 0x7f34a80a4790 2024-09-16T05:40:18.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 1 -- 172.21.15.63:0/1350216260 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f34a0001e80 con 0x7f34a80a4790 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f349bfff640 10 monclient: dump: 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:18.520 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 4 set_mon_vals no callback set 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient: shutdown 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- 172.21.15.63:0/1350216260 >> v1:172.21.15.6:6790/0 conn(0x7f34a80a4790 legacy=0x7f34a813f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- 172.21.15.63:0/1350216260 shutdown_connections 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- 172.21.15.63:0/1350216260 >> 172.21.15.63:0/1350216260 conn(0x7f34a80a67c0 msgr2=0x7f34a8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- 172.21.15.63:0/1350216260 shutdown_connections 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- 172.21.15.63:0/1350216260 wait complete. 2024-09-16T05:40:18.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient: build_initial_monmap 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient: monmap: 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 Processor -- start 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- start start 2024-09-16T05:40:18.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient: init 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient(hunting): picked mon.noname-a con 0x7f34a80a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient(hunting): picked mon.noname-c con 0x7f34a80a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient(hunting): picked mon.noname-b con 0x7f34a80a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a8125420 con 0x7f34a80a08a0 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a8126f70 con 0x7f34a80a4790 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34a8124130 con 0x7f34a80a3ca0 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34af621640 10 monclient(hunting): authenticate will time out at 1275.286972s 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34adbbe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f34a80a4790 legacy=0x7f34a81d3ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:18.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34acbbc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f34a80a3ca0 0x7f34a8123a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48408/0 (socket says 172.21.15.63:48408) 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34ad3bd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f34a80a08a0 0x7f34a8123310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47688/0 (socket says 172.21.15.63:47688) 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f34acbbc640 1 -- 172.21.15.63:0/1270877053 learned_addr learned my addr 172.21.15.63:0/1270877053 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251484421 0 0) 0x7f34a8124130 con 0x7f34a80a3ca0 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 10 monclient(hunting): my global_id is 6197 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f348c003530 con 0x7f34a80a3ca0 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2627506109 0 0) 0x7f34a4003a50 con 0x7f34a80a08a0 2024-09-16T05:40:18.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 10 monclient(hunting): my global_id is 6222 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.505+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34a8124130 con 0x7f34a80a08a0 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2161142805 0 0) 0x7f348c003530 con 0x7f34a80a3ca0 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34a4003a50 con 0x7f34a80a3ca0 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f349c0018a0 con 0x7f34a80a3ca0 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient(hunting): dump: 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:18.525 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2115814632 0 0) 0x7f34a8124130 con 0x7f34a80a08a0 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f348c003530 con 0x7f34a80a08a0 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f34a4004260 con 0x7f34a80a08a0 2024-09-16T05:40:18.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient(hunting): dump: 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:18.527 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 984914118 0 0) 0x7f34a4003a50 con 0x7f34a80a3ca0 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 >> v1:172.21.15.63:6789/0 conn(0x7f34a80a4790 legacy=0x7f34a81d3ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 >> v1:172.21.15.6:6789/0 conn(0x7f34a80a08a0 legacy=0x7f34a8123310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 monclient: found mon.c 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a8125ac0 con 0x7f34a80a3ca0 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 5 monclient: authenticate success, global_id 6197 2024-09-16T05:40:18.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 10 monclient: _renew_subs 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34a8127250 con 0x7f34a80a3ca0 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 10 monclient: _renew_subs 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f34a80a8370 con 0x7f34a80a3ca0 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f349c0047f0 con 0x7f34a80a3ca0 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f349c004b40 con 0x7f34a80a3ca0 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 10 monclient: dump: 2024-09-16T05:40:18.529 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f349affd640 4 set_mon_vals no callback set 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:18.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:18.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:18.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3488005090 con 0x7f34a80a3ca0 2024-09-16T05:40:18.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f349c00f8e0 con 0x7f34a80a3ca0 2024-09-16T05:40:18.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(226..226 src has 20..226) ==== 12464+0+0 (unknown 3222886750 0 0) 0x7f349c046360 con 0x7f34a80a3ca0 2024-09-16T05:40:18.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.509+0000 7f34ad3bd640 10 monclient: get_auth_request con 0x7f348c0369c0 auth_method 0 2024-09-16T05:40:18.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.521+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f349c015320 con 0x7f34a80a3ca0 2024-09-16T05:40:18.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.521+0000 7f3499ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:18.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.521+0000 7f3499ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.693+0000 7f34af621640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T05:40:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.693+0000 7f34af621640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T05:40:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.693+0000 7f34af621640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:18.698 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:18.693+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) -- 0x7f3488005380 con 0x7f34a80a3ca0 2024-09-16T05:40:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.269+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 removed item id -43 name 'h9' from crush map v227) ==== 118+0+0 (unknown 2088645750 0 0) 0x7f349c014c20 con 0x7f34a80a3ca0 2024-09-16T05:40:19.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.269+0000 7f3499ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T05:40:19.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.269+0000 7f3499ffb640 10 monclient: _finish_command 2 = system:0 removed item id -43 name 'h9' from crush map 2024-09-16T05:40:19.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.377+0000 7f34af621640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T05:40:19.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.377+0000 7f34af621640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T05:40:19.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.377+0000 7f34af621640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:19.381 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.377+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) -- 0x7f34880010a0 con 0x7f34a80a3ca0 2024-09-16T05:40:19.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.377+0000 7f3499ffb640 1 -- 172.21.15.63:0/1270877053 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 device 'h9' does not appear in the crush map v227) ==== 118+0+0 (unknown 3428238216 0 0) 0x7f349c018f70 con 0x7f34a80a3ca0 2024-09-16T05:40:19.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.377+0000 7f3499ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T05:40:19.383 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.377+0000 7f3499ffb640 10 monclient: _finish_command 3 = system:0 device 'h9' does not appear in the crush map 2024-09-16T05:40:19.384 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h9' does not appear in the crush map 2024-09-16T05:40:19.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.381+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 >> v1:172.21.15.63:6812/3643207211 conn(0x7f348c0369c0 legacy=0x7f348c038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:19.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.381+0000 7f34af621640 10 monclient: shutdown 2024-09-16T05:40:19.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.381+0000 7f34af621640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:19.386 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.381+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 >> v1:172.21.15.6:6790/0 conn(0x7f34a80a3ca0 legacy=0x7f34a8123a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:19.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.381+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 shutdown_connections 2024-09-16T05:40:19.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.381+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 >> 172.21.15.63:0/1270877053 conn(0x7f34a80a67c0 msgr2=0x7f34a8137320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:19.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.381+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 shutdown_connections 2024-09-16T05:40:19.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.381+0000 7f34af621640 1 -- 172.21.15.63:0/1270877053 wait complete. 2024-09-16T05:40:19.401 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:19.401 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h10 2024-09-16T05:40:19.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient: get_monmap_and_config 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient: build_initial_monmap 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient: monmap: 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:19.522673+0000 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:19.522673+0000 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:19.523 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:19.524 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 1 Processor -- start 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 1 -- start start 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient: init 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient(hunting): picked mon.noname-b con 0x7f6520136ac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient(hunting): picked mon.noname-a con 0x7f652009fd80 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient(hunting): picked mon.noname-c con 0x7f652013d0b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:19.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6520072cb0 con 0x7f652009fd80 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6520136670 con 0x7f652013d0b0 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65200a14f0 con 0x7f6520136ac0 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f6525eb2640 10 monclient(hunting): authenticate will time out at 1276.299687s 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.517+0000 7f651ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f652013d0b0 legacy=0x7f652013f4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f652009fd80 0x7f652013b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47700/0 (socket says 172.21.15.63:47700) 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651effd640 1 -- 172.21.15.63:0/3261905647 learned_addr learned my addr 172.21.15.63:0/3261905647 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633309137 0 0) 0x7f65200a14f0 con 0x7f6520136ac0 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:19.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient(hunting): my global_id is 6200 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f650c003530 con 0x7f6520136ac0 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016634511 0 0) 0x7f6508001120 con 0x7f652009fd80 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient(hunting): my global_id is 6225 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65200a14f0 con 0x7f652009fd80 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3842833529 0 0) 0x7f650c003530 con 0x7f6520136ac0 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 >> v1:172.21.15.63:6789/0 conn(0x7f652013d0b0 legacy=0x7f652013f4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 >> v1:172.21.15.6:6789/0 conn(0x7f652009fd80 legacy=0x7f652013b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:19.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 monclient: found mon.noname-b 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65200a0630 con 0x7f6520136ac0 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6510001620 con 0x7f6520136ac0 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: dump: 2024-09-16T05:40:19.530 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:19.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 5 monclient: authenticate success, global_id 6200 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6510001880 con 0x7f6520136ac0 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 1 -- 172.21.15.63:0/3261905647 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6510001c90 con 0x7f6520136ac0 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 4 set_mon_vals no callback set 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f651e7fc640 10 monclient: dump: 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:19.532 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.533 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.533 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:19.533 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:19.533 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:19.533 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:19.533 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:19.533 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 10 monclient: shutdown 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 1 -- 172.21.15.63:0/3261905647 >> v1:172.21.15.6:6790/0 conn(0x7f6520136ac0 legacy=0x7f6520136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 1 -- 172.21.15.63:0/3261905647 shutdown_connections 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 1 -- 172.21.15.63:0/3261905647 >> 172.21.15.63:0/3261905647 conn(0x7f65200a1c90 msgr2=0x7f65200a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 1 -- 172.21.15.63:0/3261905647 shutdown_connections 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 1 -- 172.21.15.63:0/3261905647 wait complete. 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 10 monclient: build_initial_monmap 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 10 monclient: monmap: 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:19.534 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:19.535 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:19.535 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:19.535 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:19.535 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:19.535 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:19.535 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:19.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 1 Processor -- start 2024-09-16T05:40:19.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 1 -- start start 2024-09-16T05:40:19.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 10 monclient: init 2024-09-16T05:40:19.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:19.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:19.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.521+0000 7f6525eb2640 10 monclient(hunting): picked mon.noname-b con 0x7f652009fd80 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:19.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient(hunting): picked mon.noname-a con 0x7f6520136ac0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:19.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient(hunting): picked mon.noname-c con 0x7f652013d0b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:19.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6520120bd0 con 0x7f6520136ac0 2024-09-16T05:40:19.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f652011f4a0 con 0x7f652009fd80 2024-09-16T05:40:19.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f651f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f652009fd80 legacy=0x7f65201cb5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:19.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6520120eb0 con 0x7f652013d0b0 2024-09-16T05:40:19.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:19.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient(hunting): authenticate will time out at 1276.304097s 2024-09-16T05:40:19.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f651ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f652013d0b0 0x7f65201d3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48432/0 (socket says 172.21.15.63:48432) 2024-09-16T05:40:19.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f651ffff640 1 -- 172.21.15.63:0/4271788548 learned_addr learned my addr 172.21.15.63:0/4271788548 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:19.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403334518 0 0) 0x7f6520120bd0 con 0x7f6520136ac0 2024-09-16T05:40:19.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:19.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:19.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:19.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): my global_id is 6228 2024-09-16T05:40:19.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6500003530 con 0x7f6520136ac0 2024-09-16T05:40:19.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1433492435 0 0) 0x7f6520120eb0 con 0x7f652013d0b0 2024-09-16T05:40:19.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:19.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): my global_id is 6203 2024-09-16T05:40:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6520120bd0 con 0x7f652013d0b0 2024-09-16T05:40:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 896477994 0 0) 0x7f6500003530 con 0x7f6520136ac0 2024-09-16T05:40:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6520120eb0 con 0x7f6520136ac0 2024-09-16T05:40:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f65080013c0 con 0x7f6520136ac0 2024-09-16T05:40:19.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): dump: 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:19.545 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4150448394 0 0) 0x7f6520120bd0 con 0x7f652013d0b0 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6500003530 con 0x7f652013d0b0 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6518004340 con 0x7f652013d0b0 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient(hunting): dump: 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.546 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 518402309 0 0) 0x7f6520120eb0 con 0x7f6520136ac0 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 >> v1:172.21.15.6:6790/0 conn(0x7f652013d0b0 legacy=0x7f65201d3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 >> v1:172.21.15.63:6789/0 conn(0x7f652009fd80 legacy=0x7f65201cb5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 monclient: found mon.a 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:19.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652013c780 con 0x7f6520136ac0 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient: _finish_auth 0 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 5 monclient: authenticate success, global_id 6228 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient: _renew_subs 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f652009d430 con 0x7f6520136ac0 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient: _renew_subs 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f652011df30 con 0x7f6520136ac0 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65080047f0 con 0x7f6520136ac0 2024-09-16T05:40:19.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6508004b40 con 0x7f6520136ac0 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f651dffb640 4 set_mon_vals no callback set 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 10 monclient: dump: 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:19.549 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6504005090 con 0x7f6520136ac0 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.525+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f650800f960 con 0x7f6520136ac0 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.529+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(227..227 src has 20..227) ==== 12254+0+0 (unknown 2436707240 0 0) 0x7f65080462d0 con 0x7f6520136ac0 2024-09-16T05:40:19.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.529+0000 7f651f7fe640 10 monclient: get_auth_request con 0x7f65000369c0 auth_method 0 2024-09-16T05:40:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.541+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6520072cb0 con 0x7f6520136ac0 2024-09-16T05:40:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.541+0000 7f6517fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:19.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.541+0000 7f6517fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:19.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.701+0000 7f6525eb2640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T05:40:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.701+0000 7f6525eb2640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T05:40:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.701+0000 7f6525eb2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:19.706 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:19.701+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) -- 0x7f6504005380 con 0x7f6520136ac0 2024-09-16T05:40:20.281 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.277+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 removed item id -46 name 'h10' from crush map v228) ==== 120+0+0 (unknown 519503782 0 0) 0x7f65080153e0 con 0x7f6520136ac0 2024-09-16T05:40:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.277+0000 7f6517fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T05:40:20.282 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.277+0000 7f6517fff640 10 monclient: _finish_command 2 = system:0 removed item id -46 name 'h10' from crush map 2024-09-16T05:40:20.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.381+0000 7f6525eb2640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T05:40:20.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.381+0000 7f6525eb2640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T05:40:20.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.381+0000 7f6525eb2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:20.387 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.381+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) -- 0x7f65040010a0 con 0x7f6520136ac0 2024-09-16T05:40:20.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.381+0000 7f6517fff640 1 -- 172.21.15.63:0/4271788548 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 device 'h10' does not appear in the crush map v228) ==== 120+0+0 (unknown 3837715775 0 0) 0x7f6508014ce0 con 0x7f6520136ac0 2024-09-16T05:40:20.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.381+0000 7f6517fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T05:40:20.388 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.381+0000 7f6517fff640 10 monclient: _finish_command 3 = system:0 device 'h10' does not appear in the crush map 2024-09-16T05:40:20.388 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h10' does not appear in the crush map 2024-09-16T05:40:20.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.385+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 >> v1:172.21.15.63:6812/3643207211 conn(0x7f65000369c0 legacy=0x7f6500038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:20.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.385+0000 7f6525eb2640 10 monclient: shutdown 2024-09-16T05:40:20.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.385+0000 7f6525eb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:20.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.385+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 >> v1:172.21.15.6:6789/0 conn(0x7f6520136ac0 legacy=0x7f65201d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:20.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.385+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 shutdown_connections 2024-09-16T05:40:20.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.385+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 >> 172.21.15.63:0/4271788548 conn(0x7f65200a1c90 msgr2=0x7f652009cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:20.390 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.385+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 shutdown_connections 2024-09-16T05:40:20.391 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.385+0000 7f6525eb2640 1 -- 172.21.15.63:0/4271788548 wait complete. 2024-09-16T05:40:20.405 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:20.405 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h11 2024-09-16T05:40:20.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.521+0000 7f0c06d40640 10 monclient: get_monmap_and_config 2024-09-16T05:40:20.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.521+0000 7f0c06d40640 10 monclient: build_initial_monmap 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.521+0000 7f0c06d40640 10 monclient: monmap: 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:20.527751+0000 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:20.527751+0000 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:20.528 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 Processor -- start 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- start start 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient: init 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient(hunting): picked mon.noname-a con 0x7f0c0009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient(hunting): picked mon.noname-c con 0x7f0c0006dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient(hunting): picked mon.noname-b con 0x7f0c000a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0009f800 con 0x7f0c0009ac90 2024-09-16T05:40:20.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000a0ab0 con 0x7f0c0006dd30 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000a0d90 con 0x7f0c000a0210 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient(hunting): authenticate will time out at 1277.304962s 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bfffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0c0006dd30 legacy=0x7f0c001399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c04adc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0c0009ac90 0x7f0c0009b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47724/0 (socket says 172.21.15.63:47724) 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c04adc640 1 -- 172.21.15.63:0/374269922 learned_addr learned my addr 172.21.15.63:0/374269922 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1227643880 0 0) 0x7f0c0009f800 con 0x7f0c0009ac90 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient(hunting): my global_id is 6231 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bec003530 con 0x7f0c0009ac90 2024-09-16T05:40:20.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2758484801 0 0) 0x7f0c000a0d90 con 0x7f0c000a0210 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient(hunting): my global_id is 6206 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c0009f800 con 0x7f0c000a0210 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3426296010 0 0) 0x7f0bec003530 con 0x7f0c0009ac90 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 >> v1:172.21.15.6:6790/0 conn(0x7f0c000a0210 legacy=0x7f0c0013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 >> v1:172.21.15.63:6789/0 conn(0x7f0c0006dd30 legacy=0x7f0c001399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 monclient: found mon.noname-a 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:20.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:20.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0009e770 con 0x7f0c0009ac90 2024-09-16T05:40:20.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:20.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:20.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:20.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 5 monclient: authenticate success, global_id 6231 2024-09-16T05:40:20.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:20.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0be8001550 con 0x7f0c0009ac90 2024-09-16T05:40:20.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: dump: 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:20.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0be80017b0 con 0x7f0c0009ac90 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 1 -- 172.21.15.63:0/374269922 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0be8001b80 con 0x7f0c0009ac90 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 4 set_mon_vals no callback set 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0bff7fe640 10 monclient: dump: 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.542 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:20.543 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:20.543 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:20.543 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:20.543 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:20.543 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:20.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 10 monclient: shutdown 2024-09-16T05:40:20.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:20.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- 172.21.15.63:0/374269922 >> v1:172.21.15.6:6789/0 conn(0x7f0c0009ac90 legacy=0x7f0c0009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- 172.21.15.63:0/374269922 shutdown_connections 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- 172.21.15.63:0/374269922 >> 172.21.15.63:0/374269922 conn(0x7f0c000a1c90 msgr2=0x7f0c000a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- 172.21.15.63:0/374269922 shutdown_connections 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.525+0000 7f0c06d40640 1 -- 172.21.15.63:0/374269922 wait complete. 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: build_initial_monmap 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: monmap: 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:20.544 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 1 Processor -- start 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 1 -- start start 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: init 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient(hunting): picked mon.noname-a con 0x7f0c0006dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient(hunting): picked mon.noname-c con 0x7f0c0009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient(hunting): picked mon.noname-b con 0x7f0c000a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:20.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c001243c0 con 0x7f0c0006dd30 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0009c0f0 con 0x7f0c000a0210 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000760a0 con 0x7f0c0009ac90 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient(hunting): authenticate will time out at 1277.308985s 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c052dd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0c000a0210 legacy=0x7f0c001d1850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c04adc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0c0006dd30 0x7f0c001349e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47728/0 (socket says 172.21.15.63:47728) 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c04adc640 1 -- 172.21.15.63:0/430080977 learned_addr learned my addr 172.21.15.63:0/430080977 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426550218 0 0) 0x7f0c001243c0 con 0x7f0c0006dd30 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): my global_id is 6234 2024-09-16T05:40:20.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0be0003530 con 0x7f0c0006dd30 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92150041 0 0) 0x7f0c000760a0 con 0x7f0c0009ac90 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): my global_id is 6209 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c001243c0 con 0x7f0c0009ac90 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2761171563 0 0) 0x7f0be0003530 con 0x7f0c0006dd30 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c000760a0 con 0x7f0c0006dd30 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0be8001510 con 0x7f0c0006dd30 2024-09-16T05:40:20.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): dump: 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:20.548 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2623405113 0 0) 0x7f0c001243c0 con 0x7f0c0009ac90 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0be0003530 con 0x7f0c0009ac90 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0bf0004450 con 0x7f0c0009ac90 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient(hunting): dump: 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:20.549 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:20.550 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2009217571 0 0) 0x7f0c000760a0 con 0x7f0c0006dd30 2024-09-16T05:40:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 >> v1:172.21.15.6:6790/0 conn(0x7f0c0009ac90 legacy=0x7f0c001ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 >> v1:172.21.15.63:6789/0 conn(0x7f0c000a0210 legacy=0x7f0c001d1850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:40:20.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 monclient: found mon.a 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0006e2f0 con 0x7f0c0006dd30 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient: _finish_auth 0 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 5 monclient: authenticate success, global_id 6234 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: _renew_subs 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:20.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c00123e00 con 0x7f0c0006dd30 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: _renew_subs 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c001350f0 con 0x7f0c0006dd30 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0be80017b0 con 0x7f0c0006dd30 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfeffd640 4 set_mon_vals no callback set 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0be8004cc0 con 0x7f0c0006dd30 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0bfd7fa640 10 monclient: dump: 2024-09-16T05:40:20.552 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:20.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.529+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be4005090 con 0x7f0c0006dd30 2024-09-16T05:40:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.533+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0be8004ee0 con 0x7f0c0006dd30 2024-09-16T05:40:20.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.533+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(228..228 src has 20..228) ==== 12041+0+0 (unknown 2131293097 0 0) 0x7f0be8046310 con 0x7f0c0006dd30 2024-09-16T05:40:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.533+0000 7f0bfffff640 10 monclient: get_auth_request con 0x7f0be00369a0 auth_method 0 2024-09-16T05:40:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.545+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0be8015530 con 0x7f0c0006dd30 2024-09-16T05:40:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.545+0000 7f0bfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:20.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.545+0000 7f0bfd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:20.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.713+0000 7f0c06d40640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T05:40:20.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.713+0000 7f0c06d40640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T05:40:20.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.713+0000 7f0c06d40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:20.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:20.713+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) -- 0x7f0be4005380 con 0x7f0c0006dd30 2024-09-16T05:40:21.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.053+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 removed item id -49 name 'h11' from crush map v229) ==== 120+0+0 (unknown 1271626526 0 0) 0x7f0be8014e30 con 0x7f0c0006dd30 2024-09-16T05:40:21.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.053+0000 7f0bfd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T05:40:21.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.053+0000 7f0bfd7fa640 10 monclient: _finish_command 2 = system:0 removed item id -49 name 'h11' from crush map 2024-09-16T05:40:21.163 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.157+0000 7f0c06d40640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T05:40:21.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.157+0000 7f0c06d40640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T05:40:21.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.157+0000 7f0c06d40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:21.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.157+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) -- 0x7f0be40010a0 con 0x7f0c0006dd30 2024-09-16T05:40:21.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0bfd7fa640 1 -- 172.21.15.63:0/430080977 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 device 'h11' does not appear in the crush map v229) ==== 120+0+0 (unknown 1351011728 0 0) 0x7f0be801d980 con 0x7f0c0006dd30 2024-09-16T05:40:21.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0bfd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T05:40:21.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0bfd7fa640 10 monclient: _finish_command 3 = system:0 device 'h11' does not appear in the crush map 2024-09-16T05:40:21.165 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h11' does not appear in the crush map 2024-09-16T05:40:21.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0be00369a0 legacy=0x7f0be0038e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:21.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0c06d40640 10 monclient: shutdown 2024-09-16T05:40:21.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0c06d40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:21.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 >> v1:172.21.15.6:6789/0 conn(0x7f0c0006dd30 legacy=0x7f0c001349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:21.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 shutdown_connections 2024-09-16T05:40:21.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 >> 172.21.15.63:0/430080977 conn(0x7f0c000a1c90 msgr2=0x7f0c000a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:21.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 shutdown_connections 2024-09-16T05:40:21.167 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.161+0000 7f0c06d40640 1 -- 172.21.15.63:0/430080977 wait complete. 2024-09-16T05:40:21.181 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:21.182 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h12 2024-09-16T05:40:21.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient: get_monmap_and_config 2024-09-16T05:40:21.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient: build_initial_monmap 2024-09-16T05:40:21.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient: monmap: 2024-09-16T05:40:21.305 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:21.305558+0000 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:21.305558+0000 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:21.306 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 1 Processor -- start 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 1 -- start start 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient: init 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient(hunting): picked mon.noname-a con 0x7fc39c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient(hunting): picked mon.noname-c con 0x7fc39c06dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient(hunting): picked mon.noname-b con 0x7fc39c0a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c09f800 con 0x7fc39c09ac90 2024-09-16T05:40:21.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c0a0ab0 con 0x7fc39c06dd30 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c0a0d90 con 0x7fc39c0a0210 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a47a5640 10 monclient(hunting): authenticate will time out at 1278.082617s 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a1d40640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc39c06dd30 legacy=0x7fc39c1399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a2541640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc39c09ac90 0x7fc39c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47734/0 (socket says 172.21.15.63:47734) 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a2541640 1 -- 172.21.15.63:0/1615336766 learned_addr learned my addr 172.21.15.63:0/1615336766 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1861452135 0 0) 0x7fc39c09f800 con 0x7fc39c09ac90 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient(hunting): _negotiate 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient(hunting): my global_id is 6237 2024-09-16T05:40:21.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc388003530 con 0x7fc39c09ac90 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382285534 0 0) 0x7fc39c0a0d90 con 0x7fc39c0a0210 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient(hunting): _negotiate 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient(hunting): my global_id is 6212 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc39c09f800 con 0x7fc39c0a0210 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1339208384 0 0) 0x7fc39c09f800 con 0x7fc39c0a0210 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 >> v1:172.21.15.63:6789/0 conn(0x7fc39c06dd30 legacy=0x7fc39c1399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 >> v1:172.21.15.6:6789/0 conn(0x7fc39c09ac90 legacy=0x7fc39c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 10 monclient: _finish_hunting 0 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.301+0000 7fc3a153f640 1 monclient: found mon.noname-b 2024-09-16T05:40:21.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c09e770 con 0x7fc39c0a0210 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: _finish_auth 0 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: _check_auth_tickets 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 5 monclient: authenticate success, global_id 6212 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc394001580 con 0x7fc39c0a0210 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:21.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: dump: 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3940017e0 con 0x7fc39c0a0210 2024-09-16T05:40:21.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 1 -- 172.21.15.63:0/1615336766 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc394001c30 con 0x7fc39c0a0210 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a153f640 10 monclient: dump: 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:21.315 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 4 set_mon_vals no callback set 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient: shutdown 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1615336766 >> v1:172.21.15.6:6790/0 conn(0x7fc39c0a0210 legacy=0x7fc39c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1615336766 shutdown_connections 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1615336766 >> 172.21.15.63:0/1615336766 conn(0x7fc39c0a1c90 msgr2=0x7fc39c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1615336766 shutdown_connections 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1615336766 wait complete. 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient: build_initial_monmap 2024-09-16T05:40:21.316 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient: monmap: 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 Processor -- start 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- start start 2024-09-16T05:40:21.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient: init 2024-09-16T05:40:21.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:21.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:21.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient(hunting): picked mon.noname-b con 0x7fc39c06dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:21.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient(hunting): picked mon.noname-a con 0x7fc39c09ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient(hunting): picked mon.noname-c con 0x7fc39c0a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c1275d0 con 0x7fc39c09ac90 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c126080 con 0x7fc39c06dd30 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a2541640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fc39c06dd30 legacy=0x7fc39c134bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c09c0f0 con 0x7fc39c0a0210 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a47a5640 10 monclient(hunting): authenticate will time out at 1278.086553s 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a1d40640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc39c09ac90 0x7fc39c1d00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47746/0 (socket says 172.21.15.63:47746) 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc3a1d40640 1 -- 172.21.15.63:0/1656370619 learned_addr learned my addr 172.21.15.63:0/1656370619 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:21.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750161789 0 0) 0x7fc39c09c0f0 con 0x7fc39c0a0210 2024-09-16T05:40:21.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:21.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:21.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:21.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): my global_id is 6215 2024-09-16T05:40:21.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc380003530 con 0x7fc39c0a0210 2024-09-16T05:40:21.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018412562 0 0) 0x7fc39c1275d0 con 0x7fc39c09ac90 2024-09-16T05:40:21.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): my global_id is 6240 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc39c09c0f0 con 0x7fc39c09ac90 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2201597109 0 0) 0x7fc380003530 con 0x7fc39c0a0210 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc39c1275d0 con 0x7fc39c0a0210 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2990285271 0 0) 0x7fc39c09c0f0 con 0x7fc39c09ac90 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc380003530 con 0x7fc39c09ac90 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc394001b30 con 0x7fc39c0a0210 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:21.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): dump: 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:21.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3900044d0 con 0x7fc39c09ac90 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.305+0000 7fc38effd640 10 monclient(hunting): dump: 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:21.327 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1614421515 0 0) 0x7fc39c1275d0 con 0x7fc39c0a0210 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 >> v1:172.21.15.63:6789/0 conn(0x7fc39c06dd30 legacy=0x7fc39c134bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 >> v1:172.21.15.6:6789/0 conn(0x7fc39c09ac90 legacy=0x7fc39c1d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 monclient: found mon.c 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:21.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c125e40 con 0x7fc39c0a0210 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 10 monclient: _finish_auth 0 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 5 monclient: authenticate success, global_id 6215 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 10 monclient: _renew_subs 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc39c1272f0 con 0x7fc39c0a0210 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 10 monclient: _renew_subs 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc39c1c7860 con 0x7fc39c0a0210 2024-09-16T05:40:21.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3940017e0 con 0x7fc39c0a0210 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc394003b10 con 0x7fc39c0a0210 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a0d3e640 4 set_mon_vals no callback set 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 10 monclient: dump: 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.330 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc384005090 con 0x7fc39c0a0210 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc394003d30 con 0x7fc39c0a0210 2024-09-16T05:40:21.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(229..229 src has 20..229) ==== 11828+0+0 (unknown 1034095209 0 0) 0x7fc394044f40 con 0x7fc39c0a0210 2024-09-16T05:40:21.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.309+0000 7fc3a2541640 10 monclient: get_auth_request con 0x7fc3800369c0 auth_method 0 2024-09-16T05:40:21.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.321+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc394014240 con 0x7fc39c0a0210 2024-09-16T05:40:21.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.321+0000 7fc38effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:21.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.321+0000 7fc38effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:21.498 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.493+0000 7fc3a47a5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T05:40:21.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.493+0000 7fc3a47a5640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T05:40:21.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.493+0000 7fc3a47a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:21.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:21.493+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) -- 0x7fc384005380 con 0x7fc39c0a0210 2024-09-16T05:40:22.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.293+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 removed item id -52 name 'h12' from crush map v230) ==== 120+0+0 (unknown 1934994579 0 0) 0x7fc394013b40 con 0x7fc39c0a0210 2024-09-16T05:40:22.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.297+0000 7fc38effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T05:40:22.300 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.297+0000 7fc38effd640 10 monclient: _finish_command 2 = system:0 removed item id -52 name 'h12' from crush map 2024-09-16T05:40:22.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.401+0000 7fc3a47a5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T05:40:22.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.401+0000 7fc3a47a5640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T05:40:22.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.401+0000 7fc3a47a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:22.408 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.401+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) -- 0x7fc3840010a0 con 0x7fc39c0a0210 2024-09-16T05:40:22.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.405+0000 7fc38effd640 1 -- 172.21.15.63:0/1656370619 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 device 'h12' does not appear in the crush map v230) ==== 120+0+0 (unknown 2300756624 0 0) 0x7fc39401c690 con 0x7fc39c0a0210 2024-09-16T05:40:22.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.405+0000 7fc38effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T05:40:22.410 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.405+0000 7fc38effd640 10 monclient: _finish_command 3 = system:0 device 'h12' does not appear in the crush map 2024-09-16T05:40:22.410 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h12' does not appear in the crush map 2024-09-16T05:40:22.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.409+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc3800369c0 legacy=0x7fc380038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:22.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.409+0000 7fc3a47a5640 10 monclient: shutdown 2024-09-16T05:40:22.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.409+0000 7fc3a47a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:22.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.409+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 >> v1:172.21.15.6:6790/0 conn(0x7fc39c0a0210 legacy=0x7fc39c1d3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:22.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.409+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 shutdown_connections 2024-09-16T05:40:22.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.409+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 >> 172.21.15.63:0/1656370619 conn(0x7fc39c0a1c90 msgr2=0x7fc39c0a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:22.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.409+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 shutdown_connections 2024-09-16T05:40:22.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.409+0000 7fc3a47a5640 1 -- 172.21.15.63:0/1656370619 wait complete. 2024-09-16T05:40:22.428 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:22.428 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h13 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient: get_monmap_and_config 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient: build_initial_monmap 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient: monmap: 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:22.550055+0000 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:22.550055+0000 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:22.550 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:22.551 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:22.551 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:22.551 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:22.551 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:22.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 1 Processor -- start 2024-09-16T05:40:22.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 1 -- start start 2024-09-16T05:40:22.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient: init 2024-09-16T05:40:22.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:22.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:22.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient(hunting): picked mon.noname-b con 0x7f656c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:22.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient(hunting): picked mon.noname-c con 0x7f656c06dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient(hunting): picked mon.noname-a con 0x7f656c0a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656c09f800 con 0x7f656c0a0210 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656c0a0ab0 con 0x7f656c06dd30 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656c0a0d90 con 0x7f656c09ac90 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f6572180640 10 monclient(hunting): authenticate will time out at 1279.327064s 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f656affd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f656c06dd30 legacy=0x7f656c1399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f656bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f656c0a0210 0x7f656c13d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47752/0 (socket says 172.21.15.63:47752) 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f656bfff640 1 -- 172.21.15.63:0/1988795577 learned_addr learned my addr 172.21.15.63:0/1988795577 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.545+0000 7f656b7fe640 1 --1- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6790/0 conn(0x7f656c09ac90 0x7f656c09b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48460/0 (socket says 172.21.15.63:48460) 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4199271526 0 0) 0x7f656c0a0d90 con 0x7f656c09ac90 2024-09-16T05:40:22.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): my global_id is 6218 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6554003530 con 0x7f656c09ac90 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 684146728 0 0) 0x7f656c09f800 con 0x7f656c0a0210 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): my global_id is 6243 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f656c0a0d90 con 0x7f656c0a0210 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2541467152 0 0) 0x7f656c0a0d90 con 0x7f656c0a0210 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6790/0 conn(0x7f656c09ac90 legacy=0x7f656c09b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.63:6789/0 conn(0x7f656c06dd30 legacy=0x7f656c1399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:22.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 monclient: found mon.noname-a 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c09e770 con 0x7f656c0a0210 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6564001980 con 0x7f656c0a0210 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: dump: 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:22.556 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 5 monclient: authenticate success, global_id 6243 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656bfff640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6789/0 conn(0x7f656c0a0210 legacy=0x7f656c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6564001ce0 con 0x7f656c0a0210 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:22.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656bfff640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6789/0 conn(0x7f656c0a0210 legacy=0x7f656c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:40:22.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656bfff640 1 --1- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6789/0 conn(0x7f656c0a0210 0x7f656c13d0f0 :-1 s=OPENED pgs=815 cs=1 l=1).handle_message read tag failed 2024-09-16T05:40:22.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656bfff640 1 --1- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6789/0 conn(0x7f656c0a0210 0x7f656c13d0f0 :-1 s=OPENED pgs=815 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:40:22.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:22.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 4 set_mon_vals no callback set 2024-09-16T05:40:22.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.6:6789/0 2024-09-16T05:40:22.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:22.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6789/0 conn(0x7f656c0a0210 legacy=0x7f656c13d0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:22.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient: shutdown 2024-09-16T05:40:22.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:22.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): picked mon.a con 0x7f6554004d10 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): picked mon.b con 0x7f656c06dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): picked mon.c con 0x7f656c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f656c0a0d90 con 0x7f6554004d10 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f656c0a0ab0 con 0x7f656c06dd30 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 1 -- 172.21.15.63:0/1988795577 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6554003530 con 0x7f656c09ac90 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 20 monclient(hunting): shutdown discarding 1 pending message(s) 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6790/0 conn(0x7f656c09ac90 legacy=0x7f655400d480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.63:6789/0 conn(0x7f656c06dd30 legacy=0x7f6554009aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656affd640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.63:6789/0 conn(0x7f656c06dd30 legacy=0x7f6554009aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- 172.21.15.63:0/1988795577 >> v1:172.21.15.6:6789/0 conn(0x7f6554004d10 legacy=0x7f65540050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T05:40:22.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- 172.21.15.63:0/1988795577 shutdown_connections 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- 172.21.15.63:0/1988795577 >> 172.21.15.63:0/1988795577 conn(0x7f656c0a1c90 msgr2=0x7f656c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- 172.21.15.63:0/1988795577 shutdown_connections 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- 172.21.15.63:0/1988795577 wait complete. 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient: build_initial_monmap 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient: monmap: 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:22.565 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 Processor -- start 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- start start 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient: init 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient(hunting): picked mon.noname-b con 0x7f6554004d10 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient(hunting): picked mon.noname-a con 0x7f656c06dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 10 monclient(hunting): picked mon.noname-c con 0x7f656c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f656b7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6554004d10 legacy=0x7f656c1349f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656c0a0d90 con 0x7f656c06dd30 2024-09-16T05:40:22.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.549+0000 7f6572180640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656c0a0ab0 con 0x7f6554004d10 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6554003530 con 0x7f656c09ac90 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 monclient(hunting): authenticate will time out at 1279.332041s 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f656affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f656c06dd30 0x7f656c1ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47760/0 (socket says 172.21.15.63:47760) 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f656affd640 1 -- 172.21.15.63:0/393180191 learned_addr learned my addr 172.21.15.63:0/393180191 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847728448 0 0) 0x7f6554003530 con 0x7f656c09ac90 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): my global_id is 6221 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6558003530 con 0x7f656c09ac90 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3150796074 0 0) 0x7f656c0a0d90 con 0x7f656c06dd30 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:22.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): my global_id is 6246 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6554003530 con 0x7f656c06dd30 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 43197783 0 0) 0x7f6558003530 con 0x7f656c09ac90 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f656c0a0d90 con 0x7f656c09ac90 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6564004d10 con 0x7f656c09ac90 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): dump: 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:22.568 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2963982664 0 0) 0x7f6554003530 con 0x7f656c06dd30 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6558003530 con 0x7f656c06dd30 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6560004150 con 0x7f656c06dd30 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient(hunting): dump: 2024-09-16T05:40:22.569 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3080744837 0 0) 0x7f656c0a0d90 con 0x7f656c09ac90 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 >> v1:172.21.15.63:6789/0 conn(0x7f6554004d10 legacy=0x7f656c1349f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:22.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 >> v1:172.21.15.6:6789/0 conn(0x7f656c06dd30 legacy=0x7f656c1ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 monclient: found mon.c 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c09f800 con 0x7f656c09ac90 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient: _finish_auth 0 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 5 monclient: authenticate success, global_id 6221 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 monclient: _renew_subs 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f656c1223d0 con 0x7f656c09ac90 2024-09-16T05:40:22.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 monclient: _renew_subs 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f656c1220f0 con 0x7f656c09ac90 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6564001ce0 con 0x7f656c09ac90 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6564005160 con 0x7f656c09ac90 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6569ffb640 4 set_mon_vals no callback set 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 10 monclient: dump: 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:22.572 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f654c005090 con 0x7f656c09ac90 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f656400ff00 con 0x7f656c09ac90 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(230..230 src has 20..230) ==== 11615+0+0 (unknown 229245572 0 0) 0x7f6564046450 con 0x7f656c09ac90 2024-09-16T05:40:22.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.553+0000 7f656b7fe640 10 monclient: get_auth_request con 0x7f6558036980 auth_method 0 2024-09-16T05:40:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.565+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6564015880 con 0x7f656c09ac90 2024-09-16T05:40:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.565+0000 7f6553fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:22.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.565+0000 7f6553fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:22.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.733+0000 7f6572180640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T05:40:22.739 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.733+0000 7f6572180640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T05:40:22.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.733+0000 7f6572180640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:22.740 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:22.733+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) -- 0x7f654c005380 con 0x7f656c09ac90 2024-09-16T05:40:23.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.305+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 removed item id -55 name 'h13' from crush map v231) ==== 120+0+0 (unknown 1142768376 0 0) 0x7f6564015180 con 0x7f656c09ac90 2024-09-16T05:40:23.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.305+0000 7f6553fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T05:40:23.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.305+0000 7f6553fff640 10 monclient: _finish_command 2 = system:0 removed item id -55 name 'h13' from crush map 2024-09-16T05:40:23.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.409+0000 7f6572180640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T05:40:23.412 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.409+0000 7f6572180640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T05:40:23.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.409+0000 7f6572180640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:23.413 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.409+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) -- 0x7f654c0010a0 con 0x7f656c09ac90 2024-09-16T05:40:23.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.409+0000 7f6553fff640 1 -- 172.21.15.63:0/393180191 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 device 'h13' does not appear in the crush map v231) ==== 120+0+0 (unknown 1025174079 0 0) 0x7f65640407f0 con 0x7f656c09ac90 2024-09-16T05:40:23.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.409+0000 7f6553fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T05:40:23.415 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.409+0000 7f6553fff640 10 monclient: _finish_command 3 = system:0 device 'h13' does not appear in the crush map 2024-09-16T05:40:23.415 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h13' does not appear in the crush map 2024-09-16T05:40:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.413+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6558036980 legacy=0x7f6558038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.413+0000 7f6572180640 10 monclient: shutdown 2024-09-16T05:40:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.413+0000 7f6572180640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:23.417 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.413+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 >> v1:172.21.15.6:6790/0 conn(0x7f656c09ac90 legacy=0x7f656c1d1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:23.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.413+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 shutdown_connections 2024-09-16T05:40:23.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.413+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 >> 172.21.15.63:0/393180191 conn(0x7f656c0a1c90 msgr2=0x7f656c0a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:23.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.413+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 shutdown_connections 2024-09-16T05:40:23.418 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.413+0000 7f6572180640 1 -- 172.21.15.63:0/393180191 wait complete. 2024-09-16T05:40:23.432 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:23.432 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h14 2024-09-16T05:40:23.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient: get_monmap_and_config 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient: build_initial_monmap 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient: monmap: 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:23.553863+0000 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:23.553863+0000 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:23.554 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 1 Processor -- start 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 1 -- start start 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient: init 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient(hunting): picked mon.noname-c con 0x7f6d4809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4806dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient(hunting): picked mon.noname-b con 0x7f6d480a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:23.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d4809f800 con 0x7f6d4806dd30 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d480a0ab0 con 0x7f6d4809ac90 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d480a0d90 con 0x7f6d480a0210 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4caaa640 10 monclient(hunting): authenticate will time out at 1280.330773s 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6d4809ac90 legacy=0x7f6d4809b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d45d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6d4806dd30 0x7f6d481399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:47762/0 (socket says 172.21.15.63:47762) 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d45d9b640 1 -- 172.21.15.63:0/1912981094 learned_addr learned my addr 172.21.15.63:0/1912981094 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1014511342 0 0) 0x7f6d480a0d90 con 0x7f6d480a0210 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 10 monclient(hunting): my global_id is 6224 2024-09-16T05:40:23.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d30003530 con 0x7f6d480a0210 2024-09-16T05:40:23.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 391722714 0 0) 0x7f6d4809f800 con 0x7f6d4806dd30 2024-09-16T05:40:23.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:23.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:23.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:23.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 10 monclient(hunting): my global_id is 6249 2024-09-16T05:40:23.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.549+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d480a0d90 con 0x7f6d4806dd30 2024-09-16T05:40:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2243592308 0 0) 0x7f6d480a0d90 con 0x7f6d4806dd30 2024-09-16T05:40:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 >> v1:172.21.15.6:6790/0 conn(0x7f6d480a0210 legacy=0x7f6d4813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 >> v1:172.21.15.63:6789/0 conn(0x7f6d4809ac90 legacy=0x7f6d4809b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 1 monclient: found mon.noname-a 2024-09-16T05:40:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:23.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4809e770 con 0x7f6d4806dd30 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: _finish_auth 0 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d3c001600 con 0x7f6d4806dd30 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: dump: 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.564 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:23.565 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:23.568 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:23.568 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:23.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 5 monclient: authenticate success, global_id 6249 2024-09-16T05:40:23.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:23.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d3c001860 con 0x7f6d4806dd30 2024-09-16T05:40:23.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 1 -- 172.21.15.63:0/1912981094 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d3c001cb0 con 0x7f6d4806dd30 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 4 set_mon_vals no callback set 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4559a640 10 monclient: dump: 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:23.569 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient: shutdown 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- 172.21.15.63:0/1912981094 >> v1:172.21.15.6:6789/0 conn(0x7f6d4806dd30 legacy=0x7f6d481399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- 172.21.15.63:0/1912981094 shutdown_connections 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- 172.21.15.63:0/1912981094 >> 172.21.15.63:0/1912981094 conn(0x7f6d480a1c90 msgr2=0x7f6d480a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- 172.21.15.63:0/1912981094 shutdown_connections 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- 172.21.15.63:0/1912981094 wait complete. 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient: build_initial_monmap 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient: monmap: 2024-09-16T05:40:23.570 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 Processor -- start 2024-09-16T05:40:23.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- start start 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient: init 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient(hunting): picked mon.noname-b con 0x7f6d4806dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient(hunting): picked mon.noname-c con 0x7f6d480a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d4806e5b0 con 0x7f6d4809ac90 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d48124100 con 0x7f6d4806dd30 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d4806e160 con 0x7f6d480a0210 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4caaa640 10 monclient(hunting): authenticate will time out at 1280.335104s 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d4659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f6d4806dd30 legacy=0x7f6d48134d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:23.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d46d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6d480a0210 0x7f6d481d1740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:48488/0 (socket says 172.21.15.63:48488) 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d46d9d640 1 -- 172.21.15.63:0/3330311873 learned_addr learned my addr 172.21.15.63:0/3330311873 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1851485661 0 0) 0x7f6d4806e160 con 0x7f6d480a0210 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 10 monclient(hunting): my global_id is 6227 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d28003530 con 0x7f6d480a0210 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1451193592 0 0) 0x7f6d4809e770 con 0x7f6d4809ac90 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:23.573 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 10 monclient(hunting): my global_id is 6252 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.553+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d4806e160 con 0x7f6d4809ac90 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 197238514 0 0) 0x7f6d28003530 con 0x7f6d480a0210 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d4809e770 con 0x7f6d480a0210 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d40001cf0 con 0x7f6d480a0210 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:23.574 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1930905535 0 0) 0x7f6d4806e160 con 0x7f6d4809ac90 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d28003530 con 0x7f6d4809ac90 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d3c001bb0 con 0x7f6d4809ac90 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:23.575 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1573137932 0 0) 0x7f6d4809e770 con 0x7f6d480a0210 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 >> v1:172.21.15.63:6789/0 conn(0x7f6d4806dd30 legacy=0x7f6d48134d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 >> v1:172.21.15.6:6789/0 conn(0x7f6d4809ac90 legacy=0x7f6d481ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 monclient: found mon.c 2024-09-16T05:40:23.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d48125420 con 0x7f6d480a0210 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 5 monclient: authenticate success, global_id 6227 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 10 monclient: _renew_subs 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d48125700 con 0x7f6d480a0210 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 10 monclient: _renew_subs 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d481c8cd0 con 0x7f6d480a0210 2024-09-16T05:40:23.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d40001fd0 con 0x7f6d480a0210 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6d40003dd0 con 0x7f6d480a0210 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 10 monclient: dump: 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:23.578 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d37fff640 4 set_mon_vals no callback set 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d2c005090 con 0x7f6d480a0210 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6d40003ff0 con 0x7f6d480a0210 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(231..231 src has 20..231) ==== 11402+0+0 (unknown 1092850981 0 0) 0x7f6d40046350 con 0x7f6d480a0210 2024-09-16T05:40:23.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.557+0000 7f6d4659c640 10 monclient: get_auth_request con 0x7f6d28036a20 auth_method 0 2024-09-16T05:40:23.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.569+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6d40015730 con 0x7f6d480a0210 2024-09-16T05:40:23.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.569+0000 7f6d36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:23.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.569+0000 7f6d36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:23.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.733+0000 7f6d4caaa640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T05:40:23.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.733+0000 7f6d4caaa640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T05:40:23.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.733+0000 7f6d4caaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:23.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:23.733+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) -- 0x7f6d2c005380 con 0x7f6d480a0210 2024-09-16T05:40:24.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.313+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 removed item id -58 name 'h14' from crush map v232) ==== 120+0+0 (unknown 262369804 0 0) 0x7f6d40015030 con 0x7f6d480a0210 2024-09-16T05:40:24.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.313+0000 7f6d36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T05:40:24.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.313+0000 7f6d36ffd640 10 monclient: _finish_command 2 = system:0 removed item id -58 name 'h14' from crush map 2024-09-16T05:40:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.417+0000 7f6d4caaa640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T05:40:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.417+0000 7f6d4caaa640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T05:40:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.417+0000 7f6d4caaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:24.422 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.417+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) -- 0x7f6d2c0010a0 con 0x7f6d480a0210 2024-09-16T05:40:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.417+0000 7f6d36ffd640 1 -- 172.21.15.63:0/3330311873 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 device 'h14' does not appear in the crush map v232) ==== 120+0+0 (unknown 1448980209 0 0) 0x7f6d40019380 con 0x7f6d480a0210 2024-09-16T05:40:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.417+0000 7f6d36ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T05:40:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.417+0000 7f6d36ffd640 10 monclient: _finish_command 3 = system:0 device 'h14' does not appear in the crush map 2024-09-16T05:40:24.424 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h14' does not appear in the crush map 2024-09-16T05:40:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.421+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6d28036a20 legacy=0x7f6d28038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.421+0000 7f6d4caaa640 10 monclient: shutdown 2024-09-16T05:40:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.421+0000 7f6d4caaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.421+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 >> v1:172.21.15.6:6790/0 conn(0x7f6d480a0210 legacy=0x7f6d481d1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.421+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 shutdown_connections 2024-09-16T05:40:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.421+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 >> 172.21.15.63:0/3330311873 conn(0x7f6d480a1c90 msgr2=0x7f6d4811b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:24.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.421+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 shutdown_connections 2024-09-16T05:40:24.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.421+0000 7f6d4caaa640 1 -- 172.21.15.63:0/3330311873 wait complete. 2024-09-16T05:40:24.441 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:24.441 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h15 2024-09-16T05:40:24.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.553+0000 7f834f001640 10 monclient: get_monmap_and_config 2024-09-16T05:40:24.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.553+0000 7f834f001640 10 monclient: build_initial_monmap 2024-09-16T05:40:24.559 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.553+0000 7f834f001640 10 monclient: monmap: 2024-09-16T05:40:24.559 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:24.559 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:24.559 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:24.559490+0000 2024-09-16T05:40:24.559 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:24.559490+0000 2024-09-16T05:40:24.560 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:24.560 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:24.560 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:24.560 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:24.560 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:24.560 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:24.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.553+0000 7f834f001640 1 Processor -- start 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.553+0000 7f834f001640 1 -- start start 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.553+0000 7f834f001640 10 monclient: init 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient(hunting): picked mon.noname-a con 0x7f83480a3d80 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient(hunting): picked mon.noname-b con 0x7f83480a4870 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient(hunting): picked mon.noname-c con 0x7f834813d4a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83480a0880 con 0x7f83480a3d80 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83480a5340 con 0x7f834813d4a0 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83480a10b0 con 0x7f83480a4870 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:24.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient(hunting): authenticate will time out at 1281.336399s 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834d59e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f834813d4a0 legacy=0x7f834813f890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f8347fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f83480a4870 0x7f834813bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:58690/0 (socket says 172.21.15.63:58690) 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f8347fff640 1 -- 172.21.15.63:0/1353225041 learned_addr learned my addr 172.21.15.63:0/1353225041 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076805965 0 0) 0x7f83480a10b0 con 0x7f83480a4870 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient(hunting): my global_id is 6230 2024-09-16T05:40:24.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8334003530 con 0x7f83480a4870 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 271482117 0 0) 0x7f83480a0880 con 0x7f83480a3d80 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient(hunting): my global_id is 6255 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83480a10b0 con 0x7f83480a3d80 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3195857410 0 0) 0x7f8334003530 con 0x7f83480a4870 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 >> v1:172.21.15.63:6789/0 conn(0x7f834813d4a0 legacy=0x7f834813f890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 >> v1:172.21.15.6:6789/0 conn(0x7f83480a3d80 legacy=0x7f83480a4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 monclient: found mon.noname-b 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:24.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83480a1390 con 0x7f83480a4870 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8338001620 con 0x7f83480a4870 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: dump: 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.567 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 5 monclient: authenticate success, global_id 6230 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8338001880 con 0x7f83480a4870 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 1 -- 172.21.15.63:0/1353225041 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8338001bd0 con 0x7f83480a4870 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:24.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f83477fe640 10 monclient: dump: 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 4 set_mon_vals no callback set 2024-09-16T05:40:24.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient: shutdown 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- 172.21.15.63:0/1353225041 >> v1:172.21.15.6:6790/0 conn(0x7f83480a4870 legacy=0x7f834813bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- 172.21.15.63:0/1353225041 shutdown_connections 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- 172.21.15.63:0/1353225041 >> 172.21.15.63:0/1353225041 conn(0x7f8348075470 msgr2=0x7f8348075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- 172.21.15.63:0/1353225041 shutdown_connections 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- 172.21.15.63:0/1353225041 wait complete. 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient: build_initial_monmap 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient: monmap: 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:24.570 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:24.571 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:24.571 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:24.571 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:24.571 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:24.571 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:24.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 Processor -- start 2024-09-16T05:40:24.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 1 -- start start 2024-09-16T05:40:24.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.557+0000 7f834f001640 10 monclient: init 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient(hunting): picked mon.noname-a con 0x7f83480a3d80 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient(hunting): picked mon.noname-c con 0x7f83480a4870 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient(hunting): picked mon.noname-b con 0x7f834813d4a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8348070cf0 con 0x7f83480a3d80 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8348072450 con 0x7f834813d4a0 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83480a3020 con 0x7f83480a4870 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834d59e640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f834813d4a0 legacy=0x7f83481d3670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:24.575 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient(hunting): authenticate will time out at 1281.340444s 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834cd9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f83480a3d80 0x7f834806fed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:38036/0 (socket says 172.21.15.63:38036) 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834cd9d640 1 -- 172.21.15.63:0/1660245852 learned_addr learned my addr 172.21.15.63:0/1660245852 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2518254763 0 0) 0x7f8348070cf0 con 0x7f83480a3d80 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): my global_id is 6258 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8328003530 con 0x7f83480a3d80 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217890360 0 0) 0x7f83480a3020 con 0x7f83480a4870 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:24.576 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): my global_id is 6233 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8348070cf0 con 0x7f83480a4870 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2764960828 0 0) 0x7f8328003530 con 0x7f83480a3d80 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83480a3020 con 0x7f83480a3d80 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8330001980 con 0x7f83480a3d80 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): dump: 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:24.577 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2949364663 0 0) 0x7f8348070cf0 con 0x7f83480a4870 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8328003530 con 0x7f83480a4870 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8338001560 con 0x7f83480a4870 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient(hunting): dump: 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:24.578 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4173028971 0 0) 0x7f83480a3020 con 0x7f83480a3d80 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 >> v1:172.21.15.6:6790/0 conn(0x7f83480a4870 legacy=0x7f83480705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 >> v1:172.21.15.63:6789/0 conn(0x7f834813d4a0 legacy=0x7f83481d3670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient: _finish_hunting 0 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 monclient: found mon.a 2024-09-16T05:40:24.579 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83480a0570 con 0x7f83480a3d80 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient: _finish_auth 0 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient: _check_auth_tickets 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 5 monclient: authenticate success, global_id 6258 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient: _renew_subs 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83481383d0 con 0x7f83480a3d80 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient: _renew_subs 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83481390f0 con 0x7f83480a3d80 2024-09-16T05:40:24.580 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8330001c60 con 0x7f83480a3d80 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8330003a70 con 0x7f83480a3d80 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f8346ffd640 4 set_mon_vals no callback set 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 10 monclient: dump: 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:24.581 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f832c005090 con 0x7f83480a3d80 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8330003c90 con 0x7f83480a3d80 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.561+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(232..232 src has 20..232) ==== 11189+0+0 (unknown 377724488 0 0) 0x7f8330046010 con 0x7f83480a3d80 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.565+0000 7f8347fff640 10 monclient: get_auth_request con 0x7f8328036a20 auth_method 0 2024-09-16T05:40:24.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.577+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f83300154f0 con 0x7f83480a3d80 2024-09-16T05:40:24.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.577+0000 7f83457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:24.583 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.577+0000 7f83457fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.741+0000 7f834f001640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T05:40:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.741+0000 7f834f001640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T05:40:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.741+0000 7f834f001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:24.747 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:24.741+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) -- 0x7f832c005380 con 0x7f83480a3d80 2024-09-16T05:40:25.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.317+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 removed item id -61 name 'h15' from crush map v233) ==== 120+0+0 (unknown 4183218660 0 0) 0x7f8330014df0 con 0x7f83480a3d80 2024-09-16T05:40:25.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.317+0000 7f83457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T05:40:25.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.317+0000 7f83457fa640 10 monclient: _finish_command 2 = system:0 removed item id -61 name 'h15' from crush map 2024-09-16T05:40:25.352 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:40:25.352 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:40:25.353 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:40:25.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:40:25.356 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:40:25.356 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:40:25.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:40:25.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f834f001640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T05:40:25.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f834f001640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T05:40:25.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f834f001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:25.425 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) -- 0x7f832c0010a0 con 0x7f83480a3d80 2024-09-16T05:40:25.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f83457fa640 1 -- 172.21.15.63:0/1660245852 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 device 'h15' does not appear in the crush map v233) ==== 120+0+0 (unknown 3798304350 0 0) 0x7f8330019140 con 0x7f83480a3d80 2024-09-16T05:40:25.426 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f83457fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T05:40:25.427 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f83457fa640 10 monclient: _finish_command 3 = system:0 device 'h15' does not appear in the crush map 2024-09-16T05:40:25.427 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h15' does not appear in the crush map 2024-09-16T05:40:25.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8328036a20 legacy=0x7f8328038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f834f001640 10 monclient: shutdown 2024-09-16T05:40:25.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f834f001640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:25.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.421+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 >> v1:172.21.15.6:6789/0 conn(0x7f83480a3d80 legacy=0x7f834806fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.428 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.425+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 shutdown_connections 2024-09-16T05:40:25.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.425+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 >> 172.21.15.63:0/1660245852 conn(0x7f8348075470 msgr2=0x7f8348133fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:25.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.425+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 shutdown_connections 2024-09-16T05:40:25.429 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.425+0000 7f834f001640 1 -- 172.21.15.63:0/1660245852 wait complete. 2024-09-16T05:40:25.444 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:25.444 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h16 2024-09-16T05:40:25.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient: get_monmap_and_config 2024-09-16T05:40:25.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient: build_initial_monmap 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient: monmap: 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:25.488260+0000 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:25.488260+0000 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:25.488 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 1 Processor -- start 2024-09-16T05:40:25.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 1 -- start start 2024-09-16T05:40:25.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient: init 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient(hunting): picked mon.noname-c con 0x7f123409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient(hunting): picked mon.noname-b con 0x7f123409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient(hunting): picked mon.noname-a con 0x7f12340a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12340a1370 con 0x7f12340a0da0 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f123409ead0 con 0x7f123409aa30 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f123409edb0 con 0x7f123409e230 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient(hunting): authenticate will time out at 1287.294918s 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1238b1e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12340a0da0 0x7f123413d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33580/0 (socket says 172.21.15.6:33580) 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1238b1e640 1 -- 172.21.15.6:0/502980591 learned_addr learned my addr 172.21.15.6:0/502980591 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:25.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2770262803 0 0) 0x7f12340a1370 con 0x7f12340a0da0 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient(hunting): my global_id is 6261 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f121c003530 con 0x7f12340a0da0 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 861589523 0 0) 0x7f123409edb0 con 0x7f123409e230 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient(hunting): my global_id is 6236 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12340a1370 con 0x7f123409e230 2024-09-16T05:40:25.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1591229859 0 0) 0x7f121c003530 con 0x7f12340a0da0 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 >> v1:172.21.15.6:6790/0 conn(0x7f123409e230 legacy=0x7f12341399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 >> v1:172.21.15.63:6789/0 conn(0x7f123409aa30 legacy=0x7f123409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 monclient: found mon.noname-a 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123409dd50 con 0x7f12340a0da0 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1224001620 con 0x7f12340a0da0 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: dump: 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 5 monclient: authenticate success, global_id 6261 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12240018d0 con 0x7f12340a0da0 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f123a581640 4 set_mon_vals no callback set 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 1 -- 172.21.15.6:0/502980591 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1224001ca0 con 0x7f12340a0da0 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.484+0000 7f1232ffd640 10 monclient: dump: 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.496 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: shutdown 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- 172.21.15.6:0/502980591 >> v1:172.21.15.6:6789/0 conn(0x7f12340a0da0 legacy=0x7f123413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- 172.21.15.6:0/502980591 shutdown_connections 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- 172.21.15.6:0/502980591 >> 172.21.15.6:0/502980591 conn(0x7f12340a1650 msgr2=0x7f12340a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- 172.21.15.6:0/502980591 shutdown_connections 2024-09-16T05:40:25.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- 172.21.15.6:0/502980591 wait complete. 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: build_initial_monmap 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: monmap: 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 Processor -- start 2024-09-16T05:40:25.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- start start 2024-09-16T05:40:25.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: init 2024-09-16T05:40:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient(hunting): picked mon.noname-a con 0x7f123409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient(hunting): picked mon.noname-c con 0x7f123409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient(hunting): picked mon.noname-b con 0x7f12340a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f123411eb70 con 0x7f123409aa30 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f123413da30 con 0x7f12340a0da0 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f123406ea20 con 0x7f123409e230 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient(hunting): authenticate will time out at 1287.298005s 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1233fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f123409aa30 0x7f12341cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33594/0 (socket says 172.21.15.6:33594) 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1233fff640 1 -- 172.21.15.6:0/731665888 learned_addr learned my addr 172.21.15.6:0/731665888 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f12337fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f123409e230 0x7f12341d0a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60952/0 (socket says 172.21.15.6:60952) 2024-09-16T05:40:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778354818 0 0) 0x7f123411eb70 con 0x7f123409aa30 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): my global_id is 6264 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1220003530 con 0x7f123409aa30 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1650144472 0 0) 0x7f123406ea20 con 0x7f123409e230 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): my global_id is 6239 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f123411eb70 con 0x7f123409e230 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2453159239 0 0) 0x7f123411eb70 con 0x7f123409e230 2024-09-16T05:40:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f123406ea20 con 0x7f123409e230 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12280018c0 con 0x7f123409e230 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): dump: 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3008575294 0 0) 0x7f1220003530 con 0x7f123409aa30 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f123411eb70 con 0x7f123409aa30 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f122c001c20 con 0x7f123409aa30 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient(hunting): dump: 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 43165647 0 0) 0x7f123406ea20 con 0x7f123409e230 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 >> v1:172.21.15.63:6789/0 conn(0x7f12340a0da0 legacy=0x7f12341d4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 >> v1:172.21.15.6:6789/0 conn(0x7f123409aa30 legacy=0x7f12341cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 monclient: found mon.c 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12340740c0 con 0x7f123409e230 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient: _finish_auth 0 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 5 monclient: authenticate success, global_id 6239 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: _renew_subs 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f123411e5b0 con 0x7f123409e230 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: _renew_subs 2024-09-16T05:40:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f123411d160 con 0x7f123409e230 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1228003620 con 0x7f123409e230 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1228003970 con 0x7f123409e230 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 10 monclient: dump: 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1231ffb640 4 set_mon_vals no callback set 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1228003b90 con 0x7f123409e230 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.488+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1214005090 con 0x7f123409e230 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.492+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(233..233 src has 20..233) ==== 10976+0+0 (unknown 3366826510 0 0) 0x7f1228045e90 con 0x7f123409e230 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.492+0000 7f1233fff640 10 monclient: get_auth_request con 0x7f1220036a70 auth_method 0 2024-09-16T05:40:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.496+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1228015440 con 0x7f123409e230 2024-09-16T05:40:25.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.496+0000 7f1230ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:25.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.496+0000 7f1230ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:25.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient: get_monmap_and_config 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient: build_initial_monmap 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient: monmap: 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:25.550006+0000 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:25.550006+0000 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:25.550 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:25.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 1 Processor -- start 2024-09-16T05:40:25.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 1 -- start start 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient: init 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient(hunting): picked mon.noname-c con 0x7f29a80a1c40 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient(hunting): picked mon.noname-b con 0x7f29a80a2730 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient(hunting): picked mon.noname-a con 0x7f29a80a0c50 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a809ec90 con 0x7f29a80a0c50 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a809fc30 con 0x7f29a80a1c40 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a809ff10 con 0x7f29a80a2730 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29ad592640 10 monclient(hunting): authenticate will time out at 1282.327259s 2024-09-16T05:40:25.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29a6ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f29a80a1c40 0x7f29a80a2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54122/0 (socket says 172.21.15.63:54122) 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29a6ffd640 1 -- 172.21.15.63:0/689924992 learned_addr learned my addr 172.21.15.63:0/689924992 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2446514750 0 0) 0x7f29a809ff10 con 0x7f29a80a2730 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29a5ffb640 10 monclient(hunting): my global_id is 6242 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.545+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2990003530 con 0x7f29a80a2730 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3971360578 0 0) 0x7f2990003530 con 0x7f29a80a2730 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 >> v1:172.21.15.63:6789/0 conn(0x7f29a80a1c40 legacy=0x7f29a80a2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 >> v1:172.21.15.6:6789/0 conn(0x7f29a80a0c50 legacy=0x7f29a813f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 1 monclient: found mon.noname-b 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:25.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a80a01f0 con 0x7f29a80a2730 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f299c001620 con 0x7f29a80a2730 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: dump: 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.555 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 5 monclient: authenticate success, global_id 6242 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f299c001910 con 0x7f29a80a2730 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 1 -- 172.21.15.63:0/689924992 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f299c001ce0 con 0x7f29a80a2730 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:25.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a5ffb640 10 monclient: dump: 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:25.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 4 set_mon_vals no callback set 2024-09-16T05:40:25.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient: shutdown 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- 172.21.15.63:0/689924992 >> v1:172.21.15.6:6790/0 conn(0x7f29a80a2730 legacy=0x7f29a813bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- 172.21.15.63:0/689924992 shutdown_connections 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- 172.21.15.63:0/689924992 >> 172.21.15.63:0/689924992 conn(0x7f29a80a3e90 msgr2=0x7f29a80a42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- 172.21.15.63:0/689924992 shutdown_connections 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- 172.21.15.63:0/689924992 wait complete. 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient: build_initial_monmap 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient: monmap: 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:25.562 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 Processor -- start 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- start start 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient: init 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient(hunting): picked mon.noname-c con 0x7f29a80a0c50 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient(hunting): picked mon.noname-b con 0x7f29a80a1c40 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:25.563 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient(hunting): picked mon.noname-a con 0x7f29a80a2730 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a8132250 con 0x7f29a80a2730 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a809b030 con 0x7f29a80a1c40 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29a81337d0 con 0x7f29a80a0c50 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29ad592640 10 monclient(hunting): authenticate will time out at 1282.330988s 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a67fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f29a80a1c40 0x7f29a8071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54124/0 (socket says 172.21.15.63:54124) 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.549+0000 7f29a67fc640 1 -- 172.21.15.63:0/2249911186 learned_addr learned my addr 172.21.15.63:0/2249911186 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 119377449 0 0) 0x7f29a80a01f0 con 0x7f29a80a1c40 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): my global_id is 114100 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2994003530 con 0x7f29a80a1c40 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 88609703 0 0) 0x7f29a8132250 con 0x7f29a80a2730 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): my global_id is 6267 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a80a01f0 con 0x7f29a80a2730 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1490146366 0 0) 0x7f2994003530 con 0x7f29a80a1c40 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29a8132250 con 0x7f29a80a1c40 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f299c0016a0 con 0x7f29a80a1c40 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:25.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33874038 0 0) 0x7f29a809fc30 con 0x7f29a80a0c50 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): my global_id is 6245 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2994003530 con 0x7f29a80a0c50 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2782467175 0 0) 0x7f29a80a01f0 con 0x7f29a80a2730 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29a809fc30 con 0x7f29a80a2730 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f29a0001ab0 con 0x7f29a80a2730 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.567 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1570229343 0 0) 0x7f29a8132250 con 0x7f29a80a1c40 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 >> v1:172.21.15.6:6790/0 conn(0x7f29a80a0c50 legacy=0x7f29a80710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 >> v1:172.21.15.6:6789/0 conn(0x7f29a80a2730 legacy=0x7f29a81d3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 monclient: found mon.b 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a809e180 con 0x7f29a80a1c40 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 5 monclient: authenticate success, global_id 114100 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 10 monclient: _renew_subs 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29a8134070 con 0x7f29a80a1c40 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 10 monclient: _renew_subs 2024-09-16T05:40:25.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29a80729e0 con 0x7f29a80a1c40 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f299c003b80 con 0x7f29a80a1c40 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f299c003db0 con 0x7f29a80a1c40 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 10 monclient: dump: 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.570 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29a57fa640 4 set_mon_vals no callback set 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f299c003fd0 con 0x7f29a80a1c40 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(233..233 src has 20..233) ==== 10976+0+0 (unknown 3366826510 0 0) 0x7f299c045fe0 con 0x7f29a80a1c40 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a8133cf0 con 0x7f29a80a1c40 2024-09-16T05:40:25.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.553+0000 7f29a6ffd640 10 monclient: get_auth_request con 0x7f2994036de0 auth_method 0 2024-09-16T05:40:25.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.557+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f29a809b030 con 0x7f29a80a1c40 2024-09-16T05:40:25.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.557+0000 7f298f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:25.572 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.557+0000 7f298f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:25.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.721+0000 7f29ad592640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T05:40:25.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.721+0000 7f29ad592640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T05:40:25.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.721+0000 7f29ad592640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:25.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:25.721+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) -- 0x7f29a8073c40 con 0x7f29a80a1c40 2024-09-16T05:40:25.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.724+0000 7f123a581640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:40:25.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.724+0000 7f123a581640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:25.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.724+0000 7f123a581640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:25.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.724+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1214005380 con 0x7f123409e230 2024-09-16T05:40:25.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f1230ff9640 1 -- 172.21.15.6:0/731665888 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3397411323) 0x7f1228014d40 con 0x7f123409e230 2024-09-16T05:40:25.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f1230ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:25.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f1230ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:25.731 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:40:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1220036a70 legacy=0x7f1220038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f123a581640 10 monclient: shutdown 2024-09-16T05:40:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f123a581640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 >> v1:172.21.15.6:6790/0 conn(0x7f123409e230 legacy=0x7f12341d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 shutdown_connections 2024-09-16T05:40:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 >> 172.21.15.6:0/731665888 conn(0x7f12340a1650 msgr2=0x7f1234071e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:25.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 shutdown_connections 2024-09-16T05:40:25.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.728+0000 7f123a581640 1 -- 172.21.15.6:0/731665888 wait complete. 2024-09-16T05:40:25.750 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:40:25.750 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:40:25.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient: get_monmap_and_config 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient: build_initial_monmap 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient: monmap: 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:25.871504+0000 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:25.871504+0000 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.871 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:25.872 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 1 Processor -- start 2024-09-16T05:40:25.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 1 -- start start 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient: init 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient(hunting): picked mon.noname-c con 0x7f78a80a0df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient(hunting): picked mon.noname-a con 0x7f78a8135330 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient(hunting): picked mon.noname-b con 0x7f78a8138e40 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a809d2f0 con 0x7f78a8135330 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a813d960 con 0x7f78a80a0df0 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a813eb40 con 0x7f78a8138e40 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient(hunting): authenticate will time out at 1287.678608s 2024-09-16T05:40:25.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78ad879640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f78a8138e40 0x7f78a813b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60954/0 (socket says 172.21.15.6:60954) 2024-09-16T05:40:25.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78ac877640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f78a8135330 0x7f78a8137720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33596/0 (socket says 172.21.15.6:33596) 2024-09-16T05:40:25.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78ad879640 1 -- 172.21.15.6:0/2282677153 learned_addr learned my addr 172.21.15.6:0/2282677153 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:25.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 791215154 0 0) 0x7f78a813eb40 con 0x7f78a8138e40 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient(hunting): my global_id is 6248 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7894003450 con 0x7f78a8138e40 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165647542 0 0) 0x7f78a809d2f0 con 0x7f78a8135330 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient(hunting): my global_id is 6270 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a813eb40 con 0x7f78a8135330 2024-09-16T05:40:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 971565756 0 0) 0x7f7894003450 con 0x7f78a8138e40 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 >> v1:172.21.15.63:6789/0 conn(0x7f78a80a0df0 legacy=0x7f78a809de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 >> v1:172.21.15.6:6789/0 conn(0x7f78a8135330 legacy=0x7f78a8137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 monclient: found mon.noname-b 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a813fd20 con 0x7f78a8138e40 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: _finish_auth 0 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78a0001620 con 0x7f78a8138e40 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: dump: 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 5 monclient: authenticate success, global_id 6248 2024-09-16T05:40:25.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78a00018d0 con 0x7f78a8138e40 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 1 -- 172.21.15.6:0/2282677153 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78a0001ce0 con 0x7f78a8138e40 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f789bfff640 10 monclient: dump: 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.880 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.868+0000 7f78af2dc640 4 set_mon_vals no callback set 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: shutdown 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- 172.21.15.6:0/2282677153 >> v1:172.21.15.6:6790/0 conn(0x7f78a8138e40 legacy=0x7f78a813b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- 172.21.15.6:0/2282677153 shutdown_connections 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- 172.21.15.6:0/2282677153 >> 172.21.15.6:0/2282677153 conn(0x7f78a80a1650 msgr2=0x7f78a80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- 172.21.15.6:0/2282677153 shutdown_connections 2024-09-16T05:40:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- 172.21.15.6:0/2282677153 wait complete. 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: build_initial_monmap 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: monmap: 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:25.887 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 Processor -- start 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- start start 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: init 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient(hunting): picked mon.noname-a con 0x7f78a80a0df0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient(hunting): picked mon.noname-b con 0x7f78a8135330 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient(hunting): picked mon.noname-c con 0x7f78a8138e40 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a806ef70 con 0x7f78a80a0df0 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a8070270 con 0x7f78a8135330 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a806f410 con 0x7f78a8138e40 2024-09-16T05:40:25.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient(hunting): authenticate will time out at 1287.682278s 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78ad078640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f78a80a0df0 0x7f78a8134770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33604/0 (socket says 172.21.15.6:33604) 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78ad078640 1 -- 172.21.15.6:0/2943274316 learned_addr learned my addr 172.21.15.6:0/2943274316 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530151070 0 0) 0x7f78a806ef70 con 0x7f78a80a0df0 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): my global_id is 6273 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f788c003560 con 0x7f78a80a0df0 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3878514537 0 0) 0x7f78a806f410 con 0x7f78a8138e40 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): my global_id is 6251 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a806ef70 con 0x7f78a8138e40 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3224634192 0 0) 0x7f78a8070270 con 0x7f78a8135330 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): my global_id is 114106 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a806f410 con 0x7f78a8135330 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3417972343 0 0) 0x7f788c003560 con 0x7f78a80a0df0 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78a8070270 con 0x7f78a80a0df0 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f789c001920 con 0x7f78a80a0df0 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:25.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): dump: 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3998218038 0 0) 0x7f78a806ef70 con 0x7f78a8138e40 2024-09-16T05:40:25.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f788c003560 con 0x7f78a8138e40 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78a0001a80 con 0x7f78a8138e40 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient(hunting): dump: 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.892 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3773578516 0 0) 0x7f78a8070270 con 0x7f78a80a0df0 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 >> v1:172.21.15.6:6790/0 conn(0x7f78a8138e40 legacy=0x7f78a81cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 >> v1:172.21.15.63:6789/0 conn(0x7f78a8135330 legacy=0x7f78a81cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 monclient: found mon.a 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a8072f20 con 0x7f78a80a0df0 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:25.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 5 monclient: authenticate success, global_id 6273 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: _renew_subs 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78a813c120 con 0x7f78a80a0df0 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: _renew_subs 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78a8135050 con 0x7f78a80a0df0 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f789c003620 con 0x7f78a80a0df0 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f789c003970 con 0x7f78a80a0df0 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f789b7fe640 4 set_mon_vals no callback set 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.872+0000 7f7899ffb640 10 monclient: dump: 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f78af2dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:25.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78880065d0 con 0x7f7888002870 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3083969149 0 0) 0x7f78880065d0 con 0x7f7888002870 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient(hunting): my global_id is 6276 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a8070270 con 0x7f7888002870 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f789c003b90 con 0x7f78a80a0df0 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(233..233 src has 20..233) ==== 10976+0+0 (unknown 3366826510 0 0) 0x7f789c045ec0 con 0x7f78a80a0df0 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3935409620 0 0) 0x7f78a8070270 con 0x7f7888002870 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78880065d0 con 0x7f7888002870 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78a4001c00 con 0x7f7888002870 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient: dump: 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f78ad879640 10 monclient: get_auth_request con 0x7f788c036d50 auth_method 0 2024-09-16T05:40:25.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1537337519 0 0) 0x7f78880065d0 con 0x7f7888002870 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f788800a160 con 0x7f7888002870 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f78a4001940 con 0x7f7888002870 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 >> v1:172.21.15.6:6789/0 conn(0x7f7888002870 legacy=0x7f7888004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.876+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 >> v1:172.21.15.6:6789/0 conn(0x7f7888002870 legacy=0x7f7888004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.880+0000 7f78af2dc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.880+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7888009e30 con 0x7f78880090f0 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699001032 0 0) 0x7f7888009e30 con 0x7f78880090f0 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:25.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient(hunting): my global_id is 6279 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78880065d0 con 0x7f78880090f0 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3820496126 0 0) 0x7f78880065d0 con 0x7f78880090f0 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7888009e30 con 0x7f78880090f0 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78a4004d10 con 0x7f78880090f0 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient: dump: 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:25.899 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2792620510 0 0) 0x7f7888009e30 con 0x7f78880090f0 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f788800c3a0 con 0x7f78880090f0 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 4242380170) 0x7f78a4006160 con 0x7f78880090f0 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 >> v1:172.21.15.6:6789/0 conn(0x7f78880090f0 legacy=0x7f7888009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f7899ffb640 1 -- 172.21.15.6:0/2943274316 >> v1:172.21.15.6:6789/0 conn(0x7f78880090f0 legacy=0x7f7888009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 26, 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 354940, 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:25.911 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:25.912 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:40:25.913 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 >> v1:172.21.15.63:6812/3643207211 conn(0x7f788c036d50 legacy=0x7f788c039210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78af2dc640 10 monclient: shutdown 2024-09-16T05:40:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78af2dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 >> v1:172.21.15.6:6789/0 conn(0x7f78a80a0df0 legacy=0x7f78a8134770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78ad879640 1 -- 172.21.15.6:0/2943274316 reap_dead start 2024-09-16T05:40:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78ad879640 1 -- 172.21.15.6:0/2943274316 reap_dead start 2024-09-16T05:40:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 shutdown_connections 2024-09-16T05:40:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 >> 172.21.15.6:0/2943274316 conn(0x7f78a80a1650 msgr2=0x7f78a809b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 shutdown_connections 2024-09-16T05:40:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:25.884+0000 7f78af2dc640 1 -- 172.21.15.6:0/2943274316 wait complete. 2024-09-16T05:40:25.915 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:40:25.915 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:40:25.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:40:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.024+0000 7f224dd70640 10 monclient: get_monmap_and_config 2024-09-16T05:40:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.024+0000 7f224dd70640 10 monclient: build_initial_monmap 2024-09-16T05:40:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.024+0000 7f224dd70640 10 monclient: monmap: 2024-09-16T05:40:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:26.029881+0000 2024-09-16T05:40:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:26.029881+0000 2024-09-16T05:40:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:26.031 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 Processor -- start 2024-09-16T05:40:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- start start 2024-09-16T05:40:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient: init 2024-09-16T05:40:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient(hunting): picked mon.noname-c con 0x7f22480a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient(hunting): picked mon.noname-b con 0x7f22480a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient(hunting): picked mon.noname-a con 0x7f22480a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22480a33a0 con 0x7f22480a2980 2024-09-16T05:40:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22480a07a0 con 0x7f22480a2190 2024-09-16T05:40:26.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22480a2d90 con 0x7f22480a2570 2024-09-16T05:40:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient(hunting): authenticate will time out at 1287.837140s 2024-09-16T05:40:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f2247fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f22480a2980 0x7f224813f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33630/0 (socket says 172.21.15.6:33630) 2024-09-16T05:40:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f2246ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f22480a2570 0x7f224813b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60968/0 (socket says 172.21.15.6:60968) 2024-09-16T05:40:26.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f2247fff640 1 -- 172.21.15.6:0/232981808 learned_addr learned my addr 172.21.15.6:0/232981808 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:26.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069916801 0 0) 0x7f22480a2d90 con 0x7f22480a2570 2024-09-16T05:40:26.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient(hunting): my global_id is 6254 2024-09-16T05:40:26.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2230003530 con 0x7f22480a2570 2024-09-16T05:40:26.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3842062558 0 0) 0x7f22480a33a0 con 0x7f22480a2980 2024-09-16T05:40:26.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient(hunting): my global_id is 6282 2024-09-16T05:40:26.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22480a2d90 con 0x7f22480a2980 2024-09-16T05:40:26.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2130292285 0 0) 0x7f22480a2d90 con 0x7f22480a2980 2024-09-16T05:40:26.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 >> v1:172.21.15.6:6790/0 conn(0x7f22480a2570 legacy=0x7f224813b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 >> v1:172.21.15.63:6789/0 conn(0x7f22480a2190 legacy=0x7f22480a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:26.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 monclient: found mon.noname-a 2024-09-16T05:40:26.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:26.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:26.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22480a66e0 con 0x7f22480a2980 2024-09-16T05:40:26.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:26.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:26.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:26.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2240001980 con 0x7f22480a2980 2024-09-16T05:40:26.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 5 monclient: authenticate success, global_id 6282 2024-09-16T05:40:26.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:26.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:26.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: dump: 2024-09-16T05:40:26.057 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.057 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.057 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.057 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.057 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2240001c60 con 0x7f22480a2980 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 1 -- 172.21.15.6:0/232981808 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2240002070 con 0x7f22480a2980 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 4 set_mon_vals no callback set 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:26.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f22467fc640 10 monclient: dump: 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient: shutdown 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:26.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- 172.21.15.6:0/232981808 >> v1:172.21.15.6:6789/0 conn(0x7f22480a2980 legacy=0x7f224813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- 172.21.15.6:0/232981808 shutdown_connections 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- 172.21.15.6:0/232981808 >> 172.21.15.6:0/232981808 conn(0x7f22480a9850 msgr2=0x7f22480a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- 172.21.15.6:0/232981808 shutdown_connections 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- 172.21.15.6:0/232981808 wait complete. 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient: build_initial_monmap 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient: monmap: 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.060 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 Processor -- start 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 1 -- start start 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.028+0000 7f224dd70640 10 monclient: init 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient(hunting): picked mon.noname-b con 0x7f22480a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient(hunting): picked mon.noname-c con 0x7f22480a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient(hunting): picked mon.noname-a con 0x7f22480a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2248135a70 con 0x7f22480a2980 2024-09-16T05:40:26.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2248138be0 con 0x7f22480a2190 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22480a7c50 con 0x7f22480a2570 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient(hunting): authenticate will time out at 1287.840118s 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f2247fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f22480a2980 0x7f22481d3ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33632/0 (socket says 172.21.15.6:33632) 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f2247fff640 1 -- 172.21.15.6:0/593891288 learned_addr learned my addr 172.21.15.6:0/593891288 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f2246ffd640 1 --1- 172.21.15.6:0/593891288 >> v1:172.21.15.6:6790/0 conn(0x7f22480a2570 0x7f2248135360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60974/0 (socket says 172.21.15.6:60974) 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740059117 0 0) 0x7f22480a7c50 con 0x7f22480a2570 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): my global_id is 6257 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2234003530 con 0x7f22480a2570 2024-09-16T05:40:26.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3648801905 0 0) 0x7f2248135a70 con 0x7f22480a2980 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): my global_id is 6285 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22480a7c50 con 0x7f22480a2980 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1460421450 0 0) 0x7f2234003530 con 0x7f22480a2570 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2248135a70 con 0x7f22480a2570 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f223c001970 con 0x7f22480a2570 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): dump: 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.063 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3754845762 0 0) 0x7f22480a7c50 con 0x7f22480a2980 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2234003530 con 0x7f22480a2980 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020335473 0 0) 0x7f2248138be0 con 0x7f22480a2190 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): my global_id is 114112 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22480a7c50 con 0x7f22480a2190 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2240001880 con 0x7f22480a2980 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): dump: 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.065 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4155362871 0 0) 0x7f2248135a70 con 0x7f22480a2570 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 >> v1:172.21.15.63:6789/0 conn(0x7f22480a2190 legacy=0x7f2248134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 >> v1:172.21.15.6:6789/0 conn(0x7f22480a2980 legacy=0x7f22481d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 monclient: found mon.c 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:26.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22480a81a0 con 0x7f22480a2570 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: _finish_auth 0 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 5 monclient: authenticate success, global_id 6257 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient: _renew_subs 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22480a0100 con 0x7f22480a2570 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient: _renew_subs 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2248138900 con 0x7f22480a2570 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f223c0047f0 con 0x7f22480a2570 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:26.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f223c004b40 con 0x7f22480a2570 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: dump: 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.068 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f2245ffb640 4 set_mon_vals no callback set 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22280065d0 con 0x7f2228002870 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f223c00f8e0 con 0x7f22480a2570 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(233..233 src has 20..233) ==== 10976+0+0 (unknown 3366826510 0 0) 0x7f223c045f10 con 0x7f22480a2570 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1541845718 0 0) 0x7f22280065d0 con 0x7f2228002870 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient(hunting): my global_id is 6260 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2248135a70 con 0x7f2228002870 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f2247fff640 10 monclient: get_auth_request con 0x7f2234036c10 auth_method 0 2024-09-16T05:40:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3714580107 0 0) 0x7f2248135a70 con 0x7f2228002870 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22280065d0 con 0x7f2228002870 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22380017a0 con 0x7f2228002870 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: dump: 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 432098352 0 0) 0x7f22280065d0 con 0x7f2228002870 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f222800a160 con 0x7f2228002870 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2238001b30 con 0x7f2228002870 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 >> v1:172.21.15.6:6790/0 conn(0x7f2228002870 legacy=0x7f2228004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.032+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 >> v1:172.21.15.6:6790/0 conn(0x7f2228002870 legacy=0x7f2228004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2228009e50 con 0x7f2228009110 2024-09-16T05:40:26.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2198836220 0 0) 0x7f2228009e50 con 0x7f2228009110 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient(hunting): my global_id is 6263 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22280065d0 con 0x7f2228009110 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3468091589 0 0) 0x7f22280065d0 con 0x7f2228009110 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2228009e50 con 0x7f2228009110 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22380037a0 con 0x7f2228009110 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient: dump: 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.072 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1675802762 0 0) 0x7f2228009e50 con 0x7f2228009110 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f222800c3e0 con 0x7f2228009110 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 170056366) 0x7f2238003820 con 0x7f2228009110 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 >> v1:172.21.15.6:6790/0 conn(0x7f2228009110 legacy=0x7f22280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f222ffff640 1 -- 172.21.15.6:0/593891288 >> v1:172.21.15.6:6790/0 conn(0x7f2228009110 legacy=0x7f22280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:26.074 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:26.074 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:26.074 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:26.074 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:40:26.074 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 26, 2024-09-16T05:40:26.074 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 355096, 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:26.075 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:26.076 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:40:26.077 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:40:26.078 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.079 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.080 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:26.081 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:26.082 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:26.083 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.084 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:40:26.085 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2234036c10 legacy=0x7f22340390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 10 monclient: shutdown 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 >> v1:172.21.15.6:6790/0 conn(0x7f22480a2570 legacy=0x7f2248135360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f2247fff640 1 -- 172.21.15.6:0/593891288 reap_dead start 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f2247fff640 1 -- 172.21.15.6:0/593891288 reap_dead start 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 shutdown_connections 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 >> 172.21.15.6:0/593891288 conn(0x7f22480a9850 msgr2=0x7f224809d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 shutdown_connections 2024-09-16T05:40:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.040+0000 7f224dd70640 1 -- 172.21.15.6:0/593891288 wait complete. 2024-09-16T05:40:26.088 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:40:26.088 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:40:26.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:40:26.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient: get_monmap_and_config 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient: build_initial_monmap 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient: monmap: 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:26.205293+0000 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:26.205293+0000 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:26.205 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 1 Processor -- start 2024-09-16T05:40:26.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 1 -- start start 2024-09-16T05:40:26.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient: init 2024-09-16T05:40:26.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:26.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc809aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc809e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc80a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.200+0000 7f1fcdc63640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc80a1370 con 0x7f1fc80a0da0 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc809ead0 con 0x7f1fc809e230 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc809edb0 con 0x7f1fc809aa30 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient(hunting): authenticate will time out at 1288.012056s 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc77fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1fc809aa30 0x7f1fc809ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60998/0 (socket says 172.21.15.6:60998) 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc77fe640 1 -- 172.21.15.6:0/241068889 learned_addr learned my addr 172.21.15.6:0/241068889 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421100715 0 0) 0x7f1fc809edb0 con 0x7f1fc809aa30 2024-09-16T05:40:26.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient(hunting): my global_id is 6266 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb4003530 con 0x7f1fc809aa30 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531449374 0 0) 0x7f1fc80a1370 con 0x7f1fc80a0da0 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient(hunting): my global_id is 6288 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc809edb0 con 0x7f1fc80a0da0 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2910530282 0 0) 0x7f1fb4003530 con 0x7f1fc809aa30 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 >> v1:172.21.15.63:6789/0 conn(0x7f1fc809e230 legacy=0x7f1fc81399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 >> v1:172.21.15.6:6789/0 conn(0x7f1fc80a0da0 legacy=0x7f1fc813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 monclient: found mon.noname-b 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc809dd50 con 0x7f1fc809aa30 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1fb8001620 con 0x7f1fc809aa30 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 5 monclient: authenticate success, global_id 6266 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: dump: 2024-09-16T05:40:26.214 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fb80018d0 con 0x7f1fc809aa30 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 1 -- 172.21.15.6:0/241068889 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1fb8001ce0 con 0x7f1fc809aa30 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc67fc640 10 monclient: dump: 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 4 set_mon_vals no callback set 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient: shutdown 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- 172.21.15.6:0/241068889 >> v1:172.21.15.6:6790/0 conn(0x7f1fc809aa30 legacy=0x7f1fc809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- 172.21.15.6:0/241068889 shutdown_connections 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- 172.21.15.6:0/241068889 >> 172.21.15.6:0/241068889 conn(0x7f1fc80a1650 msgr2=0x7f1fc80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- 172.21.15.6:0/241068889 shutdown_connections 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- 172.21.15.6:0/241068889 wait complete. 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient: build_initial_monmap 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient: monmap: 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 Processor -- start 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- start start 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient: init 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc809e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:26.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc80a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc809d810 con 0x7f1fc809aa30 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc80740c0 con 0x7f1fc809e230 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc813bbd0 con 0x7f1fc80a0da0 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fcdc63640 10 monclient(hunting): authenticate will time out at 1288.015379s 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1fc809aa30 0x7f1fc809bdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33650/0 (socket says 172.21.15.6:33650) 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fc77fe640 1 -- 172.21.15.6:0/3792492834 learned_addr learned my addr 172.21.15.6:0/3792492834 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345824456 0 0) 0x7f1fc809d810 con 0x7f1fc809aa30 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 10 monclient(hunting): my global_id is 6291 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fa8003530 con 0x7f1fc809aa30 2024-09-16T05:40:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 478120427 0 0) 0x7f1fc813bbd0 con 0x7f1fc80a0da0 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 10 monclient(hunting): my global_id is 6269 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc809d810 con 0x7f1fc80a0da0 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.204+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1125021196 0 0) 0x7f1fa8003530 con 0x7f1fc809aa30 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fc813bbd0 con 0x7f1fc809aa30 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247593531 0 0) 0x7f1fc80740c0 con 0x7f1fc809e230 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): my global_id is 114118 2024-09-16T05:40:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fa8003530 con 0x7f1fc809e230 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1fb8001c20 con 0x7f1fc809aa30 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): dump: 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.221 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1828470061 0 0) 0x7f1fc809d810 con 0x7f1fc80a0da0 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fc80740c0 con 0x7f1fc80a0da0 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1fc0001c00 con 0x7f1fc80a0da0 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): dump: 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3830960352 0 0) 0x7f1fc813bbd0 con 0x7f1fc809aa30 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 >> v1:172.21.15.6:6790/0 conn(0x7f1fc80a0da0 legacy=0x7f1fc81d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 >> v1:172.21.15.63:6789/0 conn(0x7f1fc809e230 legacy=0x7f1fc809c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 monclient: found mon.a 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc813a780 con 0x7f1fc809aa30 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: _finish_auth 0 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 5 monclient: authenticate success, global_id 6291 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 10 monclient: _renew_subs 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc8135160 con 0x7f1fc809aa30 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 10 monclient: _renew_subs 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1fc809cbf0 con 0x7f1fc809aa30 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fb8004cd0 con 0x7f1fc809aa30 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1fb80050a0 con 0x7f1fc809aa30 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fc5ffb640 4 set_mon_vals no callback set 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: dump: 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1fb8005240 con 0x7f1fc809aa30 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1fac0065d0 con 0x7f1fac002870 2024-09-16T05:40:26.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(233..233 src has 20..233) ==== 10976+0+0 (unknown 3366826510 0 0) 0x7f1fb80460e0 con 0x7f1fc809aa30 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fc7fff640 10 monclient: get_auth_request con 0x7f1fa8036c20 auth_method 0 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2286328273 0 0) 0x7f1fac0065d0 con 0x7f1fac002870 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient(hunting): my global_id is 114121 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc813bbd0 con 0x7f1fac002870 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3258903389 0 0) 0x7f1fc813bbd0 con 0x7f1fac002870 2024-09-16T05:40:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fac0065d0 con 0x7f1fac002870 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1fb0001710 con 0x7f1fac002870 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: dump: 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.232 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.232 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.232 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 666234713 0 0) 0x7f1fac0065d0 con 0x7f1fac002870 2024-09-16T05:40:26.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fac00a160 con 0x7f1fac002870 2024-09-16T05:40:26.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1fb00019f0 con 0x7f1fac002870 2024-09-16T05:40:26.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 >> v1:172.21.15.63:6789/0 conn(0x7f1fac002870 legacy=0x7f1fac004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.208+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 >> v1:172.21.15.63:6789/0 conn(0x7f1fac002870 legacy=0x7f1fac004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fcdc63640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1fac009e50 con 0x7f1fac009110 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887916594 0 0) 0x7f1fac009e50 con 0x7f1fac009110 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 10 monclient(hunting): my global_id is 114124 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fac0065d0 con 0x7f1fac009110 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4191766457 0 0) 0x7f1fac0065d0 con 0x7f1fac009110 2024-09-16T05:40:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fac009e50 con 0x7f1fac009110 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1fb00037c0 con 0x7f1fac009110 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 10 monclient: dump: 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.234 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3715373932 0 0) 0x7f1fac009e50 con 0x7f1fac009110 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.216+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1fac00c3e0 con 0x7f1fac009110 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3959 (unknown 0 0 394875702) 0x7f1fb00037c0 con 0x7f1fac009110 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fbffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fbffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 >> v1:172.21.15.63:6789/0 conn(0x7f1fac009110 legacy=0x7f1fac0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fbffff640 1 -- 172.21.15.6:0/3792492834 >> v1:172.21.15.63:6789/0 conn(0x7f1fac009110 legacy=0x7f1fac0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:26.235 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 26, 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 783, 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:40:26.236 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:40:26.237 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.238 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:40:26.239 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:26.240 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:40:26.241 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:26.242 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.243 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.244 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:26.245 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:40:26.246 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1fa8036c20 legacy=0x7f1fa80390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fcdc63640 10 monclient: shutdown 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fcdc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 >> v1:172.21.15.6:6789/0 conn(0x7f1fc809aa30 legacy=0x7f1fc809bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fc7fff640 1 -- 172.21.15.6:0/3792492834 reap_dead start 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fc7fff640 1 -- 172.21.15.6:0/3792492834 reap_dead start 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 shutdown_connections 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 >> 172.21.15.6:0/3792492834 conn(0x7f1fc80a1650 msgr2=0x7f1fc80a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 shutdown_connections 2024-09-16T05:40:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.220+0000 7f1fcdc63640 1 -- 172.21.15.6:0/3792492834 wait complete. 2024-09-16T05:40:26.249 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:40:26.249 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:40:26.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:40:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.356+0000 7fe124d2f640 10 monclient: get_monmap_and_config 2024-09-16T05:40:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.356+0000 7fe124d2f640 10 monclient: build_initial_monmap 2024-09-16T05:40:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.356+0000 7fe124d2f640 10 monclient: monmap: 2024-09-16T05:40:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:26.362725+0000 2024-09-16T05:40:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:26.362725+0000 2024-09-16T05:40:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:26.363 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 Processor -- start 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- start start 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient: init 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient(hunting): picked mon.noname-b con 0x7fe120072300 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient(hunting): picked mon.noname-c con 0x7fe120072df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient(hunting): picked mon.noname-a con 0x7fe12013d320 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12009b330 con 0x7fe12013d320 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12009b050 con 0x7fe120072df0 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12009b610 con 0x7fe120072300 2024-09-16T05:40:26.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient(hunting): authenticate will time out at 1288.169009s 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11e59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe120072300 0x7fe1200726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32788/0 (socket says 172.21.15.6:32788) 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11e59c640 1 -- 172.21.15.6:0/1724778345 learned_addr learned my addr 172.21.15.6:0/1724778345 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11ed9d640 1 --1- 172.21.15.6:0/1724778345 >> v1:172.21.15.6:6789/0 conn(0x7fe12013d320 0x7fe12013f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33660/0 (socket says 172.21.15.6:33660) 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2273192773 0 0) 0x7fe1180014a0 con 0x7fe12013d320 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient(hunting): my global_id is 6294 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe108003530 con 0x7fe12013d320 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999293837 0 0) 0x7fe12009b610 con 0x7fe120072300 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient(hunting): my global_id is 6272 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1180014a0 con 0x7fe120072300 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3708646882 0 0) 0x7fe108003530 con 0x7fe12013d320 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 >> v1:172.21.15.6:6790/0 conn(0x7fe120072300 legacy=0x7fe1200726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 >> v1:172.21.15.63:6789/0 conn(0x7fe120072df0 legacy=0x7fe12013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 monclient: found mon.noname-a 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12009c070 con 0x7fe12013d320 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: _finish_auth 0 2024-09-16T05:40:26.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 5 monclient: authenticate success, global_id 6294 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe118001b70 con 0x7fe12013d320 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: dump: 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.373 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe118001e50 con 0x7fe12013d320 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 1 -- 172.21.15.6:0/1724778345 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe118004b70 con 0x7fe12013d320 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 4 set_mon_vals no callback set 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe11d59a640 10 monclient: dump: 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient: shutdown 2024-09-16T05:40:26.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- 172.21.15.6:0/1724778345 >> v1:172.21.15.6:6789/0 conn(0x7fe12013d320 legacy=0x7fe12013f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- 172.21.15.6:0/1724778345 shutdown_connections 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- 172.21.15.6:0/1724778345 >> 172.21.15.6:0/1724778345 conn(0x7fe1200a1650 msgr2=0x7fe1200a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- 172.21.15.6:0/1724778345 shutdown_connections 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- 172.21.15.6:0/1724778345 wait complete. 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient: build_initial_monmap 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient: monmap: 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:26.376 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 Processor -- start 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 1 -- start start 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.360+0000 7fe124d2f640 10 monclient: init 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient(hunting): picked mon.noname-b con 0x7fe120072300 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient(hunting): picked mon.noname-c con 0x7fe120072df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient(hunting): picked mon.noname-a con 0x7fe12013d320 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:26.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12013a7e0 con 0x7fe12013d320 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1200740c0 con 0x7fe120072300 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe12013c2a0 con 0x7fe120072df0 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient(hunting): authenticate will time out at 1288.172185s 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe11dd9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe120072df0 0x7fe120136f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32792/0 (socket says 172.21.15.6:32792) 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe11dd9b640 1 -- 172.21.15.6:0/3387721037 learned_addr learned my addr 172.21.15.6:0/3387721037 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe11ed9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe12013d320 0x7fe1201d34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33664/0 (socket says 172.21.15.6:33664) 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1811660253 0 0) 0x7fe12013a7e0 con 0x7fe12013d320 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): my global_id is 6297 2024-09-16T05:40:26.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe100003530 con 0x7fe12013d320 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250859267 0 0) 0x7fe12013c2a0 con 0x7fe120072df0 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): my global_id is 6275 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe12013a7e0 con 0x7fe120072df0 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1684411270 0 0) 0x7fe12013a7e0 con 0x7fe120072df0 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe12013c2a0 con 0x7fe120072df0 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe1140019d0 con 0x7fe120072df0 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): dump: 2024-09-16T05:40:26.379 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1610717055 0 0) 0x7fe100003530 con 0x7fe12013d320 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe12013a7e0 con 0x7fe12013d320 2024-09-16T05:40:26.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe118001370 con 0x7fe12013d320 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): dump: 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.381 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560934237 0 0) 0x7fe1200740c0 con 0x7fe120072300 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient(hunting): my global_id is 114130 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe100003530 con 0x7fe120072300 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 679444344 0 0) 0x7fe12013c2a0 con 0x7fe120072df0 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 >> v1:172.21.15.63:6789/0 conn(0x7fe120072300 legacy=0x7fe1201cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 >> v1:172.21.15.6:6789/0 conn(0x7fe12013d320 legacy=0x7fe1201d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 monclient: found mon.c 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1201376a0 con 0x7fe120072df0 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient: _finish_auth 0 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 5 monclient: authenticate success, global_id 6275 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient: _renew_subs 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe12013b340 con 0x7fe120072df0 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient: _renew_subs 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1201d4da0 con 0x7fe120072df0 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe114001d40 con 0x7fe120072df0 2024-09-16T05:40:26.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe114003af0 con 0x7fe120072df0 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe11cd99640 4 set_mon_vals no callback set 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 10 monclient: dump: 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe114003c90 con 0x7fe120072df0 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe104005090 con 0x7fe120072df0 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(233..233 src has 20..233) ==== 10976+0+0 (unknown 3366826510 0 0) 0x7fe114046040 con 0x7fe120072df0 2024-09-16T05:40:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.364+0000 7fe11e59c640 10 monclient: get_auth_request con 0x7fe100036bd0 auth_method 0 2024-09-16T05:40:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.368+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe114015630 con 0x7fe120072df0 2024-09-16T05:40:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.368+0000 7fe10effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.368+0000 7fe10effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:26.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.561+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 removed item id -64 name 'h16' from crush map v234) ==== 120+0+0 (unknown 1899368347 0 0) 0x7f299c0155f0 con 0x7f29a80a1c40 2024-09-16T05:40:26.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.561+0000 7f298f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T05:40:26.566 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.561+0000 7f298f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -64 name 'h16' from crush map 2024-09-16T05:40:26.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.592+0000 7fe124d2f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:40:26.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.592+0000 7fe124d2f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:40:26.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.592+0000 7fe124d2f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:26.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.592+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe104005b90 con 0x7fe120072df0 2024-09-16T05:40:26.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.596+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.596+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=11,logm=57} crc {auth=3992608835,config=3541143443,health=1015318134,kv=2361287224,logm=336069585}) 2024-09-16T05:40:26.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.596+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=3611306962,config=3541143443,health=983635299,kv=3130718821,logm=1625169662}) 2024-09-16T05:40:26.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.596+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.596+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=11,logm=57} crc {auth=3992608835,config=3541143443,health=1015318134,kv=2361287224,logm=336069585}) 2024-09-16T05:40:26.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.596+0000 7fe10effd640 1 -- 172.21.15.6:0/3387721037 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fe114014f30 con 0x7fe120072df0 2024-09-16T05:40:26.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.596+0000 7fe10effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:40:26.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.596+0000 7fe10effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:26.602 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.596+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=11,logm=59} crc {auth=3640188992,config=3541143443,health=1015318134,kv=2361287224,logm=795795320}) 2024-09-16T05:40:26.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432752296}) 2024-09-16T05:40:26.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279169155}) 2024-09-16T05:40:26.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432752296}) 2024-09-16T05:40:26.604 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=150883101}) 2024-09-16T05:40:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.600+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe100036bd0 legacy=0x7fe100039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.600+0000 7fe124d2f640 10 monclient: shutdown 2024-09-16T05:40:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.600+0000 7fe124d2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.600+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 >> v1:172.21.15.6:6790/0 conn(0x7fe120072df0 legacy=0x7fe120136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.600+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 shutdown_connections 2024-09-16T05:40:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.600+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 >> 172.21.15.6:0/3387721037 conn(0x7fe1200a1650 msgr2=0x7fe1200a3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.600+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 shutdown_connections 2024-09-16T05:40:26.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:26.600+0000 7fe124d2f640 1 -- 172.21.15.6:0/3387721037 wait complete. 2024-09-16T05:40:26.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812039730}) 2024-09-16T05:40:26.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4141927239}) 2024-09-16T05:40:26.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812039730}) 2024-09-16T05:40:26.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.600+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955776676}) 2024-09-16T05:40:26.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2086691263}) 2024-09-16T05:40:26.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464864543}) 2024-09-16T05:40:26.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2086691263}) 2024-09-16T05:40:26.608 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416396883}) 2024-09-16T05:40:26.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252769867}) 2024-09-16T05:40:26.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=439364317}) 2024-09-16T05:40:26.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252769867}) 2024-09-16T05:40:26.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.604+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2906218677}) 2024-09-16T05:40:26.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634861852}) 2024-09-16T05:40:26.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4144684632}) 2024-09-16T05:40:26.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634861852}) 2024-09-16T05:40:26.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952048423}) 2024-09-16T05:40:26.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4097385365}) 2024-09-16T05:40:26.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1217310971}) 2024-09-16T05:40:26.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4097385365}) 2024-09-16T05:40:26.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.608+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=202896145}) 2024-09-16T05:40:26.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864865823}) 2024-09-16T05:40:26.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1363105583}) 2024-09-16T05:40:26.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864865823}) 2024-09-16T05:40:26.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2891857326}) 2024-09-16T05:40:26.616 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217131542}) 2024-09-16T05:40:26.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1107213693}) 2024-09-16T05:40:26.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217131542}) 2024-09-16T05:40:26.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.612+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046108330}) 2024-09-16T05:40:26.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395823680}) 2024-09-16T05:40:26.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2558505664}) 2024-09-16T05:40:26.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395823680}) 2024-09-16T05:40:26.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289978363}) 2024-09-16T05:40:26.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:40:26.620 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802679947}) 2024-09-16T05:40:26.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1288199062}) 2024-09-16T05:40:26.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802679947}) 2024-09-16T05:40:26.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.616+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1513863513}) 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114940953}) 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960245168}) 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114940953}) 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268296541}) 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487656307}) 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3948182526}) 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487656307}) 2024-09-16T05:40:26.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781970285}) 2024-09-16T05:40:26.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364614080}) 2024-09-16T05:40:26.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3931718559}) 2024-09-16T05:40:26.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364614080}) 2024-09-16T05:40:26.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.620+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3506052035}) 2024-09-16T05:40:26.629 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.629 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070711406}) 2024-09-16T05:40:26.629 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2301350436}) 2024-09-16T05:40:26.629 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.629 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070711406}) 2024-09-16T05:40:26.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2881143218}) 2024-09-16T05:40:26.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2507141694}) 2024-09-16T05:40:26.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3825373351}) 2024-09-16T05:40:26.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2507141694}) 2024-09-16T05:40:26.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.624+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4162436409}) 2024-09-16T05:40:26.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117038854}) 2024-09-16T05:40:26.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651331273}) 2024-09-16T05:40:26.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117038854}) 2024-09-16T05:40:26.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367780813}) 2024-09-16T05:40:26.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.633 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220636657}) 2024-09-16T05:40:26.633 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382036707}) 2024-09-16T05:40:26.633 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.633 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220636657}) 2024-09-16T05:40:26.633 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.628+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3514921671}) 2024-09-16T05:40:26.635 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.635 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=794560763}) 2024-09-16T05:40:26.635 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1588542150}) 2024-09-16T05:40:26.635 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.635 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=794560763}) 2024-09-16T05:40:26.635 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=794502906}) 2024-09-16T05:40:26.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93645979}) 2024-09-16T05:40:26.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3675809646}) 2024-09-16T05:40:26.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93645979}) 2024-09-16T05:40:26.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657610235}) 2024-09-16T05:40:26.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230612290}) 2024-09-16T05:40:26.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366589008}) 2024-09-16T05:40:26.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230612290}) 2024-09-16T05:40:26.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.632+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1370849114}) 2024-09-16T05:40:26.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858844467}) 2024-09-16T05:40:26.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=143807145}) 2024-09-16T05:40:26.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858844467}) 2024-09-16T05:40:26.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3531275616}) 2024-09-16T05:40:26.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506437736}) 2024-09-16T05:40:26.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=477909396}) 2024-09-16T05:40:26.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506437736}) 2024-09-16T05:40:26.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.636+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706814490}) 2024-09-16T05:40:26.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511133277}) 2024-09-16T05:40:26.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3005314789}) 2024-09-16T05:40:26.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511133277}) 2024-09-16T05:40:26.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2320686121}) 2024-09-16T05:40:26.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250853713}) 2024-09-16T05:40:26.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950301948}) 2024-09-16T05:40:26.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250853713}) 2024-09-16T05:40:26.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=84158997}) 2024-09-16T05:40:26.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187073135}) 2024-09-16T05:40:26.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3536404366}) 2024-09-16T05:40:26.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187073135}) 2024-09-16T05:40:26.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.640+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1153962224}) 2024-09-16T05:40:26.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261176760}) 2024-09-16T05:40:26.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2372046406}) 2024-09-16T05:40:26.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261176760}) 2024-09-16T05:40:26.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=821081067}) 2024-09-16T05:40:26.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642104894}) 2024-09-16T05:40:26.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1421326836}) 2024-09-16T05:40:26.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642104894}) 2024-09-16T05:40:26.650 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.644+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503153457}) 2024-09-16T05:40:26.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485713910}) 2024-09-16T05:40:26.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3388353187}) 2024-09-16T05:40:26.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485713910}) 2024-09-16T05:40:26.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003124781}) 2024-09-16T05:40:26.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.653 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187830798}) 2024-09-16T05:40:26.653 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2114498634}) 2024-09-16T05:40:26.653 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.653 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187830798}) 2024-09-16T05:40:26.653 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1004258399}) 2024-09-16T05:40:26.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.648+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2970088800,mdsmap=3035854586}) 2024-09-16T05:40:26.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=3946441562,mdsmap=1832746017}) 2024-09-16T05:40:26.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2970088800,mdsmap=3035854586}) 2024-09-16T05:40:26.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1212217882,mdsmap=1832746017,mgr=3189685207}) 2024-09-16T05:40:26.657 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.657 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mdsmap=3140514009,mgr=848307873,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2756164545,mon_config_key=3178366981,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1576632878,osd_snap=723727393,osdmap=1672206339}) 2024-09-16T05:40:26.657 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mdsmap=3140514009,mgr=1552851386,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2749514200,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1576632878,osd_snap=723727393,osdmap=2032308257}) 2024-09-16T05:40:26.657 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.657 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mdsmap=3140514009,mgr=848307873,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2756164545,mon_config_key=3178366981,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1576632878,osd_snap=723727393,osdmap=1672206339}) 2024-09-16T05:40:26.657 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.652+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mdsmap=3140514009,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2749514200,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1576632878,osd_snap=723727393,osdmap=2990742644}) 2024-09-16T05:40:26.659 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.656+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.659 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.656+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2416920616}) 2024-09-16T05:40:26.659 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.656+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1717433947}) 2024-09-16T05:40:26.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.656+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.656+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2416920616}) 2024-09-16T05:40:26.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.656+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2206715834}) 2024-09-16T05:40:26.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.660+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.660+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=136135273}) 2024-09-16T05:40:26.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.660+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2635599652}) 2024-09-16T05:40:26.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.660+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.660+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=136135273}) 2024-09-16T05:40:26.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.660+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1614719654}) 2024-09-16T05:40:26.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.665+0000 7f29ad592640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T05:40:26.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.665+0000 7f29ad592640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T05:40:26.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.665+0000 7f29ad592640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:26.671 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.665+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) -- 0x7f29a809dc20 con 0x7f29a80a1c40 2024-09-16T05:40:26.674 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.672+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.675 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.672+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=162819873}) 2024-09-16T05:40:26.675 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.672+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=3426555607}) 2024-09-16T05:40:26.675 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.672+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:40:26.675 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.672+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=162819873}) 2024-09-16T05:40:26.675 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:40:26.672+0000 7f3419a35640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=62} crc {osdmap=2706241563}) 2024-09-16T05:40:26.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.669+0000 7f298f7fe640 1 -- 172.21.15.63:0/2249911186 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 device 'h16' does not appear in the crush map v234) ==== 120+0+0 (unknown 1002539358 0 0) 0x7f299c01db00 con 0x7f29a80a1c40 2024-09-16T05:40:26.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.669+0000 7f298f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T05:40:26.676 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.669+0000 7f298f7fe640 10 monclient: _finish_command 3 = system:0 device 'h16' does not appear in the crush map 2024-09-16T05:40:26.676 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h16' does not appear in the crush map 2024-09-16T05:40:26.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.673+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2994036de0 legacy=0x7f29940392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.673+0000 7f29ad592640 10 monclient: shutdown 2024-09-16T05:40:26.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.673+0000 7f29ad592640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:26.678 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.673+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 >> v1:172.21.15.63:6789/0 conn(0x7f29a80a1c40 legacy=0x7f29a8071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.673+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 shutdown_connections 2024-09-16T05:40:26.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.673+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 >> 172.21.15.63:0/2249911186 conn(0x7f29a80a3e90 msgr2=0x7f29a80a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:26.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.673+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 shutdown_connections 2024-09-16T05:40:26.679 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.673+0000 7f29ad592640 1 -- 172.21.15.63:0/2249911186 wait complete. 2024-09-16T05:40:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:26.693 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h17 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient: get_monmap_and_config 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient: build_initial_monmap 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient: monmap: 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:26.810547+0000 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:26.810547+0000 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 1 Processor -- start 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 1 -- start start 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient: init 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient(hunting): picked mon.noname-b con 0x7f8570136ac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient(hunting): picked mon.noname-a con 0x7f857009fd80 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient(hunting): picked mon.noname-c con 0x7f857013d0b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8570072cb0 con 0x7f857009fd80 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8570136670 con 0x7f857013d0b0 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85700a14f0 con 0x7f8570136ac0 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:26.821 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8577fa5640 10 monclient(hunting): authenticate will time out at 1283.587494s 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8576542640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f857013d0b0 0x7f857013f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54136/0 (socket says 172.21.15.63:54136) 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.805+0000 7f8576542640 1 -- 172.21.15.63:0/1000944993 learned_addr learned my addr 172.21.15.63:0/1000944993 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2438722362 0 0) 0x7f8570136670 con 0x7f857013d0b0 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): my global_id is 114133 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f855c003530 con 0x7f857013d0b0 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2447909232 0 0) 0x7f85700a14f0 con 0x7f8570136ac0 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.822 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): my global_id is 6278 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8570136670 con 0x7f8570136ac0 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176732091 0 0) 0x7f8560001120 con 0x7f857009fd80 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient(hunting): my global_id is 6300 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85700a14f0 con 0x7f857009fd80 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 317820316 0 0) 0x7f855c003530 con 0x7f857013d0b0 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 >> v1:172.21.15.6:6790/0 conn(0x7f8570136ac0 legacy=0x7f8570136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 >> v1:172.21.15.6:6789/0 conn(0x7f857009fd80 legacy=0x7f857013b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: _finish_hunting 0 2024-09-16T05:40:26.823 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 monclient: found mon.noname-c 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85700a0630 con 0x7f857013d0b0 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: _finish_auth 0 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: _check_auth_tickets 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8568001620 con 0x7f857013d0b0 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: dump: 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.824 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 5 monclient: authenticate success, global_id 114133 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8568001940 con 0x7f857013d0b0 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 1 -- 172.21.15.63:0/1000944993 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8568001d10 con 0x7f857013d0b0 2024-09-16T05:40:26.825 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8574d3f640 10 monclient: dump: 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 4 set_mon_vals no callback set 2024-09-16T05:40:26.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: shutdown 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- 172.21.15.63:0/1000944993 >> v1:172.21.15.63:6789/0 conn(0x7f857013d0b0 legacy=0x7f857013f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- 172.21.15.63:0/1000944993 shutdown_connections 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- 172.21.15.63:0/1000944993 >> 172.21.15.63:0/1000944993 conn(0x7f85700a1c90 msgr2=0x7f85700a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- 172.21.15.63:0/1000944993 shutdown_connections 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- 172.21.15.63:0/1000944993 wait complete. 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: build_initial_monmap 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: monmap: 2024-09-16T05:40:26.827 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 Processor -- start 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- start start 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: init 2024-09-16T05:40:26.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient(hunting): picked mon.noname-b con 0x7f857009fd80 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient(hunting): picked mon.noname-c con 0x7f8570136ac0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient(hunting): picked mon.noname-a con 0x7f857013d0b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f857011eee0 con 0x7f857013d0b0 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f857011f4a0 con 0x7f857009fd80 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f857009c0f0 con 0x7f8570136ac0 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient(hunting): authenticate will time out at 1283.590847s 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8575d41640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f857009fd80 0x7f8570133b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54148/0 (socket says 172.21.15.63:54148) 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8575d41640 1 -- 172.21.15.63:0/3009995439 learned_addr learned my addr 172.21.15.63:0/3009995439 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3442746812 0 0) 0x7f857011f4a0 con 0x7f857009fd80 2024-09-16T05:40:26.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): my global_id is 114136 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8554003560 con 0x7f857009fd80 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714258947 0 0) 0x7f857009c0f0 con 0x7f8570136ac0 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): my global_id is 6281 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f857011f4a0 con 0x7f8570136ac0 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3772825261 0 0) 0x7f8554003560 con 0x7f857009fd80 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f857009c0f0 con 0x7f857009fd80 2024-09-16T05:40:26.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f856c001c40 con 0x7f857009fd80 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): dump: 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.831 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120762381 0 0) 0x7f857011eee0 con 0x7f857013d0b0 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient(hunting): my global_id is 6303 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8554003560 con 0x7f857013d0b0 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2758355597 0 0) 0x7f857009c0f0 con 0x7f857009fd80 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 >> v1:172.21.15.6:6790/0 conn(0x7f8570136ac0 legacy=0x7f85701d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 >> v1:172.21.15.6:6789/0 conn(0x7f857013d0b0 legacy=0x7f85701d4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:26.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 monclient: found mon.b 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857009d430 con 0x7f857009fd80 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f85667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 5 monclient: authenticate success, global_id 114136 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: _renew_subs 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f857011dd10 con 0x7f857009fd80 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: _renew_subs 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.809+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f857011f1c0 con 0x7f857009fd80 2024-09-16T05:40:26.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f856c003a20 con 0x7f857009fd80 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f85667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f856c003df0 con 0x7f857009fd80 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f85667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f85667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f8567fff640 4 set_mon_vals no callback set 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f85667fc640 10 monclient: dump: 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:26.834 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f856c003f90 con 0x7f857009fd80 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f8577fa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f8577fa5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f8577fa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8558005090 con 0x7f857009fd80 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(234..234 src has 20..234) ==== 10763+0+0 (unknown 1952770268 0 0) 0x7f856c0460c0 con 0x7f857009fd80 2024-09-16T05:40:26.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.813+0000 7f8575540640 10 monclient: get_auth_request con 0x7f8554036ce0 auth_method 0 2024-09-16T05:40:26.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.817+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f856c0157a0 con 0x7f857009fd80 2024-09-16T05:40:26.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.817+0000 7f85667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:26.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.817+0000 7f85667fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:26.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.985+0000 7f8577fa5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T05:40:26.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.985+0000 7f8577fa5640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T05:40:26.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.985+0000 7f8577fa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:26.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:26.985+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) -- 0x7f8558005380 con 0x7f857009fd80 2024-09-16T05:40:27.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.577+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 removed item id -67 name 'h17' from crush map v235) ==== 120+0+0 (unknown 4122111969 0 0) 0x7f856c0150a0 con 0x7f857009fd80 2024-09-16T05:40:27.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.577+0000 7f85667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T05:40:27.582 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.577+0000 7f85667fc640 10 monclient: _finish_command 2 = system:0 removed item id -67 name 'h17' from crush map 2024-09-16T05:40:27.621 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:40:27.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:40:27.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T05:40:27.688 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T05:40:27.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:27.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) -- 0x7f8558002220 con 0x7f857009fd80 2024-09-16T05:40:27.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f85667fc640 1 -- 172.21.15.63:0/3009995439 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 device 'h17' does not appear in the crush map v235) ==== 120+0+0 (unknown 2415501809 0 0) 0x7f856c01dc50 con 0x7f857009fd80 2024-09-16T05:40:27.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f85667fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T05:40:27.690 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f85667fc640 10 monclient: _finish_command 3 = system:0 device 'h17' does not appear in the crush map 2024-09-16T05:40:27.690 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h17' does not appear in the crush map 2024-09-16T05:40:27.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8554036ce0 legacy=0x7f85540391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 10 monclient: shutdown 2024-09-16T05:40:27.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:27.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 >> v1:172.21.15.63:6789/0 conn(0x7f857009fd80 legacy=0x7f8570133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 shutdown_connections 2024-09-16T05:40:27.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 >> 172.21.15.63:0/3009995439 conn(0x7f85700a1c90 msgr2=0x7f85700a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:27.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.685+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 shutdown_connections 2024-09-16T05:40:27.692 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.689+0000 7f8577fa5640 1 -- 172.21.15.63:0/3009995439 wait complete. 2024-09-16T05:40:27.707 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:27.707 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h18 2024-09-16T05:40:27.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient: get_monmap_and_config 2024-09-16T05:40:27.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient: build_initial_monmap 2024-09-16T05:40:27.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient: monmap: 2024-09-16T05:40:27.752 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:27.752943+0000 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:27.752943+0000 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:27.753 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 1 Processor -- start 2024-09-16T05:40:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 1 -- start start 2024-09-16T05:40:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient: init 2024-09-16T05:40:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient(hunting): picked mon.noname-b con 0x7feab80a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient(hunting): picked mon.noname-c con 0x7feab80a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:27.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 10 monclient(hunting): picked mon.noname-a con 0x7feab80a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab80a33a0 con 0x7feab80a2980 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab80a07a0 con 0x7feab80a2570 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.748+0000 7feabcdb0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab80a2d90 con 0x7feab80a2190 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7feab80a2190 0x7feab80a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32794/0 (socket says 172.21.15.6:32794) 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab659c640 1 -- 172.21.15.6:0/2039654460 learned_addr learned my addr 172.21.15.6:0/2039654460 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient(hunting): authenticate will time out at 1289.560057s 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3249847616 0 0) 0x7feab80a2d90 con 0x7feab80a2190 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient(hunting): my global_id is 6284 2024-09-16T05:40:27.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaa4003530 con 0x7feab80a2190 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4153374953 0 0) 0x7feab80a33a0 con 0x7feab80a2980 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient(hunting): my global_id is 6306 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feab80a2d90 con 0x7feab80a2980 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 352257746 0 0) 0x7feab80a2d90 con 0x7feab80a2980 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 >> v1:172.21.15.6:6790/0 conn(0x7feab80a2190 legacy=0x7feab80a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 >> v1:172.21.15.63:6789/0 conn(0x7feab80a2570 legacy=0x7feab813b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 monclient: found mon.noname-a 2024-09-16T05:40:27.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab80a66e0 con 0x7feab80a2980 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: _finish_auth 0 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 5 monclient: authenticate success, global_id 6306 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feab0001600 con 0x7feab80a2980 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: dump: 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:27.759 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feab0001960 con 0x7feab80a2980 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:27.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 1 -- 172.21.15.6:0/2039654460 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feab0003620 con 0x7feab80a2980 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 4 set_mon_vals no callback set 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab559a640 10 monclient: dump: 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.761 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient: shutdown 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- 172.21.15.6:0/2039654460 >> v1:172.21.15.6:6789/0 conn(0x7feab80a2980 legacy=0x7feab813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- 172.21.15.6:0/2039654460 shutdown_connections 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- 172.21.15.6:0/2039654460 >> 172.21.15.6:0/2039654460 conn(0x7feab80a9850 msgr2=0x7feab80a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- 172.21.15.6:0/2039654460 shutdown_connections 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- 172.21.15.6:0/2039654460 wait complete. 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient: build_initial_monmap 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient: monmap: 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:27.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:27.763 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:27.763 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:27.763 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:27.763 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:27.763 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:27.763 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:27.763 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:27.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 Processor -- start 2024-09-16T05:40:27.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- start start 2024-09-16T05:40:27.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient: init 2024-09-16T05:40:27.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:27.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient(hunting): picked mon.noname-a con 0x7feab80a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient(hunting): picked mon.noname-c con 0x7feab80a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient(hunting): picked mon.noname-b con 0x7feab80a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab80a7c50 con 0x7feab80a2190 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab8138580 con 0x7feab80a2980 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab8135a20 con 0x7feab80a2570 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feabcdb0640 10 monclient(hunting): authenticate will time out at 1289.563081s 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab5d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7feab80a2570 0x7feab8135310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32804/0 (socket says 172.21.15.6:32804) 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feab5d9b640 1 -- 172.21.15.6:0/173423661 learned_addr learned my addr 172.21.15.6:0/173423661 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485758163 0 0) 0x7feaa8001030 con 0x7feab80a2190 2024-09-16T05:40:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): my global_id is 6309 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea98003530 con 0x7feab80a2190 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310445751 0 0) 0x7feab8135a20 con 0x7feab80a2570 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): my global_id is 6287 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaa8001030 con 0x7feab80a2570 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4257034542 0 0) 0x7feab8138580 con 0x7feab80a2980 2024-09-16T05:40:27.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): my global_id is 114142 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feab8135a20 con 0x7feab80a2980 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1773156207 0 0) 0x7fea98003530 con 0x7feab80a2190 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feab8138580 con 0x7feab80a2190 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1761424783 0 0) 0x7feaa8001030 con 0x7feab80a2570 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea98003530 con 0x7feab80a2570 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feaa8001a40 con 0x7feab80a2190 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:27.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): dump: 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feaa0001ba0 con 0x7feab80a2570 2024-09-16T05:40:27.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.752+0000 7feaaeffd640 10 monclient(hunting): dump: 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 345781735 0 0) 0x7fea98003530 con 0x7feab80a2570 2024-09-16T05:40:27.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 >> v1:172.21.15.63:6789/0 conn(0x7feab80a2980 legacy=0x7feab81d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 >> v1:172.21.15.6:6789/0 conn(0x7feab80a2190 legacy=0x7feab8134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 monclient: found mon.c 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab8138b40 con 0x7feab80a2570 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 5 monclient: authenticate success, global_id 6287 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 10 monclient: _renew_subs 2024-09-16T05:40:27.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feab81382a0 con 0x7feab80a2570 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 10 monclient: _renew_subs 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feab8137130 con 0x7feab80a2570 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feaa00039a0 con 0x7feab80a2570 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feaa0003b50 con 0x7feab80a2570 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feab4d99640 4 set_mon_vals no callback set 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 10 monclient: dump: 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7feaa0003cf0 con 0x7feab80a2570 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:27.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(235..235 src has 20..235) ==== 10550+0+0 (unknown 2061532035 0 0) 0x7feaa0045f60 con 0x7feab80a2570 2024-09-16T05:40:27.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9c005090 con 0x7feab80a2570 2024-09-16T05:40:27.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.756+0000 7feab659c640 10 monclient: get_auth_request con 0x7fea98036c00 auth_method 0 2024-09-16T05:40:27.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.760+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7feaa00156a0 con 0x7feab80a2570 2024-09-16T05:40:27.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.760+0000 7feaaeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:27.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.760+0000 7feaaeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.821+0000 7f7c3a8c4640 10 monclient: get_monmap_and_config 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.821+0000 7f7c3a8c4640 10 monclient: build_initial_monmap 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.821+0000 7f7c3a8c4640 10 monclient: monmap: 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:27.827263+0000 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:27.827263+0000 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:27.827 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:27.828 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:27.828 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:27.828 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:27.828 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:27.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.821+0000 7f7c3a8c4640 1 Processor -- start 2024-09-16T05:40:27.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.821+0000 7f7c3a8c4640 1 -- start start 2024-09-16T05:40:27.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.821+0000 7f7c3a8c4640 10 monclient: init 2024-09-16T05:40:27.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:27.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:27.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:27.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3406dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:27.835 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient(hunting): picked mon.noname-a con 0x7f7c340a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c3409f800 con 0x7f7c340a0210 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c340a0ab0 con 0x7f7c3406dd30 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c340a0d90 con 0x7f7c3409ac90 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient(hunting): authenticate will time out at 1284.604711s 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c337fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7c3406dd30 0x7f7c341399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54150/0 (socket says 172.21.15.63:54150) 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c337fe640 1 -- 172.21.15.63:0/1680075845 learned_addr learned my addr 172.21.15.63:0/1680075845 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3194703009 0 0) 0x7f7c340a0ab0 con 0x7f7c3406dd30 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.836 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): my global_id is 114145 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c1c003530 con 0x7f7c3406dd30 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 706096312 0 0) 0x7f7c3409f800 con 0x7f7c340a0210 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): my global_id is 6312 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c340a0ab0 con 0x7f7c340a0210 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515563186 0 0) 0x7f7c340a0d90 con 0x7f7c3409ac90 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient(hunting): my global_id is 6290 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c3409f800 con 0x7f7c3409ac90 2024-09-16T05:40:27.837 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3899989825 0 0) 0x7f7c1c003530 con 0x7f7c3406dd30 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 >> v1:172.21.15.6:6790/0 conn(0x7f7c3409ac90 legacy=0x7f7c3409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 >> v1:172.21.15.6:6789/0 conn(0x7f7c340a0210 legacy=0x7f7c3413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 monclient: found mon.noname-c 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c3409e770 con 0x7f7c3406dd30 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c28001620 con 0x7f7c3406dd30 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:27.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: dump: 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 5 monclient: authenticate success, global_id 114145 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:27.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c28001940 con 0x7f7c3406dd30 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 4 set_mon_vals no callback set 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 1 -- 172.21.15.63:0/1680075845 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c28001d10 con 0x7f7c3406dd30 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c32ffd640 10 monclient: dump: 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:27.840 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: shutdown 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/1680075845 >> v1:172.21.15.63:6789/0 conn(0x7f7c3406dd30 legacy=0x7f7c341399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/1680075845 shutdown_connections 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/1680075845 >> 172.21.15.63:0/1680075845 conn(0x7f7c340a1c90 msgr2=0x7f7c340a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/1680075845 shutdown_connections 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/1680075845 wait complete. 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: build_initial_monmap 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: monmap: 2024-09-16T05:40:27.841 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 Processor -- start 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 1 -- start start 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: init 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:27.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.825+0000 7f7c3a8c4640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3406dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3409ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient(hunting): picked mon.noname-a con 0x7f7c340a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c3409df90 con 0x7f7c340a0210 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c34124480 con 0x7f7c3409ac90 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c341241a0 con 0x7f7c3406dd30 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient(hunting): authenticate will time out at 1284.608057s 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c337fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7c3409ac90 0x7f7c341d0470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54158/0 (socket says 172.21.15.63:54158) 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c337fe640 1 -- 172.21.15.63:0/752582467 learned_addr learned my addr 172.21.15.63:0/752582467 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166009571 0 0) 0x7f7c34124480 con 0x7f7c3409ac90 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): my global_id is 114148 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c20003530 con 0x7f7c3409ac90 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1990979395 0 0) 0x7f7c20003530 con 0x7f7c3409ac90 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c34124480 con 0x7f7c3409ac90 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c280015e0 con 0x7f7c3409ac90 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): dump: 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.844 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168129511 0 0) 0x7f7c3409df90 con 0x7f7c340a0210 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): my global_id is 6315 2024-09-16T05:40:27.845 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c20003530 con 0x7f7c340a0210 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3432536688 0 0) 0x7f7c341241a0 con 0x7f7c3406dd30 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient(hunting): my global_id is 6293 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c3409df90 con 0x7f7c3406dd30 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 31968986 0 0) 0x7f7c34124480 con 0x7f7c3409ac90 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 >> v1:172.21.15.6:6790/0 conn(0x7f7c3406dd30 legacy=0x7f7c34134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 >> v1:172.21.15.6:6789/0 conn(0x7f7c340a0210 legacy=0x7f7c341d3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 monclient: found mon.b 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:27.846 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c34135550 con 0x7f7c3409ac90 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient: _finish_auth 0 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 5 monclient: authenticate success, global_id 114148 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient: _renew_subs 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c341351b0 con 0x7f7c3409ac90 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient: _renew_subs 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c341cb3b0 con 0x7f7c3409ac90 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c28004cb0 con 0x7f7c3409ac90 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c28005000 con 0x7f7c3409ac90 2024-09-16T05:40:27.847 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c327fc640 4 set_mon_vals no callback set 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 10 monclient: dump: 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:27.848 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:27.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7c280051a0 con 0x7f7c3409ac90 2024-09-16T05:40:27.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:27.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:27.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:27.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c14005090 con 0x7f7c3409ac90 2024-09-16T05:40:27.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(235..235 src has 20..235) ==== 10550+0+0 (unknown 2061532035 0 0) 0x7f7c28045ee0 con 0x7f7c3409ac90 2024-09-16T05:40:27.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.829+0000 7f7c33fff640 10 monclient: get_auth_request con 0x7f7c20036d00 auth_method 0 2024-09-16T05:40:27.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.833+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7c2801dbb0 con 0x7f7c3409ac90 2024-09-16T05:40:27.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.833+0000 7f7c30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:27.850 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:27.833+0000 7f7c30ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.992+0000 7feabcdb0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:40:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.992+0000 7feabcdb0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.992+0000 7feabcdb0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.992+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea9c005380 con 0x7feab80a2570 2024-09-16T05:40:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.992+0000 7feaaeffd640 1 -- 172.21.15.6:0/173423661 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3455487405) 0x7feaa0014fa0 con 0x7feab80a2570 2024-09-16T05:40:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.992+0000 7feaaeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.992+0000 7feaaeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:27.997 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:40:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.996+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 >> v1:172.21.15.63:6812/3643207211 conn(0x7fea98036c00 legacy=0x7fea980390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.996+0000 7feabcdb0640 10 monclient: shutdown 2024-09-16T05:40:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.996+0000 7feabcdb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.996+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 >> v1:172.21.15.6:6790/0 conn(0x7feab80a2570 legacy=0x7feab8135310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.996+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 shutdown_connections 2024-09-16T05:40:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.996+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 >> 172.21.15.6:0/173423661 conn(0x7feab80a9850 msgr2=0x7feab809d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.996+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 shutdown_connections 2024-09-16T05:40:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:27.996+0000 7feabcdb0640 1 -- 172.21.15.6:0/173423661 wait complete. 2024-09-16T05:40:28.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.001+0000 7f7c3a8c4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T05:40:28.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.001+0000 7f7c3a8c4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T05:40:28.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.001+0000 7f7c3a8c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:28.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.001+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) -- 0x7f7c14005380 con 0x7f7c3409ac90 2024-09-16T05:40:28.016 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:40:28.016 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:40:28.016 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:40:28.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:40:28.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient: get_monmap_and_config 2024-09-16T05:40:28.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient: monmap: 2024-09-16T05:40:28.140 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.140 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.141 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:28.140150+0000 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:28.140150+0000 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 1 Processor -- start 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 1 -- start start 2024-09-16T05:40:28.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient: init 2024-09-16T05:40:28.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient(hunting): picked mon.noname-c con 0x7fdad80a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient(hunting): picked mon.noname-a con 0x7fdad80a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient(hunting): picked mon.noname-b con 0x7fdad80a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad80a33a0 con 0x7fdad80a2570 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad80a07a0 con 0x7fdad80a2190 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad80a2d90 con 0x7fdad80a2980 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient(hunting): authenticate will time out at 1289.946786s 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad5d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdad80a2570 0x7fdad813b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33674/0 (socket says 172.21.15.6:33674) 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad6d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdad80a2980 0x7fdad813f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32820/0 (socket says 172.21.15.6:32820) 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad5d9b640 1 -- 172.21.15.6:0/1286262907 learned_addr learned my addr 172.21.15.6:0/1286262907 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 773087507 0 0) 0x7fdad80a2d90 con 0x7fdad80a2980 2024-09-16T05:40:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient(hunting): my global_id is 6296 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac0003530 con 0x7fdad80a2980 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719983022 0 0) 0x7fdad80a33a0 con 0x7fdad80a2570 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient(hunting): my global_id is 6318 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdad80a2d90 con 0x7fdad80a2570 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3993043834 0 0) 0x7fdac0003530 con 0x7fdad80a2980 2024-09-16T05:40:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 >> v1:172.21.15.63:6789/0 conn(0x7fdad80a2190 legacy=0x7fdad80a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 >> v1:172.21.15.6:6789/0 conn(0x7fdad80a2570 legacy=0x7fdad813b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 monclient: found mon.noname-b 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad80a66e0 con 0x7fdad80a2980 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdad00018d0 con 0x7fdad80a2980 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 5 monclient: authenticate success, global_id 6296 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: dump: 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdad0001b30 con 0x7fdad80a2980 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdadcf81640 4 set_mon_vals no callback set 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 1 -- 172.21.15.6:0/1286262907 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdad0001f40 con 0x7fdad80a2980 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.136+0000 7fdad559a640 10 monclient: dump: 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: shutdown 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- 172.21.15.6:0/1286262907 >> v1:172.21.15.6:6790/0 conn(0x7fdad80a2980 legacy=0x7fdad813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- 172.21.15.6:0/1286262907 shutdown_connections 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- 172.21.15.6:0/1286262907 >> 172.21.15.6:0/1286262907 conn(0x7fdad80a9850 msgr2=0x7fdad80a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- 172.21.15.6:0/1286262907 shutdown_connections 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- 172.21.15.6:0/1286262907 wait complete. 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: monmap: 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 Processor -- start 2024-09-16T05:40:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- start start 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: init 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient(hunting): picked mon.noname-a con 0x7fdad80a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient(hunting): picked mon.noname-b con 0x7fdad80a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient(hunting): picked mon.noname-c con 0x7fdad80a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad812d640 con 0x7fdad80a2190 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad81241d0 con 0x7fdad80a2570 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad81224c0 con 0x7fdad80a2980 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient(hunting): authenticate will time out at 1289.949967s 2024-09-16T05:40:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdad6d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdad80a2980 0x7fdad81217d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32834/0 (socket says 172.21.15.6:32834) 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdad6d9d640 1 -- 172.21.15.6:0/2620925550 learned_addr learned my addr 172.21.15.6:0/2620925550 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141582377 0 0) 0x7fdad81224c0 con 0x7fdad80a2980 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): my global_id is 6299 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdab8003530 con 0x7fdad80a2980 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3774784904 0 0) 0x7fdad80a07a0 con 0x7fdad80a2190 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): my global_id is 6321 2024-09-16T05:40:28.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdad81224c0 con 0x7fdad80a2190 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1346213048 0 0) 0x7fdab8003530 con 0x7fdad80a2980 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdad80a07a0 con 0x7fdad80a2980 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdad0001f40 con 0x7fdad80a2980 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 929244343 0 0) 0x7fdad81224c0 con 0x7fdad80a2190 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdab8003530 con 0x7fdad80a2190 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdac8001880 con 0x7fdad80a2190 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4025277948 0 0) 0x7fdad81241d0 con 0x7fdad80a2570 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient(hunting): my global_id is 114154 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdad81224c0 con 0x7fdad80a2570 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2622378264 0 0) 0x7fdad80a07a0 con 0x7fdad80a2980 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 >> v1:172.21.15.63:6789/0 conn(0x7fdad80a2570 legacy=0x7fdad81210c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 >> v1:172.21.15.6:6789/0 conn(0x7fdad80a2190 legacy=0x7fdad81209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 monclient: found mon.c 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad8124790 con 0x7fdad80a2980 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 5 monclient: authenticate success, global_id 6299 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: _renew_subs 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdad8122bf0 con 0x7fdad80a2980 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: _renew_subs 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdad81d4ac0 con 0x7fdad80a2980 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdad0001b30 con 0x7fdad80a2980 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdad0005080 con 0x7fdad80a2980 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 10 monclient: dump: 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdad4d99640 4 set_mon_vals no callback set 2024-09-16T05:40:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdabc0065d0 con 0x7fdabc002870 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.140+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdad0005220 con 0x7fdad80a2980 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(235..235 src has 20..235) ==== 10550+0+0 (unknown 2061532035 0 0) 0x7fdad0046080 con 0x7fdad80a2980 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3841498313 0 0) 0x7fdabc0065d0 con 0x7fdabc002870 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient(hunting): my global_id is 6324 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdad80a07a0 con 0x7fdabc002870 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdad5d9b640 10 monclient: get_auth_request con 0x7fdab8036ca0 auth_method 0 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3070404543 0 0) 0x7fdad80a07a0 con 0x7fdabc002870 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdabc0065d0 con 0x7fdabc002870 2024-09-16T05:40:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdac80037c0 con 0x7fdabc002870 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient: dump: 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.165 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.166 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3524193063 0 0) 0x7fdabc0065d0 con 0x7fdabc002870 2024-09-16T05:40:28.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdabc00a160 con 0x7fdabc002870 2024-09-16T05:40:28.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdac8005a40 con 0x7fdabc002870 2024-09-16T05:40:28.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:28.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 >> v1:172.21.15.6:6789/0 conn(0x7fdabc002870 legacy=0x7fdabc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.144+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 >> v1:172.21.15.6:6789/0 conn(0x7fdabc002870 legacy=0x7fdabc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdadcf81640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdabc009e50 con 0x7fdabc009110 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133037615 0 0) 0x7fdabc009e50 con 0x7fdabc009110 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 10 monclient(hunting): my global_id is 6327 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdabc0065d0 con 0x7fdabc009110 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2621933401 0 0) 0x7fdabc0065d0 con 0x7fdabc009110 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdabc009e50 con 0x7fdabc009110 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdac8005090 con 0x7fdabc009110 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 10 monclient: dump: 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1864301813 0 0) 0x7fdabc009e50 con 0x7fdabc009110 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.148+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdabc00c3e0 con 0x7fdabc009110 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 467383356) 0x7fdac8004eb0 con 0x7fdabc009110 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdac6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdac6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 >> v1:172.21.15.6:6789/0 conn(0x7fdabc009110 legacy=0x7fdabc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdac6ffd640 1 -- 172.21.15.6:0/2620925550 >> v1:172.21.15.6:6789/0 conn(0x7fdabc009110 legacy=0x7fdabc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:28.170 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:28.171 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:28.171 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:28.171 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:40:28.171 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 26, 2024-09-16T05:40:28.171 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 357204, 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:40:28.172 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.173 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:40:28.175 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.176 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:28.177 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.178 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.180 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:40:28.180 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:40:28.180 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.180 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.180 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.180 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.181 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.182 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:40:28.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:28.184 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.185 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.185 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:40:28.185 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 >> v1:172.21.15.63:6812/3643207211 conn(0x7fdab8036ca0 legacy=0x7fdab8039160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdadcf81640 10 monclient: shutdown 2024-09-16T05:40:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdadcf81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 >> v1:172.21.15.6:6790/0 conn(0x7fdad80a2980 legacy=0x7fdad81217d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdad6d9d640 1 -- 172.21.15.6:0/2620925550 reap_dead start 2024-09-16T05:40:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdad6d9d640 1 -- 172.21.15.6:0/2620925550 reap_dead start 2024-09-16T05:40:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 shutdown_connections 2024-09-16T05:40:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 >> 172.21.15.6:0/2620925550 conn(0x7fdad80a9850 msgr2=0x7fdad806ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 shutdown_connections 2024-09-16T05:40:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.152+0000 7fdadcf81640 1 -- 172.21.15.6:0/2620925550 wait complete. 2024-09-16T05:40:28.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:40:28.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:40:28.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:40:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient: get_monmap_and_config 2024-09-16T05:40:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient: monmap: 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:28.304100+0000 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:28.304100+0000 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:28.304 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 1 Processor -- start 2024-09-16T05:40:28.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 1 -- start start 2024-09-16T05:40:28.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient: init 2024-09-16T05:40:28.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f92dc12dcf0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f92dc0a1680 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f92dc09ba40 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92dc09f170 con 0x7f92dc09ba40 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92dc09f6c0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92dc09f3e0 con 0x7f92dc0a1680 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92e0aa5640 10 monclient(hunting): authenticate will time out at 1290.111729s 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92d1d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f92dc0a1680 0x7f92dc139710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32848/0 (socket says 172.21.15.6:32848) 2024-09-16T05:40:28.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.300+0000 7f92d1d9b640 1 -- 172.21.15.6:0/1429097511 learned_addr learned my addr 172.21.15.6:0/1429097511 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1693925828 0 0) 0x7f92dc09f170 con 0x7f92dc09ba40 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): my global_id is 6330 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92c8003530 con 0x7f92dc09ba40 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432045248 0 0) 0x7f92dc09f3e0 con 0x7f92dc0a1680 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): my global_id is 6302 2024-09-16T05:40:28.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92dc09f170 con 0x7f92dc0a1680 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 667629803 0 0) 0x7f92dc09f6c0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient(hunting): my global_id is 114157 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92dc09f3e0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2009263451 0 0) 0x7f92c8003530 con 0x7f92dc09ba40 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 >> v1:172.21.15.6:6790/0 conn(0x7f92dc0a1680 legacy=0x7f92dc139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 >> v1:172.21.15.63:6789/0 conn(0x7f92dc12dcf0 legacy=0x7f92dc12e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 monclient: found mon.noname-a 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc12e7e0 con 0x7f92dc09ba40 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92d40019a0 con 0x7f92dc09ba40 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: dump: 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.311 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 5 monclient: authenticate success, global_id 6330 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92d4001cc0 con 0x7f92dc09ba40 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 1 -- 172.21.15.6:0/1429097511 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92d4002090 con 0x7f92dc09ba40 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d9d9b640 10 monclient: dump: 2024-09-16T05:40:28.312 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 4 set_mon_vals no callback set 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient: shutdown 2024-09-16T05:40:28.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- 172.21.15.6:0/1429097511 >> v1:172.21.15.6:6789/0 conn(0x7f92dc09ba40 legacy=0x7f92dc13ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- 172.21.15.6:0/1429097511 shutdown_connections 2024-09-16T05:40:28.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- 172.21.15.6:0/1429097511 >> 172.21.15.6:0/1429097511 conn(0x7f92dc0a1f00 msgr2=0x7f92dc0a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- 172.21.15.6:0/1429097511 shutdown_connections 2024-09-16T05:40:28.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- 172.21.15.6:0/1429097511 wait complete. 2024-09-16T05:40:28.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient: monmap: 2024-09-16T05:40:28.318 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.318 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:28.318 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:28.318 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 Processor -- start 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- start start 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient: init 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f92dc09ba40 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f92dc12dcf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f92dc0738c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92dc0a4f90 con 0x7f92dc09ba40 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92dc09c420 con 0x7f92dc0738c0 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92dc137970 con 0x7f92dc12dcf0 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92e0aa5640 10 monclient(hunting): authenticate will time out at 1290.115201s 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92da59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f92dc09ba40 0x7f92dc1c98f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33716/0 (socket says 172.21.15.6:33716) 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d1d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f92dc12dcf0 0x7f92dc1ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32852/0 (socket says 172.21.15.6:32852) 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92da59c640 1 -- 172.21.15.6:0/3583721861 learned_addr learned my addr 172.21.15.6:0/3583721861 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276316040 0 0) 0x7f92dc137970 con 0x7f92dc12dcf0 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): my global_id is 6305 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92bc003530 con 0x7f92dc12dcf0 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3243672492 0 0) 0x7f92dc09f3e0 con 0x7f92dc09ba40 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): my global_id is 6333 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92dc137970 con 0x7f92dc09ba40 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 383065608 0 0) 0x7f92bc003530 con 0x7f92dc12dcf0 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92dc09f3e0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92c40018c0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): dump: 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 53452799 0 0) 0x7f92dc137970 con 0x7f92dc09ba40 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92bc003530 con 0x7f92dc09ba40 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92cc0018a0 con 0x7f92dc09ba40 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): dump: 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1008395106 0 0) 0x7f92dc12e7e0 con 0x7f92dc0738c0 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.304+0000 7f92d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient(hunting): my global_id is 114160 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92dc137970 con 0x7f92dc0738c0 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3072841034 0 0) 0x7f92dc09f3e0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 >> v1:172.21.15.63:6789/0 conn(0x7f92dc0738c0 legacy=0x7f92dc1d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 >> v1:172.21.15.6:6789/0 conn(0x7f92dc09ba40 legacy=0x7f92dc1c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 monclient: found mon.c 2024-09-16T05:40:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc137080 con 0x7f92dc12dcf0 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 5 monclient: authenticate success, global_id 6305 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 10 monclient: _renew_subs 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92dc137690 con 0x7f92dc12dcf0 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 10 monclient: _renew_subs 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f92dc1ca1d0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92c40047f0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92c4004bc0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d959a640 4 set_mon_vals no callback set 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: dump: 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.326 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92c00065d0 con 0x7f92c0002870 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f92c4004de0 con 0x7f92dc12dcf0 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(235..235 src has 20..235) ==== 10550+0+0 (unknown 2061532035 0 0) 0x7f92c4045c00 con 0x7f92dc12dcf0 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2188354766 0 0) 0x7f92c00065d0 con 0x7f92c0002870 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient(hunting): my global_id is 6308 2024-09-16T05:40:28.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92dc09f3e0 con 0x7f92c0002870 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1714786822 0 0) 0x7f92dc09f3e0 con 0x7f92c0002870 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92c00065d0 con 0x7f92c0002870 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92cc0039f0 con 0x7f92c0002870 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92dad9d640 10 monclient: get_auth_request con 0x7f92bc036b90 auth_method 0 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: dump: 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.328 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2869295070 0 0) 0x7f92c00065d0 con 0x7f92c0002870 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92c000a160 con 0x7f92c0002870 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f92cc003e30 con 0x7f92c0002870 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 >> v1:172.21.15.6:6790/0 conn(0x7f92c0002870 legacy=0x7f92c0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.308+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 >> v1:172.21.15.6:6790/0 conn(0x7f92c0002870 legacy=0x7f92c0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:40:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92c0009e30 con 0x7f92c00090f0 2024-09-16T05:40:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176765318 0 0) 0x7f92c0009e30 con 0x7f92c00090f0 2024-09-16T05:40:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient(hunting): my global_id is 6311 2024-09-16T05:40:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92c00065d0 con 0x7f92c00090f0 2024-09-16T05:40:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 943133136 0 0) 0x7f92c00065d0 con 0x7f92c00090f0 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92c0009e30 con 0x7f92c00090f0 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f92cc0043f0 con 0x7f92c00090f0 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient: dump: 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4088339038 0 0) 0x7f92c0009e30 con 0x7f92c00090f0 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f92c000c3a0 con 0x7f92c00090f0 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 1191058877) 0x7f92cc0057d0 con 0x7f92c00090f0 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 >> v1:172.21.15.6:6790/0 conn(0x7f92c00090f0 legacy=0x7f92c0009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92d37fe640 1 -- 172.21.15.6:0/3583721861 >> v1:172.21.15.6:6790/0 conn(0x7f92c00090f0 legacy=0x7f92c0009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 26, 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 357372, 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:40:28.334 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.335 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.336 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:40:28.337 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.338 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:40:28.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.340 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.341 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:40:28.343 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:28.344 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.345 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-09-16T05:40:28.346 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.346 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.346 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:40:28.346 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 >> v1:172.21.15.63:6812/3643207211 conn(0x7f92bc036b90 legacy=0x7f92bc039050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 10 monclient: shutdown 2024-09-16T05:40:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 >> v1:172.21.15.6:6790/0 conn(0x7f92dc12dcf0 legacy=0x7f92dc1ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92dad9d640 1 -- 172.21.15.6:0/3583721861 reap_dead start 2024-09-16T05:40:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92dad9d640 1 -- 172.21.15.6:0/3583721861 reap_dead start 2024-09-16T05:40:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 shutdown_connections 2024-09-16T05:40:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 >> 172.21.15.6:0/3583721861 conn(0x7f92dc0a1f00 msgr2=0x7f92dc130db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 shutdown_connections 2024-09-16T05:40:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.316+0000 7f92e0aa5640 1 -- 172.21.15.6:0/3583721861 wait complete. 2024-09-16T05:40:28.348 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:40:28.348 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:40:28.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:40:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.456+0000 7f9217ec0640 10 monclient: get_monmap_and_config 2024-09-16T05:40:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.456+0000 7f9217ec0640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.456+0000 7f9217ec0640 10 monclient: monmap: 2024-09-16T05:40:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:28.462852+0000 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:28.462852+0000 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:28.463 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 Processor -- start 2024-09-16T05:40:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- start start 2024-09-16T05:40:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient: init 2024-09-16T05:40:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f921009b2f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f9210136b70 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f921009b6d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92100a19c0 con 0x7f921009b6d0 2024-09-16T05:40:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9210099ae0 con 0x7f921009b2f0 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9210099dc0 con 0x7f9210136b70 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient(hunting): authenticate will time out at 1290.269410s 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f921645d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f921009b6d0 0x7f921013f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33726/0 (socket says 172.21.15.6:33726) 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f921645d640 1 -- 172.21.15.6:0/3308503029 learned_addr learned my addr 172.21.15.6:0/3308503029 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4255717967 0 0) 0x7f92100a19c0 con 0x7f921009b6d0 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient(hunting): my global_id is 6336 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91fc003530 con 0x7f921009b6d0 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 392920182 0 0) 0x7f9210099dc0 con 0x7f9210136b70 2024-09-16T05:40:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient(hunting): my global_id is 6314 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92100a19c0 con 0x7f9210136b70 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4079793673 0 0) 0x7f91fc003530 con 0x7f921009b6d0 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 >> v1:172.21.15.6:6790/0 conn(0x7f9210136b70 legacy=0x7f92100a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 >> v1:172.21.15.63:6789/0 conn(0x7f921009b2f0 legacy=0x7f92100a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 monclient: found mon.noname-a 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921009bae0 con 0x7f921009b6d0 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9208001620 con 0x7f921009b6d0 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 5 monclient: authenticate success, global_id 6336 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:28.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: dump: 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9208001900 con 0x7f921009b6d0 2024-09-16T05:40:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 1 -- 172.21.15.6:0/3308503029 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9208001cd0 con 0x7f921009b6d0 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 4 set_mon_vals no callback set 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9214c5a640 10 monclient: dump: 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient: shutdown 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- 172.21.15.6:0/3308503029 >> v1:172.21.15.6:6789/0 conn(0x7f921009b6d0 legacy=0x7f921013f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- 172.21.15.6:0/3308503029 shutdown_connections 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- 172.21.15.6:0/3308503029 >> 172.21.15.6:0/3308503029 conn(0x7f92100a1c90 msgr2=0x7f92100a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- 172.21.15.6:0/3308503029 shutdown_connections 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- 172.21.15.6:0/3308503029 wait complete. 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient: monmap: 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 Processor -- start 2024-09-16T05:40:28.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 1 -- start start 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.460+0000 7f9217ec0640 10 monclient: init 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f921009b2f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f921009b6d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f9210136b70 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92101319d0 con 0x7f921009b6d0 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9210131cb0 con 0x7f921009b2f0 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92100a4260 con 0x7f9210136b70 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient(hunting): authenticate will time out at 1290.272204s 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f921645d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9210136b70 0x7f92101d3ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32880/0 (socket says 172.21.15.6:32880) 2024-09-16T05:40:28.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f921545b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f921009b6d0 0x7f92101d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33734/0 (socket says 172.21.15.6:33734) 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f921645d640 1 -- 172.21.15.6:0/2551998864 learned_addr learned my addr 172.21.15.6:0/2551998864 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 268634048 0 0) 0x7f92100a4260 con 0x7f9210136b70 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): my global_id is 6317 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91f4003530 con 0x7f9210136b70 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3899715816 0 0) 0x7f92101319d0 con 0x7f921009b6d0 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): my global_id is 6339 2024-09-16T05:40:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92100a4260 con 0x7f921009b6d0 2024-09-16T05:40:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4175153310 0 0) 0x7f92100a4260 con 0x7f921009b6d0 2024-09-16T05:40:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92101319d0 con 0x7f921009b6d0 2024-09-16T05:40:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9200001950 con 0x7f921009b6d0 2024-09-16T05:40:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:40:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): dump: 2024-09-16T05:40:28.482 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.482 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.482 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3345933026 0 0) 0x7f91f4003530 con 0x7f9210136b70 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92100a4260 con 0x7f9210136b70 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1279480235 0 0) 0x7f9210131cb0 con 0x7f921009b2f0 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): my global_id is 114166 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91f4003530 con 0x7f921009b2f0 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9208001ab0 con 0x7f9210136b70 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): dump: 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1691961144 0 0) 0x7f92101319d0 con 0x7f921009b6d0 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 >> v1:172.21.15.6:6790/0 conn(0x7f9210136b70 legacy=0x7f92101d3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 >> v1:172.21.15.63:6789/0 conn(0x7f921009b2f0 legacy=0x7f921012e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 monclient: found mon.a 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92100a3dc0 con 0x7f921009b6d0 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 5 monclient: authenticate success, global_id 6339 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient: _renew_subs 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92101316f0 con 0x7f921009b6d0 2024-09-16T05:40:28.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient: _renew_subs 2024-09-16T05:40:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:40:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f921012e970 con 0x7f921009b6d0 2024-09-16T05:40:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9200003620 con 0x7f921009b6d0 2024-09-16T05:40:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9200003970 con 0x7f921009b6d0 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: dump: 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9207fff640 4 set_mon_vals no callback set 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f91f80065d0 con 0x7f91f8002870 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9200003b90 con 0x7f921009b6d0 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(235..235 src has 20..235) ==== 10550+0+0 (unknown 2061532035 0 0) 0x7f9200045c80 con 0x7f921009b6d0 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f921645d640 10 monclient: get_auth_request con 0x7f91f4036bb0 auth_method 0 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 531834880 0 0) 0x7f91f80065d0 con 0x7f91f8002870 2024-09-16T05:40:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient(hunting): my global_id is 114169 2024-09-16T05:40:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92101319d0 con 0x7f91f8002870 2024-09-16T05:40:28.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1800271025 0 0) 0x7f92101319d0 con 0x7f91f8002870 2024-09-16T05:40:28.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91f80065d0 con 0x7f91f8002870 2024-09-16T05:40:28.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f920c001ae0 con 0x7f91f8002870 2024-09-16T05:40:28.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:28.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 10 monclient: dump: 2024-09-16T05:40:28.492 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.492 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1037027897 0 0) 0x7f91f80065d0 con 0x7f91f8002870 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.464+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91f800a160 con 0x7f91f8002870 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.468+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f920c001e90 con 0x7f91f8002870 2024-09-16T05:40:28.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.468+0000 7f92067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.468+0000 7f92067fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.468+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 >> v1:172.21.15.63:6789/0 conn(0x7f91f8002870 legacy=0x7f91f8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.468+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 >> v1:172.21.15.63:6789/0 conn(0x7f91f8002870 legacy=0x7f91f8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f9217ec0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:40:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f91f8009e50 con 0x7f91f8009110 2024-09-16T05:40:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 406414313 0 0) 0x7f91f8009e50 con 0x7f91f8009110 2024-09-16T05:40:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f92067fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f92067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f92067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f92067fc640 10 monclient(hunting): my global_id is 114172 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91f80065d0 con 0x7f91f8009110 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.472+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2774338573 0 0) 0x7f91f80065d0 con 0x7f91f8009110 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91f8009e50 con 0x7f91f8009110 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f920c003b40 con 0x7f91f8009110 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 10 monclient: dump: 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2785366011 0 0) 0x7f91f8009e50 con 0x7f91f8009110 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f91f800c3e0 con 0x7f91f8009110 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3960 (unknown 0 0 3580985217) 0x7f920c003a70 con 0x7f91f8009110 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:40:28.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 >> v1:172.21.15.63:6789/0 conn(0x7f91f8009110 legacy=0x7f91f80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f92067fc640 1 -- 172.21.15.6:0/2551998864 >> v1:172.21.15.63:6789/0 conn(0x7f91f8009110 legacy=0x7f91f80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 26, 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 3039, 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:40:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:40:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:40:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:40:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:40:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:40:28.510 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 >> v1:172.21.15.63:6812/3643207211 conn(0x7f91f4036bb0 legacy=0x7f91f4039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f9217ec0640 10 monclient: shutdown 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f9217ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 >> v1:172.21.15.6:6789/0 conn(0x7f921009b6d0 legacy=0x7f92101d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f921645d640 1 -- 172.21.15.6:0/2551998864 reap_dead start 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f921645d640 1 -- 172.21.15.6:0/2551998864 reap_dead start 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 shutdown_connections 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 >> 172.21.15.6:0/2551998864 conn(0x7f92100a1c90 msgr2=0x7f9210135010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 shutdown_connections 2024-09-16T05:40:28.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.476+0000 7f9217ec0640 1 -- 172.21.15.6:0/2551998864 wait complete. 2024-09-16T05:40:28.512 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:40:28.512 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T05:40:28.512 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:40:28.512 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T05:40:28.512 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-16T05:40:28.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-16T05:40:28.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.573+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 removed item id -70 name 'h18' from crush map v236) ==== 120+0+0 (unknown 674477975 0 0) 0x7f7c28042190 con 0x7f7c3409ac90 2024-09-16T05:40:28.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.573+0000 7f7c30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T05:40:28.578 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.573+0000 7f7c30ff9640 10 monclient: _finish_command 2 = system:0 removed item id -70 name 'h18' from crush map 2024-09-16T05:40:28.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.624+0000 7f12b0326640 10 monclient: get_monmap_and_config 2024-09-16T05:40:28.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.624+0000 7f12b0326640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.624+0000 7f12b0326640 10 monclient: monmap: 2024-09-16T05:40:28.629 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.629 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.629 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:28.630002+0000 2024-09-16T05:40:28.629 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:28.630002+0000 2024-09-16T05:40:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:28.630 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:28.630 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.624+0000 7f12b0326640 1 Processor -- start 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.624+0000 7f12b0326640 1 -- start start 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.624+0000 7f12b0326640 10 monclient: init 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): picked mon.noname-a con 0x7f12a809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): picked mon.noname-b con 0x7f12a809b520 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): picked mon.noname-c con 0x7f12a8141690 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a80a1350 con 0x7f12a809aa30 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a809cdf0 con 0x7f12a8141690 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a809dad0 con 0x7f12a809b520 2024-09-16T05:40:28.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): authenticate will time out at 1290.436662s 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad8c1640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f12a809b520 0x7f12a813ff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32886/0 (socket says 172.21.15.6:32886) 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad8c1640 1 -- 172.21.15.6:0/1211415943 learned_addr learned my addr 172.21.15.6:0/1211415943 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ae0c2640 1 --1- 172.21.15.6:0/1211415943 >> v1:172.21.15.6:6789/0 conn(0x7f12a809aa30 0x7f12a809ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33744/0 (socket says 172.21.15.6:33744) 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1559090295 0 0) 0x7f12a809dad0 con 0x7f12a809b520 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): my global_id is 6320 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1294003530 con 0x7f12a809b520 2024-09-16T05:40:28.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530589726 0 0) 0x7f12a4001120 con 0x7f12a809aa30 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): my global_id is 6342 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a809dad0 con 0x7f12a809aa30 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614032280 0 0) 0x7f12a809cdf0 con 0x7f12a8141690 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient(hunting): my global_id is 114175 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a4001120 con 0x7f12a8141690 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2376097886 0 0) 0x7f1294003530 con 0x7f12a809b520 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 >> v1:172.21.15.63:6789/0 conn(0x7f12a8141690 legacy=0x7f12a8143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 >> v1:172.21.15.6:6789/0 conn(0x7f12a809aa30 legacy=0x7f12a809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 monclient: found mon.noname-b 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a80a10d0 con 0x7f12a809b520 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1298001620 con 0x7f12a809b520 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: dump: 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.646 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 5 monclient: authenticate success, global_id 6320 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1298001880 con 0x7f12a809b520 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 1 -- 172.21.15.6:0/1211415943 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1298001c10 con 0x7f12a809b520 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 4 set_mon_vals no callback set 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad0c0640 10 monclient: dump: 2024-09-16T05:40:28.647 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: shutdown 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- 172.21.15.6:0/1211415943 >> v1:172.21.15.6:6790/0 conn(0x7f12a809b520 legacy=0x7f12a813ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- 172.21.15.6:0/1211415943 shutdown_connections 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- 172.21.15.6:0/1211415943 >> 172.21.15.6:0/1211415943 conn(0x7f12a80a1650 msgr2=0x7f12a80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- 172.21.15.6:0/1211415943 shutdown_connections 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- 172.21.15.6:0/1211415943 wait complete. 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: monmap: 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.649 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 Processor -- start 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- start start 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: init 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): picked mon.noname-b con 0x7f12a809aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): picked mon.noname-c con 0x7f12a809b520 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): picked mon.noname-a con 0x7f12a8141690 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a8124a30 con 0x7f12a8141690 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a8141030 con 0x7f12a809aa30 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a81262f0 con 0x7f12a809b520 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12b0326640 10 monclient(hunting): authenticate will time out at 1290.439798s 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad8c1640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f12a809b520 0x7f12a81240c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32900/0 (socket says 172.21.15.6:32900) 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ad8c1640 1 -- 172.21.15.6:0/3674057077 learned_addr learned my addr 172.21.15.6:0/3674057077 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.628+0000 7f12ae8c3640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12a8141690 0x7f12a81d7c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33760/0 (socket says 172.21.15.6:33760) 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330196200 0 0) 0x7f12a81262f0 con 0x7f12a809b520 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): my global_id is 6323 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f128c003530 con 0x7f12a809b520 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401425611 0 0) 0x7f12a8124a30 con 0x7f12a8141690 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): my global_id is 6345 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a81262f0 con 0x7f12a8141690 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3175424486 0 0) 0x7f128c003530 con 0x7f12a809b520 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12a8124a30 con 0x7f12a809b520 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3768184433 0 0) 0x7f12a81262f0 con 0x7f12a8141690 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f128c003530 con 0x7f12a8141690 2024-09-16T05:40:28.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1298001520 con 0x7f12a809b520 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): dump: 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.653 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12a0001c40 con 0x7f12a8141690 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): dump: 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.654 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 350091272 0 0) 0x7f12a40013b0 con 0x7f12a809aa30 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): my global_id is 114178 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a81262f0 con 0x7f12a809aa30 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1440840279 0 0) 0x7f12a8124a30 con 0x7f12a809b520 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 >> v1:172.21.15.63:6789/0 conn(0x7f12a809aa30 legacy=0x7f12a81239b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 >> v1:172.21.15.6:6789/0 conn(0x7f12a8141690 legacy=0x7f12a81d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 monclient: found mon.c 2024-09-16T05:40:28.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a81265d0 con 0x7f12a809b520 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 5 monclient: authenticate success, global_id 6323 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 10 monclient: _renew_subs 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12a80740c0 con 0x7f12a809b520 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 10 monclient: _renew_subs 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f12a8124e40 con 0x7f12a809b520 2024-09-16T05:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1298001860 con 0x7f12a809b520 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1298004da0 con 0x7f12a809b520 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12ac8bf640 4 set_mon_vals no callback set 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: dump: 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f12900065d0 con 0x7f1290002870 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f129800fb40 con 0x7f12a809b520 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(236..236 src has 20..236) ==== 10337+0+0 (unknown 1376479996 0 0) 0x7f1298045be0 con 0x7f12a809b520 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333436331 0 0) 0x7f12900065d0 con 0x7f1290002870 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient(hunting): my global_id is 6348 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a8124a30 con 0x7f1290002870 2024-09-16T05:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f12ae8c3640 10 monclient: get_auth_request con 0x7f128c036c20 auth_method 0 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 213044355 0 0) 0x7f12a8124a30 con 0x7f1290002870 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12900065d0 con 0x7f1290002870 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12a4001bb0 con 0x7f1290002870 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: dump: 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 600047097 0 0) 0x7f12900065d0 con 0x7f1290002870 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f129000a160 con 0x7f1290002870 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f12a4006870 con 0x7f1290002870 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 >> v1:172.21.15.6:6789/0 conn(0x7f1290002870 legacy=0x7f1290004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.632+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 >> v1:172.21.15.6:6789/0 conn(0x7f1290002870 legacy=0x7f1290004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1290009e60 con 0x7f1290009120 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2873411015 0 0) 0x7f1290009e60 con 0x7f1290009120 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient(hunting): my global_id is 6351 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12900065d0 con 0x7f1290009120 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 110708565 0 0) 0x7f12900065d0 con 0x7f1290009120 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1290009e60 con 0x7f1290009120 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f12a4004bc0 con 0x7f1290009120 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient: dump: 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1235277390 0 0) 0x7f1290009e60 con 0x7f1290009120 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f129000c3e0 con 0x7f1290009120 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7f12a4005f50 con 0x7f1290009120 2024-09-16T05:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T05:40:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 >> v1:172.21.15.6:6789/0 conn(0x7f1290009120 legacy=0x7f12900095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f129effd640 1 -- 172.21.15.6:0/3674057077 >> v1:172.21.15.6:6789/0 conn(0x7f1290009120 legacy=0x7f12900095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:28.664 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:28.664 INFO:teuthology.orchestra.run.smithi006.stdout: "ret": 0, 2024-09-16T05:40:28.664 INFO:teuthology.orchestra.run.smithi006.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T05:40:28.664 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 >> v1:172.21.15.63:6812/3643207211 conn(0x7f128c036c20 legacy=0x7f128c0390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 10 monclient: shutdown 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 >> v1:172.21.15.6:6790/0 conn(0x7f12a809b520 legacy=0x7f12a81240c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12ae8c3640 1 -- 172.21.15.6:0/3674057077 reap_dead start 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12ae8c3640 1 -- 172.21.15.6:0/3674057077 reap_dead start 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 shutdown_connections 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 >> 172.21.15.6:0/3674057077 conn(0x7f12a80a1650 msgr2=0x7f12a80a2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 shutdown_connections 2024-09-16T05:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.640+0000 7f12b0326640 1 -- 172.21.15.6:0/3674057077 wait complete. 2024-09-16T05:40:28.666 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:40:28.667 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T05:40:28.667 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:40:28.680 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T05:40:28.680 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:40:28.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:40:28.686 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.681+0000 7f7c3a8c4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T05:40:28.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.681+0000 7f7c3a8c4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T05:40:28.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.681+0000 7f7c3a8c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:28.687 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:28.681+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) -- 0x7f7c14002220 con 0x7f7c3409ac90 2024-09-16T05:40:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient: get_monmap_and_config 2024-09-16T05:40:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient: monmap: 2024-09-16T05:40:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:28.836642+0000 2024-09-16T05:40:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:28.836642+0000 2024-09-16T05:40:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.837 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.837 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.837 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:28.837 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:28.837 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 1 Processor -- start 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 1 -- start start 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient: init 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient(hunting): picked mon.noname-c con 0x7f282409bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient(hunting): picked mon.noname-a con 0x7f282409bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient(hunting): picked mon.noname-b con 0x7f28241397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28240a1350 con 0x7f282409bf20 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28240a10d0 con 0x7f282409bb40 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f282409d940 con 0x7f28241397f0 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f2828d11640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f282409bf20 legacy=0x7f282409c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f282b776640 10 monclient(hunting): authenticate will time out at 1290.643390s 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f2829d13640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f28241397f0 0x7f282413bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32922/0 (socket says 172.21.15.6:32922) 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f2829d13640 1 -- 172.21.15.6:0/2082414344 learned_addr learned my addr 172.21.15.6:0/2082414344 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 418652370 0 0) 0x7f282409d940 con 0x7f28241397f0 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f281bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f281bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f281bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.832+0000 7f281bfff640 10 monclient(hunting): my global_id is 6326 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2810003530 con 0x7f28241397f0 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1267317809 0 0) 0x7f28240a10d0 con 0x7f282409bb40 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient(hunting): my global_id is 114181 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f282409d940 con 0x7f282409bb40 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 760382599 0 0) 0x7f2810003530 con 0x7f28241397f0 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 >> v1:172.21.15.63:6789/0 conn(0x7f282409bb40 legacy=0x7f282409e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 >> v1:172.21.15.6:6789/0 conn(0x7f282409bf20 legacy=0x7f282409c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 monclient: found mon.noname-b 2024-09-16T05:40:28.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282409dc20 con 0x7f28241397f0 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f281c0019e0 con 0x7f28241397f0 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 5 monclient: authenticate success, global_id 6326 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: dump: 2024-09-16T05:40:28.846 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f281c001c40 con 0x7f28241397f0 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 1 -- 172.21.15.6:0/2082414344 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f281c002010 con 0x7f28241397f0 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 4 set_mon_vals no callback set 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281bfff640 10 monclient: dump: 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: shutdown 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- 172.21.15.6:0/2082414344 >> v1:172.21.15.6:6790/0 conn(0x7f28241397f0 legacy=0x7f282413bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- 172.21.15.6:0/2082414344 shutdown_connections 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- 172.21.15.6:0/2082414344 >> 172.21.15.6:0/2082414344 conn(0x7f28240a1650 msgr2=0x7f28240a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- 172.21.15.6:0/2082414344 shutdown_connections 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- 172.21.15.6:0/2082414344 wait complete. 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: build_initial_monmap 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: monmap: 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 Processor -- start 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- start start 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: init 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient(hunting): picked mon.noname-b con 0x7f282409bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient(hunting): picked mon.noname-c con 0x7f282409bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient(hunting): picked mon.noname-a con 0x7f28241397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28240706c0 con 0x7f28241397f0 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f282409b8e0 con 0x7f282409bb40 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f282406def0 con 0x7f282409bf20 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient(hunting): authenticate will time out at 1290.646744s 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2828d11640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f282409bf20 0x7f28241cbf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:32934/0 (socket says 172.21.15.6:32934) 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2828d11640 1 -- 172.21.15.6:0/3882751000 learned_addr learned my addr 172.21.15.6:0/3882751000 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2829d13640 1 -- 172.21.15.6:0/3882751000 >> v1:172.21.15.6:6789/0 conn(0x7f28241397f0 legacy=0x7f28241cf630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779070756 0 0) 0x7f282406def0 con 0x7f282409bf20 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): my global_id is 6329 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2808003530 con 0x7f282409bf20 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1869368765 0 0) 0x7f2808003530 con 0x7f282409bf20 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f282406def0 con 0x7f282409bf20 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2814004160 con 0x7f282409bf20 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): dump: 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3843073679 0 0) 0x7f282409d940 con 0x7f282409bb40 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient(hunting): my global_id is 114184 2024-09-16T05:40:28.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2808003530 con 0x7f282409bb40 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 415140662 0 0) 0x7f282406def0 con 0x7f282409bf20 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 >> v1:172.21.15.63:6789/0 conn(0x7f282409bb40 legacy=0x7f282412f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 >> v1:172.21.15.6:6789/0 conn(0x7f28241397f0 legacy=0x7f28241cf630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 monclient: found mon.c 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28240715b0 con 0x7f282409bf20 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 5 monclient: authenticate success, global_id 6329 2024-09-16T05:40:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: _renew_subs 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2824071890 con 0x7f282409bf20 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: _renew_subs 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2824070440 con 0x7f282409bf20 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2814004450 con 0x7f282409bf20 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2814004820 con 0x7f282409bf20 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f281b7fe640 4 set_mon_vals no callback set 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.836+0000 7f2819ffb640 10 monclient: dump: 2024-09-16T05:40:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.840+0000 7f282b776640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.840+0000 7f282b776640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.840+0000 7f282b776640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:28.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.840+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2814004a40 con 0x7f282409bf20 2024-09-16T05:40:28.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.840+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2804005090 con 0x7f282409bf20 2024-09-16T05:40:28.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.840+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(236..236 src has 20..236) ==== 10337+0+0 (unknown 1376479996 0 0) 0x7f2814045dd0 con 0x7f282409bf20 2024-09-16T05:40:28.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.840+0000 7f2829512640 10 monclient: get_auth_request con 0x7f2808036aa0 auth_method 0 2024-09-16T05:40:28.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.844+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2814015580 con 0x7f282409bf20 2024-09-16T05:40:28.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.844+0000 7f2819ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:28.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:28.844+0000 7f2819ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:29.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:40:29.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:29.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:29.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2804005380 con 0x7f282409bf20 2024-09-16T05:40:29.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f2819ffb640 1 -- 172.21.15.6:0/3882751000 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3461217158) 0x7f2814019230 con 0x7f282409bf20 2024-09-16T05:40:29.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f2819ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:29.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f2819ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:40:29.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2808036aa0 legacy=0x7f2808038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:29.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 10 monclient: shutdown 2024-09-16T05:40:29.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:29.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 >> v1:172.21.15.6:6790/0 conn(0x7f282409bf20 legacy=0x7f28241cbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:29.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 shutdown_connections 2024-09-16T05:40:29.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 >> 172.21.15.6:0/3882751000 conn(0x7f28240a1650 msgr2=0x7f28240a3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:29.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 shutdown_connections 2024-09-16T05:40:29.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:29.068+0000 7f282b776640 1 -- 172.21.15.6:0/3882751000 wait complete. 2024-09-16T05:40:30.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:30.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:30.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:30.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:30.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:30.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:30.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:30.829+0000 7f7c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:31.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:31.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:31.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:31.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:31.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:31.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:32.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:40:32.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient: get_monmap_and_config 2024-09-16T05:40:32.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient: build_initial_monmap 2024-09-16T05:40:32.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient: monmap: 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:32.224036+0000 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:32.224036+0000 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:32.224 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:32.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 1 Processor -- start 2024-09-16T05:40:32.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 1 -- start start 2024-09-16T05:40:32.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient: init 2024-09-16T05:40:32.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient(hunting): picked mon.noname-a con 0x7efd6409bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient(hunting): picked mon.noname-c con 0x7efd6409bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient(hunting): picked mon.noname-b con 0x7efd641397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd640a1350 con 0x7efd6409bb40 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd640a10d0 con 0x7efd6409bf20 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd6409d940 con 0x7efd641397f0 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd63fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7efd6409bb40 legacy=0x7efd6409e600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient(hunting): authenticate will time out at 1294.030639s 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd68909640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7efd641397f0 0x7efd6413bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45974/0 (socket says 172.21.15.6:45974) 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd68909640 1 -- 172.21.15.6:0/535156884 learned_addr learned my addr 172.21.15.6:0/535156884 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 670356774 0 0) 0x7efd6409d940 con 0x7efd641397f0 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient(hunting): my global_id is 6332 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd4c005740 con 0x7efd641397f0 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1851713924 0 0) 0x7efd4c005740 con 0x7efd641397f0 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 >> v1:172.21.15.63:6789/0 conn(0x7efd6409bf20 legacy=0x7efd6409c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 >> v1:172.21.15.6:6789/0 conn(0x7efd6409bb40 legacy=0x7efd6409e600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 monclient: found mon.noname-b 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6409dc20 con 0x7efd641397f0 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efd5c0019e0 con 0x7efd641397f0 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: dump: 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 5 monclient: authenticate success, global_id 6332 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd5c001c80 con 0x7efd641397f0 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 1 -- 172.21.15.6:0/535156884 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efd5c002050 con 0x7efd641397f0 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:32.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd62ffd640 10 monclient: dump: 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 4 set_mon_vals no callback set 2024-09-16T05:40:32.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 10 monclient: shutdown 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.220+0000 7efd6a36c640 1 -- 172.21.15.6:0/535156884 >> v1:172.21.15.6:6790/0 conn(0x7efd641397f0 legacy=0x7efd6413bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- 172.21.15.6:0/535156884 shutdown_connections 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- 172.21.15.6:0/535156884 >> 172.21.15.6:0/535156884 conn(0x7efd640a1650 msgr2=0x7efd640a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- 172.21.15.6:0/535156884 shutdown_connections 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- 172.21.15.6:0/535156884 wait complete. 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: build_initial_monmap 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: monmap: 2024-09-16T05:40:32.236 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 Processor -- start 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- start start 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: init 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient(hunting): picked mon.noname-c con 0x7efd6409bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient(hunting): picked mon.noname-b con 0x7efd6409bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient(hunting): picked mon.noname-a con 0x7efd641397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd640709e0 con 0x7efd641397f0 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd64071510 con 0x7efd6409bf20 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd68909640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7efd641397f0 legacy=0x7efd641cf9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd6406ed10 con 0x7efd6409bb40 2024-09-16T05:40:32.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient(hunting): authenticate will time out at 1294.033780s 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd63fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7efd6409bb40 0x7efd6406def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45978/0 (socket says 172.21.15.6:45978) 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd63fff640 1 -- 172.21.15.6:0/3205965587 learned_addr learned my addr 172.21.15.6:0/3205965587 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131975546 0 0) 0x7efd6406ed10 con 0x7efd6409bb40 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): my global_id is 6335 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd50003530 con 0x7efd6409bb40 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2959608298 0 0) 0x7efd50003530 con 0x7efd6409bb40 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd6406ed10 con 0x7efd6409bb40 2024-09-16T05:40:32.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3758981734 0 0) 0x7efd64071510 con 0x7efd6409bf20 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): my global_id is 114187 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd50003530 con 0x7efd6409bf20 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efd54004160 con 0x7efd6409bb40 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): dump: 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:32.240 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1570326336 0 0) 0x7efd6406ed10 con 0x7efd6409bb40 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 >> v1:172.21.15.63:6789/0 conn(0x7efd6409bf20 legacy=0x7efd6406e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 >> v1:172.21.15.6:6789/0 conn(0x7efd641397f0 legacy=0x7efd641cf9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:40:32.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 monclient: found mon.c 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd640717f0 con 0x7efd6409bb40 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient: _finish_auth 0 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 5 monclient: authenticate success, global_id 6335 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: _renew_subs 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd640a3df0 con 0x7efd6409bb40 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: _renew_subs 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:32.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efd64071230 con 0x7efd6409bb40 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd540043c0 con 0x7efd6409bb40 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efd54004790 con 0x7efd6409bb40 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd627fc640 4 set_mon_vals no callback set 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 10 monclient: dump: 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:32.243 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd44005090 con 0x7efd6409bb40 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7efd540049b0 con 0x7efd6409bb40 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.224+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(236..236 src has 20..236) ==== 10337+0+0 (unknown 1376479996 0 0) 0x7efd54045cc0 con 0x7efd6409bb40 2024-09-16T05:40:32.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.228+0000 7efd637fe640 10 monclient: get_auth_request con 0x7efd50036ad0 auth_method 0 2024-09-16T05:40:32.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.232+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7efd54015530 con 0x7efd6409bb40 2024-09-16T05:40:32.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.232+0000 7efd60ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:32.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.232+0000 7efd60ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:32.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:40:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efd44005380 con 0x7efd6409bb40 2024-09-16T05:40:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd60ff9640 1 -- 172.21.15.6:0/3205965587 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3273348353) 0x7efd54014e30 con 0x7efd6409bb40 2024-09-16T05:40:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd60ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:32.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd60ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:32.468 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:40:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 >> v1:172.21.15.63:6812/3643207211 conn(0x7efd50036ad0 legacy=0x7efd50038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 10 monclient: shutdown 2024-09-16T05:40:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 >> v1:172.21.15.6:6790/0 conn(0x7efd6409bb40 legacy=0x7efd6406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 shutdown_connections 2024-09-16T05:40:32.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 >> 172.21.15.6:0/3205965587 conn(0x7efd640a1650 msgr2=0x7efd640a2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:32.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.464+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 shutdown_connections 2024-09-16T05:40:32.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:32.468+0000 7efd6a36c640 1 -- 172.21.15.6:0/3205965587 wait complete. 2024-09-16T05:40:32.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:32.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:32.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:32.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:32.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:32.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:33.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:33.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:33.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:33.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:33.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:33.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:34.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:34.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:34.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:34.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:35.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:40:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient: get_monmap_and_config 2024-09-16T05:40:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient: build_initial_monmap 2024-09-16T05:40:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient: monmap: 2024-09-16T05:40:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:35.620745+0000 2024-09-16T05:40:35.620 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:35.620745+0000 2024-09-16T05:40:35.621 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:35.621 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:35.621 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:35.621 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:35.621 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:35.621 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:35.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 1 Processor -- start 2024-09-16T05:40:35.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 1 -- start start 2024-09-16T05:40:35.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient: init 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient(hunting): picked mon.noname-b con 0x7f432409bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient(hunting): picked mon.noname-c con 0x7f432409bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient(hunting): picked mon.noname-a con 0x7f43241397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43240a1350 con 0x7f43241397f0 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43240a10d0 con 0x7f432409bf20 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f432409d940 con 0x7f432409bb40 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432b2e2640 10 monclient(hunting): authenticate will time out at 1297.427693s 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432987f640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f432413bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:35.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432907e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f432409bb40 0x7f432409e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45988/0 (socket says 172.21.15.6:45988) 2024-09-16T05:40:35.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.616+0000 7f432907e640 1 -- 172.21.15.6:0/4102739250 learned_addr learned my addr 172.21.15.6:0/4102739250 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:35.820+0000 7f432987f640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f432413bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:35.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:35.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:35.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:35.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:35.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:35.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:36.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:36.224+0000 7f432987f640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f432413bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:36.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:36.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:36.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:36.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:36.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:36.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:37.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:37.028+0000 7f432987f640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f432413bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:37.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:37.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:37.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:37.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:37.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:37.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:38.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 10 monclient(hunting): tick 2024-09-16T05:40:38.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:40:38.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 1 monclient(hunting): continuing hunt 2024-09-16T05:40:38.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:40:38.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6790/0 conn(0x7f432409bb40 legacy=0x7f432409e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.63:6789/0 conn(0x7f432409bf20 legacy=0x7f432409c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f432413bbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 10 monclient(hunting): picked mon.noname-c con 0x7f43080058e0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.616+0000 7f431affd640 10 monclient(hunting): picked mon.noname-a con 0x7f43241397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.620+0000 7f431affd640 10 monclient(hunting): picked mon.noname-b con 0x7f432409bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f432409dc20 con 0x7f43241397f0 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43240a1350 con 0x7f43080058e0 2024-09-16T05:40:38.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43080125f0 con 0x7f432409bb40 2024-09-16T05:40:38.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.620+0000 7f431affd640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:38.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.620+0000 7f432887d640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f430800b5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:38.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:38.824+0000 7f432887d640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f430800b5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:38.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:38.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:38.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:38.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:38.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:38.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:39.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:39.228+0000 7f432887d640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f430800b5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:39.833 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:39.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:39.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:39.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:39.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:39.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:40.032+0000 7f432887d640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f430800b5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:40.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:40.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:40.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:40.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:40.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:40.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:40.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:40.829+0000 7f7c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 10 monclient(hunting): tick 2024-09-16T05:40:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:40:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 1 monclient(hunting): continuing hunt 2024-09-16T05:40:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:40:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6790/0 conn(0x7f432409bb40 legacy=0x7f430800ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.63:6789/0 conn(0x7f43080058e0 legacy=0x7f4308007d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f430800b5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 10 monclient(hunting): picked mon.noname-b con 0x7f4308011670 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 10 monclient(hunting): picked mon.noname-c con 0x7f43080127c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 10 monclient(hunting): picked mon.noname-a con 0x7f43241397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4308013900 con 0x7f43241397f0 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f432409dc20 con 0x7f43080127c0 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 1 -- 172.21.15.6:0/4102739250 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4308013350 con 0x7f4308011670 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f431affd640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.620+0000 7f432987f640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f4308007cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:41.824+0000 7f432987f640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f4308007cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:41.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:41.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:41.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:41.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:41.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:41.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:42.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:42.228+0000 7f432987f640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f4308007cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:42.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:42.829+0000 7f7c1b7fe640 10 monclient: tick 2024-09-16T05:40:42.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:42.829+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:42.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:42.829+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:43.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.032+0000 7f432987f640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f4308007cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.624+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2385250008 0 0) 0x7f4308013350 con 0x7f4308011670 2024-09-16T05:40:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.624+0000 7f431bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.624+0000 7f431bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.624+0000 7f431bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.624+0000 7f431bfff640 10 monclient(hunting): my global_id is 6338 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.624+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4310003530 con 0x7f4308011670 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.624+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3672011903 0 0) 0x7f4310003530 con 0x7f4308011670 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.63:6789/0 conn(0x7f43080127c0 legacy=0x7f4308012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6789/0 conn(0x7f43241397f0 legacy=0x7f4308007cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 1 monclient: found mon.noname-b 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4308013520 con 0x7f4308011670 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: _finish_auth 0 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4320001920 con 0x7f4308011670 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: dump: 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 5 monclient: authenticate success, global_id 6338 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4320001bb0 con 0x7f4308011670 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 1 -- 172.21.15.6:0/4102739250 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4320001fc0 con 0x7f4308011670 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f431bfff640 10 monclient: dump: 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:43.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 4 set_mon_vals no callback set 2024-09-16T05:40:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 10 monclient: shutdown 2024-09-16T05:40:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 1 -- 172.21.15.6:0/4102739250 >> v1:172.21.15.6:6790/0 conn(0x7f4308011670 legacy=0x7f4308011a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 1 -- 172.21.15.6:0/4102739250 shutdown_connections 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 1 -- 172.21.15.6:0/4102739250 >> 172.21.15.6:0/4102739250 conn(0x7f43240a1650 msgr2=0x7f43240a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 1 -- 172.21.15.6:0/4102739250 shutdown_connections 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 1 -- 172.21.15.6:0/4102739250 wait complete. 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 10 monclient: build_initial_monmap 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 10 monclient: monmap: 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.628+0000 7f432b2e2640 1 Processor -- start 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 1 -- start start 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: init 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4308011670 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient(hunting): picked mon.noname-b con 0x7f43080127c0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient(hunting): picked mon.noname-c con 0x7f43241397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4324122910 con 0x7f4308011670 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4324072540 con 0x7f43080127c0 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432907e640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f4308011670 legacy=0x7f43241cbc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43240a3540 con 0x7f43241397f0 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient(hunting): authenticate will time out at 1305.441070s 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432987f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f43241397f0 0x7f43241d4390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42032/0 (socket says 172.21.15.6:42032) 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432987f640 1 -- 172.21.15.6:0/3316528971 learned_addr learned my addr 172.21.15.6:0/3316528971 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4227306917 0 0) 0x7f43240a3540 con 0x7f43241397f0 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient(hunting): my global_id is 6341 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4308013750 con 0x7f43241397f0 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3739664097 0 0) 0x7f4308013750 con 0x7f43241397f0 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43240a3540 con 0x7f43241397f0 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f431c004270 con 0x7f43241397f0 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient(hunting): dump: 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3261360284 0 0) 0x7f43240a3540 con 0x7f43241397f0 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 >> v1:172.21.15.63:6789/0 conn(0x7f43080127c0 legacy=0x7f43241d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 >> v1:172.21.15.6:6789/0 conn(0x7f4308011670 legacy=0x7f43241cbc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:43.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 monclient: found mon.c 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43240740c0 con 0x7f43241397f0 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 5 monclient: authenticate success, global_id 6341 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: _renew_subs 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4324122350 con 0x7f43241397f0 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: _renew_subs 2024-09-16T05:40:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4324120f00 con 0x7f43241397f0 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f431c004780 con 0x7f43241397f0 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f431c004ad0 con 0x7f43241397f0 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 10 monclient: dump: 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f431b7fe640 4 set_mon_vals no callback set 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f431c004c70 con 0x7f43241397f0 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.632+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4304005090 con 0x7f43241397f0 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.636+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(236..236 src has 20..236) ==== 10337+0+0 (unknown 1376479996 0 0) 0x7f431c045bb0 con 0x7f43241397f0 2024-09-16T05:40:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.636+0000 7f432907e640 10 monclient: get_auth_request con 0x7f4308037f40 auth_method 0 2024-09-16T05:40:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.640+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f431c015360 con 0x7f43241397f0 2024-09-16T05:40:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.640+0000 7f4319ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.640+0000 7f4319ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:43.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.633+0000 7f7c30ff9640 1 -- 172.21.15.63:0/752582467 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 device 'h18' does not appear in the crush map v236) ==== 120+0+0 (unknown 1461849458 0 0) 0x7f7c28042510 con 0x7f7c3409ac90 2024-09-16T05:40:43.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.633+0000 7f7c30ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T05:40:43.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.633+0000 7f7c30ff9640 10 monclient: _finish_command 3 = system:0 device 'h18' does not appear in the crush map 2024-09-16T05:40:43.649 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h18' does not appear in the crush map 2024-09-16T05:40:43.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.637+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7c20036d00 legacy=0x7f7c200391c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.637+0000 7f7c3a8c4640 10 monclient: shutdown 2024-09-16T05:40:43.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.637+0000 7f7c3a8c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:43.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.637+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 >> v1:172.21.15.63:6789/0 conn(0x7f7c3409ac90 legacy=0x7f7c341d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.637+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 shutdown_connections 2024-09-16T05:40:43.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.637+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 >> 172.21.15.63:0/752582467 conn(0x7f7c340a1c90 msgr2=0x7f7c3406ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:43.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.637+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 shutdown_connections 2024-09-16T05:40:43.650 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.637+0000 7f7c3a8c4640 1 -- 172.21.15.63:0/752582467 wait complete. 2024-09-16T05:40:43.661 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:43.661 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h19 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient: get_monmap_and_config 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient: build_initial_monmap 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient: monmap: 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:43.790105+0000 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:43.790105+0000 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:43.790 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:43.791 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:43.791 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:43.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 1 Processor -- start 2024-09-16T05:40:43.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 1 -- start start 2024-09-16T05:40:43.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient: init 2024-09-16T05:40:43.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:43.798 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient(hunting): picked mon.noname-a con 0x7f542809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient(hunting): picked mon.noname-c con 0x7f542806dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient(hunting): picked mon.noname-b con 0x7f54280a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542809f800 con 0x7f542809ac90 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54280a0ab0 con 0x7f542806dd30 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54280a0d90 con 0x7f54280a0210 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f542d625640 10 monclient(hunting): authenticate will time out at 1300.567403s 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f5426ffd640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f542809ac90 legacy=0x7f542809b070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f54267fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f542806dd30 0x7f54281399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59594/0 (socket says 172.21.15.63:59594) 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f54267fc640 1 -- 172.21.15.63:0/4048661094 learned_addr learned my addr 172.21.15.63:0/4048661094 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:43.799 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1843150666 0 0) 0x7f54280a0ab0 con 0x7f542806dd30 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f5425ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f5425ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f5425ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f5425ffb640 10 monclient(hunting): my global_id is 114190 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.785+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5410003530 con 0x7f542806dd30 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565051756 0 0) 0x7f54280a0d90 con 0x7f54280a0210 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient(hunting): my global_id is 6344 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54280a0ab0 con 0x7f54280a0210 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4064228029 0 0) 0x7f5410003530 con 0x7f542806dd30 2024-09-16T05:40:43.800 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 >> v1:172.21.15.6:6790/0 conn(0x7f54280a0210 legacy=0x7f542813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 >> v1:172.21.15.6:6789/0 conn(0x7f542809ac90 legacy=0x7f542809b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 monclient: found mon.noname-c 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542809e770 con 0x7f542806dd30 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 5 monclient: authenticate success, global_id 114190 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f541c001620 con 0x7f542806dd30 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:43.801 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: dump: 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:43.802 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f541c001940 con 0x7f542806dd30 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 1 -- 172.21.15.63:0/4048661094 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f541c001d10 con 0x7f542806dd30 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 4 set_mon_vals no callback set 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5425ffb640 10 monclient: dump: 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:43.803 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: shutdown 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- 172.21.15.63:0/4048661094 >> v1:172.21.15.63:6789/0 conn(0x7f542806dd30 legacy=0x7f54281399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- 172.21.15.63:0/4048661094 shutdown_connections 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- 172.21.15.63:0/4048661094 >> 172.21.15.63:0/4048661094 conn(0x7f54280a1c90 msgr2=0x7f54280a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- 172.21.15.63:0/4048661094 shutdown_connections 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- 172.21.15.63:0/4048661094 wait complete. 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: build_initial_monmap 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: monmap: 2024-09-16T05:40:43.804 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 Processor -- start 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- start start 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: init 2024-09-16T05:40:43.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient(hunting): picked mon.noname-b con 0x7f542806dd30 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient(hunting): picked mon.noname-a con 0x7f542809ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient(hunting): picked mon.noname-c con 0x7f54280a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5428122d40 con 0x7f542809ac90 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5428123020 con 0x7f542806dd30 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54281344d0 con 0x7f54280a0210 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient(hunting): authenticate will time out at 1300.570448s 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f54267fc640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f542809ac90 legacy=0x7f54281d0220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5426ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f542806dd30 0x7f5428133920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59596/0 (socket says 172.21.15.63:59596) 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f5426ffd640 1 -- 172.21.15.63:0/790154632 learned_addr learned my addr 172.21.15.63:0/790154632 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:43.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346618198 0 0) 0x7f5428123020 con 0x7f542806dd30 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): my global_id is 114193 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5414003530 con 0x7f542806dd30 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731163736 0 0) 0x7f54281344d0 con 0x7f54280a0210 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): my global_id is 6347 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5428123020 con 0x7f54280a0210 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 434330937 0 0) 0x7f5414003530 con 0x7f542806dd30 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54281344d0 con 0x7f542806dd30 2024-09-16T05:40:43.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5420004510 con 0x7f542806dd30 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:43.808 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3154300935 0 0) 0x7f54281344d0 con 0x7f542806dd30 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 >> v1:172.21.15.6:6790/0 conn(0x7f54280a0210 legacy=0x7f54281d3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 >> v1:172.21.15.6:6789/0 conn(0x7f542809ac90 legacy=0x7f54281d0220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 monclient: found mon.b 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5428134030 con 0x7f542806dd30 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:43.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 5 monclient: authenticate success, global_id 114193 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: _renew_subs 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f54281caf00 con 0x7f542806dd30 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: _renew_subs 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f54281d5070 con 0x7f542806dd30 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5420004830 con 0x7f542806dd30 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5420004c00 con 0x7f542806dd30 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f54257fa640 4 set_mon_vals no callback set 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.789+0000 7f540f7fe640 10 monclient: dump: 2024-09-16T05:40:43.810 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.793+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5420004e20 con 0x7f542806dd30 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.793+0000 7f542d625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.793+0000 7f542d625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.793+0000 7f542d625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:43.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.793+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5408005090 con 0x7f542806dd30 2024-09-16T05:40:43.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.793+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(236..236 src has 20..236) ==== 10337+0+0 (unknown 1376479996 0 0) 0x7f54200460c0 con 0x7f542806dd30 2024-09-16T05:40:43.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.793+0000 7f54267fc640 10 monclient: get_auth_request con 0x7f5414036a70 auth_method 0 2024-09-16T05:40:43.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.797+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5420015930 con 0x7f542806dd30 2024-09-16T05:40:43.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.797+0000 7f540f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:43.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.797+0000 7f540f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:43.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:40:43.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:43.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:43.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4304005380 con 0x7f43241397f0 2024-09-16T05:40:43.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f4319ffb640 1 -- 172.21.15.6:0/3316528971 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 1016853440) 0x7f431c019010 con 0x7f43241397f0 2024-09-16T05:40:43.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f4319ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:40:43.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f4319ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:43.876 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":30,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:40:43.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4308037f40 legacy=0x7f430803a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 10 monclient: shutdown 2024-09-16T05:40:43.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:43.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 >> v1:172.21.15.6:6790/0 conn(0x7f43241397f0 legacy=0x7f43241d4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:43.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 shutdown_connections 2024-09-16T05:40:43.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 >> 172.21.15.6:0/3316528971 conn(0x7f43240a1650 msgr2=0x7f432409b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:43.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 shutdown_connections 2024-09-16T05:40:43.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:43.872+0000 7f432b2e2640 1 -- 172.21.15.6:0/3316528971 wait complete. 2024-09-16T05:40:43.895 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:40:43.895 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:40:43.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:40:43.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.965+0000 7f542d625640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T05:40:43.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.965+0000 7f542d625640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T05:40:43.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.965+0000 7f542d625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:43.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:43.965+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) -- 0x7f5408005380 con 0x7f542806dd30 2024-09-16T05:40:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient: get_monmap_and_config 2024-09-16T05:40:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient: build_initial_monmap 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient: monmap: 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:44.020353+0000 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:44.020353+0000 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:44.020 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 1 Processor -- start 2024-09-16T05:40:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 1 -- start start 2024-09-16T05:40:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient: init 2024-09-16T05:40:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient(hunting): picked mon.noname-c con 0x7f95700a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient(hunting): picked mon.noname-a con 0x7f95700a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient(hunting): picked mon.noname-b con 0x7f95700a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f957006e520 con 0x7f95700a4290 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95700a0780 con 0x7f95700a37a0 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95700a4c70 con 0x7f95700a1a00 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f9577f62640 10 monclient(hunting): authenticate will time out at 1305.827571s 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f95754fd640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f95700a4290 legacy=0x7f957013bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f95764ff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f95700a1a00 0x7f957013f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42040/0 (socket says 172.21.15.6:42040) 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.016+0000 7f95764ff640 1 -- 172.21.15.6:0/3893924042 learned_addr learned my addr 172.21.15.6:0/3893924042 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2940562717 0 0) 0x7f95700a4c70 con 0x7f95700a1a00 2024-09-16T05:40:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient(hunting): my global_id is 6350 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f955c003530 con 0x7f95700a1a00 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 212203882 0 0) 0x7f955c003530 con 0x7f95700a1a00 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 >> v1:172.21.15.63:6789/0 conn(0x7f95700a37a0 legacy=0x7f95700a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 >> v1:172.21.15.6:6789/0 conn(0x7f95700a4290 legacy=0x7f957013bab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 monclient: found mon.noname-b 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95700a65f0 con 0x7f95700a1a00 2024-09-16T05:40:44.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: _finish_auth 0 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f95680019a0 con 0x7f95700a1a00 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 5 monclient: authenticate success, global_id 6350 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: dump: 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9568001c80 con 0x7f95700a1a00 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 1 -- 172.21.15.6:0/3893924042 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9568002090 con 0x7f95700a1a00 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 4 set_mon_vals no callback set 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:44.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9574cfc640 10 monclient: dump: 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: shutdown 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:44.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- 172.21.15.6:0/3893924042 >> v1:172.21.15.6:6790/0 conn(0x7f95700a1a00 legacy=0x7f957013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- 172.21.15.6:0/3893924042 shutdown_connections 2024-09-16T05:40:44.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- 172.21.15.6:0/3893924042 >> 172.21.15.6:0/3893924042 conn(0x7f95700a58a0 msgr2=0x7f95700a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:44.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- 172.21.15.6:0/3893924042 shutdown_connections 2024-09-16T05:40:44.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- 172.21.15.6:0/3893924042 wait complete. 2024-09-16T05:40:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: build_initial_monmap 2024-09-16T05:40:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: monmap: 2024-09-16T05:40:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 Processor -- start 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- start start 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: init 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient(hunting): picked mon.noname-b con 0x7f95700a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient(hunting): picked mon.noname-c con 0x7f95700a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient(hunting): picked mon.noname-a con 0x7f95700a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95701cbc90 con 0x7f95700a4290 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95700a0100 con 0x7f95700a1a00 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95700a7bb0 con 0x7f95700a37a0 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient(hunting): authenticate will time out at 1305.830723s 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95764ff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f95700a4290 legacy=0x7f95701d3c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95754fd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f95700a37a0 0x7f95701d0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42052/0 (socket says 172.21.15.6:42052) 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95754fd640 1 -- 172.21.15.6:0/2724536015 learned_addr learned my addr 172.21.15.6:0/2724536015 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 62470550 0 0) 0x7f95700a7bb0 con 0x7f95700a37a0 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): my global_id is 6353 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9554003530 con 0x7f95700a37a0 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2010178125 0 0) 0x7f9554003530 con 0x7f95700a37a0 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95700a7bb0 con 0x7f95700a37a0 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9560004160 con 0x7f95700a37a0 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): dump: 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.036 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221890045 0 0) 0x7f95700a0100 con 0x7f95700a1a00 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient(hunting): my global_id is 114199 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9554003530 con 0x7f95700a1a00 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 472479968 0 0) 0x7f95700a7bb0 con 0x7f95700a37a0 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 >> v1:172.21.15.63:6789/0 conn(0x7f95700a1a00 legacy=0x7f9570136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 >> v1:172.21.15.6:6789/0 conn(0x7f95700a4290 legacy=0x7f95701d3c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 monclient: found mon.c 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9570137670 con 0x7f95700a37a0 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 5 monclient: authenticate success, global_id 6353 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: _renew_subs 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9570135190 con 0x7f95700a37a0 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: _renew_subs 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9570134450 con 0x7f95700a37a0 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9560004450 con 0x7f95700a37a0 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9560004820 con 0x7f95700a37a0 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f9567fff640 4 set_mon_vals no callback set 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.020+0000 7f95667fc640 10 monclient: dump: 2024-09-16T05:40:44.039 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f9577f62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f95580065d0 con 0x7f9558002870 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9560004a40 con 0x7f95700a37a0 2024-09-16T05:40:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375182978 0 0) 0x7f95580065d0 con 0x7f9558002870 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient(hunting): my global_id is 6356 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95700a7bb0 con 0x7f9558002870 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(236..236 src has 20..236) ==== 10337+0+0 (unknown 1376479996 0 0) 0x7f9560045d00 con 0x7f95700a37a0 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2845868232 0 0) 0x7f95700a7bb0 con 0x7f9558002870 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95580065d0 con 0x7f9558002870 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f956c001750 con 0x7f9558002870 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95764ff640 10 monclient: get_auth_request con 0x7f9554036a30 auth_method 0 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:44.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient: dump: 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 544414143 0 0) 0x7f95580065d0 con 0x7f9558002870 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f955800a160 con 0x7f9558002870 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f956c001a70 con 0x7f9558002870 2024-09-16T05:40:44.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 >> v1:172.21.15.6:6790/0 conn(0x7f9558002870 legacy=0x7f9558004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.024+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 >> v1:172.21.15.6:6790/0 conn(0x7f9558002870 legacy=0x7f9558004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:44.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.028+0000 7f9577f62640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:40:44.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.028+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9558009e30 con 0x7f95580090f0 2024-09-16T05:40:44.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3056233161 0 0) 0x7f9558009e30 con 0x7f95580090f0 2024-09-16T05:40:44.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient(hunting): my global_id is 6359 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95580065d0 con 0x7f95580090f0 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2887429888 0 0) 0x7f95580065d0 con 0x7f95580090f0 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9558009e30 con 0x7f95580090f0 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f956c0037a0 con 0x7f95580090f0 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient: dump: 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.045 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2172442704 0 0) 0x7f9558009e30 con 0x7f95580090f0 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f955800c3a0 con 0x7f95580090f0 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4118 (unknown 0 0 3681629227) 0x7f956c003990 con 0x7f95580090f0 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 >> v1:172.21.15.6:6790/0 conn(0x7f95580090f0 legacy=0x7f9558009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95667fc640 1 -- 172.21.15.6:0/2724536015 >> v1:172.21.15.6:6790/0 conn(0x7f95580090f0 legacy=0x7f9558009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 30, 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 373088, 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:40:44.047 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:40:44.050 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:44.051 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:40:44.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:40:44.053 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:44.054 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:44.055 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:44.056 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.057 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.058 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.059 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:40:44.059 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:44.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9554036a30 legacy=0x7f9554038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f9577f62640 10 monclient: shutdown 2024-09-16T05:40:44.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f9577f62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:44.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 >> v1:172.21.15.6:6790/0 conn(0x7f95700a37a0 legacy=0x7f95701d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95764ff640 1 -- 172.21.15.6:0/2724536015 reap_dead start 2024-09-16T05:40:44.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f95764ff640 1 -- 172.21.15.6:0/2724536015 reap_dead start 2024-09-16T05:40:44.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 shutdown_connections 2024-09-16T05:40:44.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 >> 172.21.15.6:0/2724536015 conn(0x7f95700a58a0 msgr2=0x7f95700a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:44.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 shutdown_connections 2024-09-16T05:40:44.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.032+0000 7f9577f62640 1 -- 172.21.15.6:0/2724536015 wait complete. 2024-09-16T05:40:44.061 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:40:44.061 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:40:44.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:40:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.172+0000 7fd03ec5e640 10 monclient: get_monmap_and_config 2024-09-16T05:40:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.172+0000 7fd03ec5e640 10 monclient: build_initial_monmap 2024-09-16T05:40:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.172+0000 7fd03ec5e640 10 monclient: monmap: 2024-09-16T05:40:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.178 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:40:44.178067+0000 2024-09-16T05:40:44.178 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:40:44.178067+0000 2024-09-16T05:40:44.178 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:44.178 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.178 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:44.178 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:44.178 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:44.178 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 Processor -- start 2024-09-16T05:40:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- start start 2024-09-16T05:40:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient: init 2024-09-16T05:40:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient(hunting): picked mon.noname-b con 0x7fd0380a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient(hunting): picked mon.noname-c con 0x7fd0380a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient(hunting): picked mon.noname-a con 0x7fd0380a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03806e520 con 0x7fd0380a1a00 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0380a0780 con 0x7fd0380a4290 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0380a4c70 con 0x7fd0380a37a0 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient(hunting): authenticate will time out at 1305.985298s 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03d1fb640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fd0380a1a00 legacy=0x7fd03813f1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03c9fa640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd0380a37a0 0x7fd0380a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42098/0 (socket says 172.21.15.6:42098) 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03c9fa640 1 -- 172.21.15.6:0/1991340200 learned_addr learned my addr 172.21.15.6:0/1991340200 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2647579719 0 0) 0x7fd0380a4c70 con 0x7fd0380a37a0 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient(hunting): my global_id is 6362 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd020003530 con 0x7fd0380a37a0 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 745137735 0 0) 0x7fd020003530 con 0x7fd0380a37a0 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 >> v1:172.21.15.63:6789/0 conn(0x7fd0380a4290 legacy=0x7fd03813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 >> v1:172.21.15.6:6789/0 conn(0x7fd0380a1a00 legacy=0x7fd03813f1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 monclient: found mon.noname-b 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0380a65f0 con 0x7fd0380a37a0 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 5 monclient: authenticate success, global_id 6362 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd028001620 con 0x7fd0380a37a0 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: dump: 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd028001980 con 0x7fd0380a37a0 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 1 -- 172.21.15.6:0/1991340200 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd0280048b0 con 0x7fd0380a37a0 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 4 set_mon_vals no callback set 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd0377fe640 10 monclient: dump: 2024-09-16T05:40:44.185 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.186 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient: shutdown 2024-09-16T05:40:44.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:44.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1991340200 >> v1:172.21.15.6:6790/0 conn(0x7fd0380a37a0 legacy=0x7fd0380a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1991340200 shutdown_connections 2024-09-16T05:40:44.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1991340200 >> 172.21.15.6:0/1991340200 conn(0x7fd0380a58a0 msgr2=0x7fd0380a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:44.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1991340200 shutdown_connections 2024-09-16T05:40:44.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1991340200 wait complete. 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient: build_initial_monmap 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 10 monclient: monmap: 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:44.191 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.176+0000 7fd03ec5e640 1 Processor -- start 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 1 -- start start 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient: init 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient(hunting): picked mon.noname-c con 0x7fd0380a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient(hunting): picked mon.noname-b con 0x7fd0380a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient(hunting): picked mon.noname-a con 0x7fd0380a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03813ee30 con 0x7fd0380a4290 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03813bfa0 con 0x7fd0380a37a0 2024-09-16T05:40:44.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03813d980 con 0x7fd0380a1a00 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient(hunting): authenticate will time out at 1305.988650s 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03d1fb640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fd0380a4290 legacy=0x7fd0381d3d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03c9fa640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd0380a1a00 0x7fd03813b180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42112/0 (socket says 172.21.15.6:42112) 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03c9fa640 1 -- 172.21.15.6:0/1221981132 learned_addr learned my addr 172.21.15.6:0/1221981132 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3167488379 0 0) 0x7fd03813d980 con 0x7fd0380a1a00 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): my global_id is 6365 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd024003530 con 0x7fd0380a1a00 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4163253080 0 0) 0x7fd024003530 con 0x7fd0380a1a00 2024-09-16T05:40:44.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd03813d980 con 0x7fd0380a1a00 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd028001440 con 0x7fd0380a1a00 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): dump: 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.194 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122176560 0 0) 0x7fd03813bfa0 con 0x7fd0380a37a0 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): my global_id is 114205 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd024003530 con 0x7fd0380a37a0 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2838675681 0 0) 0x7fd03813d980 con 0x7fd0380a1a00 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 >> v1:172.21.15.63:6789/0 conn(0x7fd0380a37a0 legacy=0x7fd03813b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 >> v1:172.21.15.6:6789/0 conn(0x7fd0380a4290 legacy=0x7fd0381d3d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: _finish_hunting 0 2024-09-16T05:40:44.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 monclient: found mon.c 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03813f110 con 0x7fd0380a1a00 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: _finish_auth 0 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: _check_auth_tickets 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 5 monclient: authenticate success, global_id 6365 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient: _renew_subs 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd0380a7f60 con 0x7fd0380a1a00 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient: _renew_subs 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:40:44.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd03813eb50 con 0x7fd0380a1a00 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd028001980 con 0x7fd0380a1a00 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd028005010 con 0x7fd0380a1a00 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: dump: 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.197 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.198 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.198 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.198 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.198 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd036ffd640 4 set_mon_vals no callback set 2024-09-16T05:40:44.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:44.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd0180065d0 con 0x7fd018002870 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd02800fdb0 con 0x7fd0380a1a00 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(236..236 src has 20..236) ==== 10337+0+0 (unknown 1376479996 0 0) 0x7fd028045e40 con 0x7fd0380a1a00 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225645602 0 0) 0x7fd0180065d0 con 0x7fd018002870 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient(hunting): my global_id is 114208 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd03813d980 con 0x7fd018002870 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd03d1fb640 10 monclient: get_auth_request con 0x7fd024036aa0 auth_method 0 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1699006900 0 0) 0x7fd03813d980 con 0x7fd018002870 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0180065d0 con 0x7fd018002870 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd02c0017a0 con 0x7fd018002870 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 10 monclient: dump: 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1029867988 0 0) 0x7fd0180065d0 con 0x7fd018002870 2024-09-16T05:40:44.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.180+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd01800a160 con 0x7fd018002870 2024-09-16T05:40:44.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.184+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd02c001ac0 con 0x7fd018002870 2024-09-16T05:40:44.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.184+0000 7fd0357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:44.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.184+0000 7fd0357fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:44.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.184+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 >> v1:172.21.15.63:6789/0 conn(0x7fd018002870 legacy=0x7fd018004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.184+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 >> v1:172.21.15.63:6789/0 conn(0x7fd018002870 legacy=0x7fd018004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:44.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.188+0000 7fd03ec5e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:40:44.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.188+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd018009e50 con 0x7fd018009110 2024-09-16T05:40:44.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.188+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1093149906 0 0) 0x7fd018009e50 con 0x7fd018009110 2024-09-16T05:40:44.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.188+0000 7fd0357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.188+0000 7fd0357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.188+0000 7fd0357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.188+0000 7fd0357fa640 10 monclient(hunting): my global_id is 114211 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.188+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0180065d0 con 0x7fd018009110 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 475369162 0 0) 0x7fd0180065d0 con 0x7fd018009110 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd018009e50 con 0x7fd018009110 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd02c0037a0 con 0x7fd018009110 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 10 monclient: dump: 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2965983418 0 0) 0x7fd018009e50 con 0x7fd018009110 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd01800c3e0 con 0x7fd018009110 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4119 (unknown 0 0 2279146299) 0x7fd02c005a80 con 0x7fd018009110 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 >> v1:172.21.15.63:6789/0 conn(0x7fd018009110 legacy=0x7fd0180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd0357fa640 1 -- 172.21.15.6:0/1221981132 >> v1:172.21.15.63:6789/0 conn(0x7fd018009110 legacy=0x7fd0180095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 30, 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 18755, 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:40:44.205 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:44.206 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:44.207 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:40:44.208 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:40:44.209 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:40:44.210 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:44.211 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.212 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:40:44.213 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.214 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.215 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:40:44.216 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:40:44.217 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:40:44.217 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:40:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd024036aa0 legacy=0x7fd024038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03ec5e640 10 monclient: shutdown 2024-09-16T05:40:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03ec5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 >> v1:172.21.15.6:6790/0 conn(0x7fd0380a1a00 legacy=0x7fd03813b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03d1fb640 1 -- 172.21.15.6:0/1221981132 reap_dead start 2024-09-16T05:40:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03d1fb640 1 -- 172.21.15.6:0/1221981132 reap_dead start 2024-09-16T05:40:44.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 shutdown_connections 2024-09-16T05:40:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 >> 172.21.15.6:0/1221981132 conn(0x7fd0380a58a0 msgr2=0x7fd03806d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 shutdown_connections 2024-09-16T05:40:44.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:40:44.192+0000 7fd03ec5e640 1 -- 172.21.15.6:0/1221981132 wait complete. 2024-09-16T05:40:44.219 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:40:44.219 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:40:44.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.693+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 removed item id -73 name 'h19' from crush map v237) ==== 120+0+0 (unknown 2897729517 0 0) 0x7f5420015230 con 0x7f542806dd30 2024-09-16T05:40:44.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.693+0000 7f540f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T05:40:44.699 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.693+0000 7f540f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -73 name 'h19' from crush map 2024-09-16T05:40:44.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.801+0000 7f542d625640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T05:40:44.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.801+0000 7f542d625640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T05:40:44.805 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.801+0000 7f542d625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:44.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.801+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) -- 0x7f5408002220 con 0x7f542806dd30 2024-09-16T05:40:44.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.801+0000 7f540f7fe640 1 -- 172.21.15.63:0/790154632 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 device 'h19' does not appear in the crush map v237) ==== 120+0+0 (unknown 3810141661 0 0) 0x7f542001dde0 con 0x7f542806dd30 2024-09-16T05:40:44.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.801+0000 7f540f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T05:40:44.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.801+0000 7f540f7fe640 10 monclient: _finish_command 3 = system:0 device 'h19' does not appear in the crush map 2024-09-16T05:40:44.807 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h19' does not appear in the crush map 2024-09-16T05:40:44.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.805+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5414036a70 legacy=0x7f5414038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.805+0000 7f542d625640 10 monclient: shutdown 2024-09-16T05:40:44.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.805+0000 7f542d625640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:44.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.805+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 >> v1:172.21.15.63:6789/0 conn(0x7f542806dd30 legacy=0x7f5428133920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.805+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 shutdown_connections 2024-09-16T05:40:44.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.805+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 >> 172.21.15.63:0/790154632 conn(0x7f54280a1c90 msgr2=0x7f54280a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:44.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.805+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 shutdown_connections 2024-09-16T05:40:44.810 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.805+0000 7f542d625640 1 -- 172.21.15.63:0/790154632 wait complete. 2024-09-16T05:40:44.824 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:44.824 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h20 2024-09-16T05:40:44.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.937+0000 7fd06c36c640 10 monclient: get_monmap_and_config 2024-09-16T05:40:44.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.937+0000 7fd06c36c640 10 monclient: build_initial_monmap 2024-09-16T05:40:44.943 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.937+0000 7fd06c36c640 10 monclient: monmap: 2024-09-16T05:40:44.943 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:44.943 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.943 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:44.943632+0000 2024-09-16T05:40:44.944 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:44.943632+0000 2024-09-16T05:40:44.944 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:44.944 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:44.944 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:44.944 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:44.944 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:44.944 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 Processor -- start 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- start start 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient: init 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient(hunting): picked mon.noname-a con 0x7fd0640a47e0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient(hunting): picked mon.noname-b con 0x7fd06406e570 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient(hunting): picked mon.noname-c con 0x7fd064072a90 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:44.947 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06406ece0 con 0x7fd0640a47e0 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06406ce20 con 0x7fd064072a90 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd064071aa0 con 0x7fd06406e570 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient(hunting): authenticate will time out at 1301.721074s 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06a108640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fd0640a47e0 legacy=0x7fd0640a4bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06a909640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd064072a90 0x7fd06413d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56082/0 (socket says 172.21.15.63:56082) 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06a909640 1 -- 172.21.15.63:0/2267159705 learned_addr learned my addr 172.21.15.63:0/2267159705 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524341687 0 0) 0x7fd06406ce20 con 0x7fd064072a90 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.948 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient(hunting): my global_id is 114214 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd050003530 con 0x7fd064072a90 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3853940267 0 0) 0x7fd050003530 con 0x7fd064072a90 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 >> v1:172.21.15.6:6790/0 conn(0x7fd06406e570 legacy=0x7fd064139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 >> v1:172.21.15.6:6789/0 conn(0x7fd0640a47e0 legacy=0x7fd0640a4bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: _finish_hunting 0 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 monclient: found mon.noname-c 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd064075860 con 0x7fd064072a90 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: _finish_auth 0 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: _check_auth_tickets 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:44.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd05c0019a0 con 0x7fd064072a90 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 5 monclient: authenticate success, global_id 114214 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: dump: 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.950 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd05c001d00 con 0x7fd064072a90 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 1 -- 172.21.15.63:0/2267159705 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd05c004c30 con 0x7fd064072a90 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd069106640 10 monclient: dump: 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.951 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 4 set_mon_vals no callback set 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient: shutdown 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- 172.21.15.63:0/2267159705 >> v1:172.21.15.63:6789/0 conn(0x7fd064072a90 legacy=0x7fd06413d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- 172.21.15.63:0/2267159705 shutdown_connections 2024-09-16T05:40:44.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- 172.21.15.63:0/2267159705 >> 172.21.15.63:0/2267159705 conn(0x7fd0640a2b80 msgr2=0x7fd06406bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- 172.21.15.63:0/2267159705 shutdown_connections 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- 172.21.15.63:0/2267159705 wait complete. 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient: build_initial_monmap 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient: monmap: 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:44.953 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:44.954 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:44.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 Processor -- start 2024-09-16T05:40:44.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 1 -- start start 2024-09-16T05:40:44.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.941+0000 7fd06c36c640 10 monclient: init 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient(hunting): picked mon.noname-c con 0x7fd06406e570 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient(hunting): picked mon.noname-a con 0x7fd064072a90 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient(hunting): picked mon.noname-b con 0x7fd0640a47e0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06413b730 con 0x7fd064072a90 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd064138230 con 0x7fd0640a47e0 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd064132890 con 0x7fd06406e570 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient(hunting): authenticate will time out at 1301.724298s 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06a909640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd0640a47e0 0x7fd0641d1840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56084/0 (socket says 172.21.15.63:56084) 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd069907640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fd064072a90 legacy=0x7fd064137b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:44.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06a909640 1 -- 172.21.15.63:0/3834647685 learned_addr learned my addr 172.21.15.63:0/3834647685 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4177770892 0 0) 0x7fd064138230 con 0x7fd0640a47e0 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): my global_id is 114217 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd048003530 con 0x7fd0640a47e0 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3348638391 0 0) 0x7fd048003530 con 0x7fd0640a47e0 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd064138230 con 0x7fd0640a47e0 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd05c0017c0 con 0x7fd0640a47e0 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:44.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): dump: 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017010350 0 0) 0x7fd064132890 con 0x7fd06406e570 2024-09-16T05:40:44.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient(hunting): my global_id is 6371 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd048003530 con 0x7fd06406e570 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1655222108 0 0) 0x7fd064138230 con 0x7fd0640a47e0 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 >> v1:172.21.15.6:6790/0 conn(0x7fd06406e570 legacy=0x7fd064137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 >> v1:172.21.15.6:6789/0 conn(0x7fd064072a90 legacy=0x7fd064137b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 monclient: found mon.b 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd064073550 con 0x7fd0640a47e0 2024-09-16T05:40:44.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient: _finish_auth 0 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 5 monclient: authenticate success, global_id 114217 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient: _renew_subs 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd064139020 con 0x7fd0640a47e0 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient: _renew_subs 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd06413b310 con 0x7fd0640a47e0 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd05c001d00 con 0x7fd0640a47e0 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd05c0051e0 con 0x7fd0640a47e0 2024-09-16T05:40:44.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 10 monclient: dump: 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:44.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd068905640 4 set_mon_vals no callback set 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd05c00ff80 con 0x7fd0640a47e0 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(237..237 src has 20..237) ==== 10124+0+0 (unknown 2760940238 0 0) 0x7fd05c046050 con 0x7fd0640a47e0 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd04c005090 con 0x7fd0640a47e0 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.945+0000 7fd06a108640 10 monclient: get_auth_request con 0x7fd048036a80 auth_method 0 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.949+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd05c015a50 con 0x7fd0640a47e0 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.949+0000 7fd05affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:44.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:44.949+0000 7fd05affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:45.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.117+0000 7fd06c36c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T05:40:45.122 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.117+0000 7fd06c36c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T05:40:45.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.117+0000 7fd06c36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:45.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.117+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) -- 0x7fd04c005380 con 0x7fd0640a47e0 2024-09-16T05:40:45.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.697+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 removed item id -76 name 'h20' from crush map v238) ==== 120+0+0 (unknown 1161701118 0 0) 0x7fd05c015350 con 0x7fd0640a47e0 2024-09-16T05:40:45.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.697+0000 7fd05affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T05:40:45.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.697+0000 7fd05affd640 10 monclient: _finish_command 2 = system:0 removed item id -76 name 'h20' from crush map 2024-09-16T05:40:45.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.801+0000 7fd06c36c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T05:40:45.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.801+0000 7fd06c36c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T05:40:45.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.801+0000 7fd06c36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:45.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.801+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) -- 0x7fd04c002220 con 0x7fd0640a47e0 2024-09-16T05:40:45.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.801+0000 7fd05affd640 1 -- 172.21.15.63:0/3834647685 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 device 'h20' does not appear in the crush map v238) ==== 120+0+0 (unknown 57236484 0 0) 0x7fd05c0136b0 con 0x7fd0640a47e0 2024-09-16T05:40:45.806 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.801+0000 7fd05affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T05:40:45.807 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.801+0000 7fd05affd640 10 monclient: _finish_command 3 = system:0 device 'h20' does not appear in the crush map 2024-09-16T05:40:45.807 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h20' does not appear in the crush map 2024-09-16T05:40:45.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.805+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd048036a80 legacy=0x7fd048038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:45.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.805+0000 7fd06c36c640 10 monclient: shutdown 2024-09-16T05:40:45.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.805+0000 7fd06c36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:45.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.805+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 >> v1:172.21.15.63:6789/0 conn(0x7fd0640a47e0 legacy=0x7fd0641d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:45.808 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.805+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 shutdown_connections 2024-09-16T05:40:45.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.805+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 >> 172.21.15.63:0/3834647685 conn(0x7fd0640a2b80 msgr2=0x7fd0640a1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:45.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.805+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 shutdown_connections 2024-09-16T05:40:45.809 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.805+0000 7fd06c36c640 1 -- 172.21.15.63:0/3834647685 wait complete. 2024-09-16T05:40:45.822 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:45.822 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h21 2024-09-16T05:40:45.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient: get_monmap_and_config 2024-09-16T05:40:45.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient: build_initial_monmap 2024-09-16T05:40:45.942 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient: monmap: 2024-09-16T05:40:45.942 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:45.942 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:45.942 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:45.942344+0000 2024-09-16T05:40:45.943 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:45.942344+0000 2024-09-16T05:40:45.943 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:45.943 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:45.943 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:45.943 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:45.943 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:45.943 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:45.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 1 Processor -- start 2024-09-16T05:40:45.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 1 -- start start 2024-09-16T05:40:45.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient: init 2024-09-16T05:40:45.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:45.949 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient(hunting): picked mon.noname-c con 0x7f01f809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient(hunting): picked mon.noname-a con 0x7f01f806dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient(hunting): picked mon.noname-b con 0x7f01f80a0210 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f809f800 con 0x7f01f806dd30 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f80a0ab0 con 0x7f01f809ac90 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f80a0d90 con 0x7f01f80a0210 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01fdaf4640 10 monclient(hunting): authenticate will time out at 1302.719701s 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01f77fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f01f809ac90 0x7f01f809b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56100/0 (socket says 172.21.15.63:56100) 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01f6ffd640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f01f806dd30 legacy=0x7f01f81399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.937+0000 7f01f77fe640 1 -- 172.21.15.63:0/2054781773 learned_addr learned my addr 172.21.15.63:0/2054781773 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:45.950 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544212651 0 0) 0x7f01f80a0ab0 con 0x7f01f809ac90 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient(hunting): my global_id is 114220 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e0003530 con 0x7f01f809ac90 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2839952466 0 0) 0x7f01e0003530 con 0x7f01f809ac90 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 >> v1:172.21.15.6:6790/0 conn(0x7f01f80a0210 legacy=0x7f01f813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 >> v1:172.21.15.6:6789/0 conn(0x7f01f806dd30 legacy=0x7f01f81399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 monclient: found mon.noname-c 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:45.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f809e770 con 0x7f01f809ac90 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01e8001620 con 0x7f01f809ac90 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: dump: 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:45.952 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 5 monclient: authenticate success, global_id 114220 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01e8001880 con 0x7f01f809ac90 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 1 -- 172.21.15.63:0/2054781773 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01e8001cd0 con 0x7f01f809ac90 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:45.953 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f67fc640 10 monclient: dump: 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:45.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 4 set_mon_vals no callback set 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: shutdown 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- 172.21.15.63:0/2054781773 >> v1:172.21.15.63:6789/0 conn(0x7f01f809ac90 legacy=0x7f01f809b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- 172.21.15.63:0/2054781773 shutdown_connections 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- 172.21.15.63:0/2054781773 >> 172.21.15.63:0/2054781773 conn(0x7f01f80a1c90 msgr2=0x7f01f80a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- 172.21.15.63:0/2054781773 shutdown_connections 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- 172.21.15.63:0/2054781773 wait complete. 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: build_initial_monmap 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: monmap: 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:45.955 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 Processor -- start 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- start start 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: init 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient(hunting): picked mon.noname-c con 0x7f01f806dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient(hunting): picked mon.noname-b con 0x7f01f809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:45.956 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient(hunting): picked mon.noname-a con 0x7f01f80a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f809d7d0 con 0x7f01f80a0210 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f809dfd0 con 0x7f01f809ac90 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01f8122c40 con 0x7f01f806dd30 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient(hunting): authenticate will time out at 1302.722622s 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f6ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f01f809ac90 0x7f01f81d0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56106/0 (socket says 172.21.15.63:56106) 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f7fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f01f80a0210 legacy=0x7f01f81d3970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01f6ffd640 1 -- 172.21.15.63:0/3639083794 learned_addr learned my addr 172.21.15.63:0/3639083794 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3372526981 0 0) 0x7f01f809dfd0 con 0x7f01f809ac90 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:45.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): my global_id is 114223 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e4003530 con 0x7f01f809ac90 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2676667709 0 0) 0x7f01e4003530 con 0x7f01f809ac90 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01f809dfd0 con 0x7f01f809ac90 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01ec0044d0 con 0x7f01f809ac90 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): dump: 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:45.958 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572170996 0 0) 0x7f01f8122c40 con 0x7f01f806dd30 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient(hunting): my global_id is 6377 2024-09-16T05:40:45.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e4003530 con 0x7f01f806dd30 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3335949074 0 0) 0x7f01f809dfd0 con 0x7f01f809ac90 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 >> v1:172.21.15.6:6790/0 conn(0x7f01f806dd30 legacy=0x7f01f8133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 >> v1:172.21.15.6:6789/0 conn(0x7f01f80a0210 legacy=0x7f01f81d3970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 monclient: found mon.b 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f8122f20 con 0x7f01f809ac90 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient: _finish_auth 0 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 5 monclient: authenticate success, global_id 114223 2024-09-16T05:40:45.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: _renew_subs 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01f8121d50 con 0x7f01f809ac90 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: _renew_subs 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.941+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f01f81343e0 con 0x7f01f809ac90 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01ec004730 con 0x7f01f809ac90 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01dffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01ec004a80 con 0x7f01f809ac90 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01dffff640 10 monclient: dump: 2024-09-16T05:40:45.961 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01fdaf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01f5ffb640 4 set_mon_vals no callback set 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f01ec004c80 con 0x7f01f809ac90 2024-09-16T05:40:45.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(238..238 src has 20..238) ==== 9911+0+0 (unknown 3031325585 0 0) 0x7f01ec045cd0 con 0x7f01f809ac90 2024-09-16T05:40:45.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01fdaf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:45.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01fdaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:45.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01f81bbdc0 con 0x7f01f809ac90 2024-09-16T05:40:45.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.945+0000 7f01f77fe640 10 monclient: get_auth_request con 0x7f01e4036be0 auth_method 0 2024-09-16T05:40:45.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.949+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f01ec0157c0 con 0x7f01f809ac90 2024-09-16T05:40:45.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.949+0000 7f01dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:45.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:45.949+0000 7f01dffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:46.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.117+0000 7f01fdaf4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T05:40:46.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.117+0000 7f01fdaf4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T05:40:46.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.117+0000 7f01fdaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:46.121 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.117+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) -- 0x7f01f8073c40 con 0x7f01f809ac90 2024-09-16T05:40:46.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.713+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 removed item id -79 name 'h21' from crush map v239) ==== 120+0+0 (unknown 268673606 0 0) 0x7f01ec019470 con 0x7f01f809ac90 2024-09-16T05:40:46.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.713+0000 7f01dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T05:40:46.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.713+0000 7f01dffff640 10 monclient: _finish_command 2 = system:0 removed item id -79 name 'h21' from crush map 2024-09-16T05:40:46.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.821+0000 7f01fdaf4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T05:40:46.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.821+0000 7f01fdaf4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T05:40:46.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.821+0000 7f01fdaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:46.826 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.821+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) -- 0x7f01f809b780 con 0x7f01f809ac90 2024-09-16T05:40:46.827 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.821+0000 7f01dffff640 1 -- 172.21.15.63:0/3639083794 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 device 'h21' does not appear in the crush map v239) ==== 120+0+0 (unknown 3075568811 0 0) 0x7f01ec01dd30 con 0x7f01f809ac90 2024-09-16T05:40:46.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.821+0000 7f01dffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T05:40:46.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.821+0000 7f01dffff640 10 monclient: _finish_command 3 = system:0 device 'h21' does not appear in the crush map 2024-09-16T05:40:46.828 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h21' does not appear in the crush map 2024-09-16T05:40:46.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.825+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 >> v1:172.21.15.63:6812/3643207211 conn(0x7f01e4036be0 legacy=0x7f01e40390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:46.829 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.825+0000 7f01fdaf4640 10 monclient: shutdown 2024-09-16T05:40:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.825+0000 7f01fdaf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.825+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 >> v1:172.21.15.63:6789/0 conn(0x7f01f809ac90 legacy=0x7f01f81d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.825+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 shutdown_connections 2024-09-16T05:40:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.825+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 >> 172.21.15.63:0/3639083794 conn(0x7f01f80a1c90 msgr2=0x7f01f8134fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:46.830 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.825+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 shutdown_connections 2024-09-16T05:40:46.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.825+0000 7f01fdaf4640 1 -- 172.21.15.63:0/3639083794 wait complete. 2024-09-16T05:40:46.843 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:46.843 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h22 2024-09-16T05:40:46.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.949+0000 7f9df0100640 10 monclient: get_monmap_and_config 2024-09-16T05:40:46.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.949+0000 7f9df0100640 10 monclient: build_initial_monmap 2024-09-16T05:40:46.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.949+0000 7f9df0100640 10 monclient: monmap: 2024-09-16T05:40:46.955 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:46.955 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:46.956 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:46.955624+0000 2024-09-16T05:40:46.956 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:46.955624+0000 2024-09-16T05:40:46.956 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:46.956 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:46.956 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:46.956 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:46.956 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:46.956 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:46.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 Processor -- start 2024-09-16T05:40:46.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- start start 2024-09-16T05:40:46.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: init 2024-09-16T05:40:46.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:46.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:46.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient(hunting): picked mon.noname-b con 0x7f9de80a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:46.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient(hunting): picked mon.noname-a con 0x7f9de80a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:46.963 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient(hunting): picked mon.noname-c con 0x7f9de80a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de80a4fd0 con 0x7f9de80a08a0 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de80a5b40 con 0x7f9de80a4790 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de80a1a70 con 0x7f9de80a3ca0 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient(hunting): authenticate will time out at 1303.733083s 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dee69d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f9de80a4790 0x7f9de813f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56108/0 (socket says 172.21.15.63:56108) 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dee69d640 1 -- 172.21.15.63:0/1937066 learned_addr learned my addr 172.21.15.63:0/1937066 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9ded69b640 1 -- 172.21.15.63:0/1937066 >> v1:172.21.15.6:6789/0 conn(0x7f9de80a08a0 legacy=0x7f9de813beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 310708854 0 0) 0x7f9de80a5b40 con 0x7f9de80a4790 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:46.964 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient(hunting): my global_id is 114226 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dd4003530 con 0x7f9de80a4790 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 307672414 0 0) 0x7f9dd4003530 con 0x7f9de80a4790 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 >> v1:172.21.15.6:6790/0 conn(0x7f9de80a3ca0 legacy=0x7f9de80a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 >> v1:172.21.15.6:6789/0 conn(0x7f9de80a08a0 legacy=0x7f9de813beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 monclient: found mon.noname-c 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de80a53e0 con 0x7f9de80a4790 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: _finish_auth 0 2024-09-16T05:40:46.965 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9de00019a0 con 0x7f9de80a4790 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 5 monclient: authenticate success, global_id 114226 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: dump: 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:46.966 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9de0001c90 con 0x7f9de80a4790 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 1 -- 172.21.15.63:0/1937066 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9de0002060 con 0x7f9de80a4790 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9dece9a640 10 monclient: dump: 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:46.967 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 4 set_mon_vals no callback set 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: shutdown 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- 172.21.15.63:0/1937066 >> v1:172.21.15.63:6789/0 conn(0x7f9de80a4790 legacy=0x7f9de813f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- 172.21.15.63:0/1937066 shutdown_connections 2024-09-16T05:40:46.968 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- 172.21.15.63:0/1937066 >> 172.21.15.63:0/1937066 conn(0x7f9de80a67c0 msgr2=0x7f9de8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- 172.21.15.63:0/1937066 shutdown_connections 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- 172.21.15.63:0/1937066 wait complete. 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: build_initial_monmap 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: monmap: 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:46.969 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 Processor -- start 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 1 -- start start 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: init 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.953+0000 7f9df0100640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient(hunting): picked mon.noname-a con 0x7f9de80a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient(hunting): picked mon.noname-b con 0x7f9de80a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient(hunting): picked mon.noname-c con 0x7f9de80a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de8126210 con 0x7f9de80a08a0 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de8125b50 con 0x7f9de80a3ca0 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de81276c0 con 0x7f9de80a4790 2024-09-16T05:40:46.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient(hunting): authenticate will time out at 1303.736104s 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dede9c640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f9de80a08a0 legacy=0x7f9de81cb070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9ded69b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f9de80a3ca0 0x7f9de8136fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56124/0 (socket says 172.21.15.63:56124) 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9ded69b640 1 -- 172.21.15.63:0/1290672965 learned_addr learned my addr 172.21.15.63:0/1290672965 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553764292 0 0) 0x7f9de8125b50 con 0x7f9de80a3ca0 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient(hunting): my global_id is 114229 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dcc003530 con 0x7f9de80a3ca0 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 640863527 0 0) 0x7f9dcc003530 con 0x7f9de80a3ca0 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9de8125b50 con 0x7f9de80a3ca0 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9dd8004190 con 0x7f9de80a3ca0 2024-09-16T05:40:46.971 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient(hunting): dump: 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:46.972 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 10210849 0 0) 0x7f9de8125b50 con 0x7f9de80a3ca0 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 >> v1:172.21.15.6:6790/0 conn(0x7f9de80a4790 legacy=0x7f9de81d33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 >> v1:172.21.15.6:6789/0 conn(0x7f9de80a08a0 legacy=0x7f9de81cb070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 monclient: found mon.b 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:46.973 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de80a88d0 con 0x7f9de80a3ca0 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 5 monclient: authenticate success, global_id 114229 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient: _renew_subs 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9de81273e0 con 0x7f9de80a3ca0 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient: _renew_subs 2024-09-16T05:40:46.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9de81376c0 con 0x7f9de80a3ca0 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9dd80044c0 con 0x7f9de80a3ca0 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9dd8004810 con 0x7f9de80a3ca0 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9ddffff640 4 set_mon_vals no callback set 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 10 monclient: dump: 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:46.975 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9dd80049b0 con 0x7f9de80a3ca0 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd0005090 con 0x7f9de80a3ca0 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(239..239 src has 20..239) ==== 9698+0+0 (unknown 2717441714 0 0) 0x7f9dd8045a60 con 0x7f9de80a3ca0 2024-09-16T05:40:46.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.957+0000 7f9dede9c640 10 monclient: get_auth_request con 0x7f9dcc036a40 auth_method 0 2024-09-16T05:40:46.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.961+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9dd8015490 con 0x7f9de80a3ca0 2024-09-16T05:40:46.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.961+0000 7f9dde7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:46.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:46.961+0000 7f9dde7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:47.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.125+0000 7f9df0100640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T05:40:47.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.125+0000 7f9df0100640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T05:40:47.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.125+0000 7f9df0100640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:47.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.125+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) -- 0x7f9dd0005380 con 0x7f9de80a3ca0 2024-09-16T05:40:47.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.721+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 removed item id -82 name 'h22' from crush map v240) ==== 120+0+0 (unknown 4114174357 0 0) 0x7f9dd8019140 con 0x7f9de80a3ca0 2024-09-16T05:40:47.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.721+0000 7f9dde7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T05:40:47.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.721+0000 7f9dde7fc640 10 monclient: _finish_command 2 = system:0 removed item id -82 name 'h22' from crush map 2024-09-16T05:40:47.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.825+0000 7f9df0100640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T05:40:47.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.825+0000 7f9df0100640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T05:40:47.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.825+0000 7f9df0100640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:47.831 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.825+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) -- 0x7f9dd0002220 con 0x7f9de80a3ca0 2024-09-16T05:40:47.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.825+0000 7f9dde7fc640 1 -- 172.21.15.63:0/1290672965 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v240) ==== 120+0+0 (unknown 3765826643 0 0) 0x7f9dd801da00 con 0x7f9de80a3ca0 2024-09-16T05:40:47.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.825+0000 7f9dde7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T05:40:47.832 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.825+0000 7f9dde7fc640 10 monclient: _finish_command 3 = system:0 device 'h22' does not appear in the crush map 2024-09-16T05:40:47.832 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h22' does not appear in the crush map 2024-09-16T05:40:47.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.829+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 >> v1:172.21.15.63:6812/3643207211 conn(0x7f9dcc036a40 legacy=0x7f9dcc038f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:47.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.829+0000 7f9df0100640 10 monclient: shutdown 2024-09-16T05:40:47.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.829+0000 7f9df0100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:47.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.829+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 >> v1:172.21.15.63:6789/0 conn(0x7f9de80a3ca0 legacy=0x7f9de8136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:47.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.829+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 shutdown_connections 2024-09-16T05:40:47.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.829+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 >> 172.21.15.63:0/1290672965 conn(0x7f9de80a67c0 msgr2=0x7f9de80a2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:47.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.829+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 shutdown_connections 2024-09-16T05:40:47.834 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.829+0000 7f9df0100640 1 -- 172.21.15.63:0/1290672965 wait complete. 2024-09-16T05:40:47.849 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:47.849 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h23 2024-09-16T05:40:47.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.965+0000 7fdd24aab640 10 monclient: get_monmap_and_config 2024-09-16T05:40:47.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.965+0000 7fdd24aab640 10 monclient: build_initial_monmap 2024-09-16T05:40:47.970 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.965+0000 7fdd24aab640 10 monclient: monmap: 2024-09-16T05:40:47.970 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:47.970 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:47.970 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:47.970402+0000 2024-09-16T05:40:47.970 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:47.970402+0000 2024-09-16T05:40:47.971 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:47.971 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:47.971 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:47.971 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:47.971 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:47.971 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:47.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.965+0000 7fdd24aab640 1 Processor -- start 2024-09-16T05:40:47.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.965+0000 7fdd24aab640 1 -- start start 2024-09-16T05:40:47.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.965+0000 7fdd24aab640 10 monclient: init 2024-09-16T05:40:47.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.965+0000 7fdd24aab640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:47.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): picked mon.noname-c con 0x7fdd200a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): picked mon.noname-b con 0x7fdd200a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): picked mon.noname-a con 0x7fdd200a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd200a4fd0 con 0x7fdd200a4790 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd200a5b40 con 0x7fdd200a3ca0 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd200a1a70 con 0x7fdd200a08a0 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): authenticate will time out at 1304.748140s 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1ed9d640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fdd200a4790 legacy=0x7fdd2013f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1e59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fdd200a3ca0 0x7fdd200a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56138/0 (socket says 172.21.15.63:56138) 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1e59c640 1 -- 172.21.15.63:0/802830686 learned_addr learned my addr 172.21.15.63:0/802830686 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714082109 0 0) 0x7fdd200a5b40 con 0x7fdd200a3ca0 2024-09-16T05:40:47.975 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient(hunting): my global_id is 114232 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd0c003530 con 0x7fdd200a3ca0 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1435691467 0 0) 0x7fdd0c003530 con 0x7fdd200a3ca0 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 >> v1:172.21.15.6:6790/0 conn(0x7fdd200a08a0 legacy=0x7fdd2013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 >> v1:172.21.15.6:6789/0 conn(0x7fdd200a4790 legacy=0x7fdd2013f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:40:47.976 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 monclient: found mon.noname-c 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd200a53e0 con 0x7fdd200a3ca0 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: _finish_auth 0 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdd10001620 con 0x7fdd200a3ca0 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: dump: 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:47.977 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 5 monclient: authenticate success, global_id 114232 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd10001910 con 0x7fdd200a3ca0 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 1 -- 172.21.15.63:0/802830686 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdd10001d20 con 0x7fdd200a3ca0 2024-09-16T05:40:47.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1dd9b640 10 monclient: dump: 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:47.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:47.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 4 set_mon_vals no callback set 2024-09-16T05:40:47.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient: shutdown 2024-09-16T05:40:47.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:47.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- 172.21.15.63:0/802830686 >> v1:172.21.15.63:6789/0 conn(0x7fdd200a3ca0 legacy=0x7fdd200a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:47.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- 172.21.15.63:0/802830686 shutdown_connections 2024-09-16T05:40:47.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- 172.21.15.63:0/802830686 >> 172.21.15.63:0/802830686 conn(0x7fdd200a67c0 msgr2=0x7fdd20075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:47.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- 172.21.15.63:0/802830686 shutdown_connections 2024-09-16T05:40:47.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- 172.21.15.63:0/802830686 wait complete. 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient: build_initial_monmap 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient: monmap: 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:47.981 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:47.982 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:47.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 Processor -- start 2024-09-16T05:40:47.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- start start 2024-09-16T05:40:47.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient: init 2024-09-16T05:40:47.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:47.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:47.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): picked mon.noname-c con 0x7fdd200a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:47.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): picked mon.noname-a con 0x7fdd200a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:47.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): picked mon.noname-b con 0x7fdd200a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd201277b0 con 0x7fdd200a3ca0 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd20127a90 con 0x7fdd200a4790 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd200a83f0 con 0x7fdd200a08a0 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd24aab640 10 monclient(hunting): authenticate will time out at 1304.751499s 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1ed9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fdd200a4790 0x7fdd201d3b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56150/0 (socket says 172.21.15.63:56150) 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd1ed9d640 1 -- 172.21.15.63:0/826916382 learned_addr learned my addr 172.21.15.63:0/826916382 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd17fff640 1 -- 172.21.15.63:0/826916382 >> v1:172.21.15.6:6789/0 conn(0x7fdd200a3ca0 legacy=0x7fdd201d0400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:47.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 300850446 0 0) 0x7fdd20127a90 con 0x7fdd200a4790 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd16ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd16ffd640 10 monclient(hunting): my global_id is 114235 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.969+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd00003530 con 0x7fdd200a4790 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2769844195 0 0) 0x7fdd00003530 con 0x7fdd200a4790 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd20127a90 con 0x7fdd200a4790 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdd18004450 con 0x7fdd200a4790 2024-09-16T05:40:47.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:47.988 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3996691902 0 0) 0x7fdd200a83f0 con 0x7fdd200a08a0 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient(hunting): my global_id is 6389 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd00003530 con 0x7fdd200a08a0 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3203417464 0 0) 0x7fdd20127a90 con 0x7fdd200a4790 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 >> v1:172.21.15.6:6790/0 conn(0x7fdd200a08a0 legacy=0x7fdd201370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 >> v1:172.21.15.6:6789/0 conn(0x7fdd200a3ca0 legacy=0x7fdd201d0400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:47.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:47.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 monclient: found mon.b 2024-09-16T05:40:47.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:47.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:47.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd200a8940 con 0x7fdd200a4790 2024-09-16T05:40:47.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:47.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:47.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:47.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 5 monclient: authenticate success, global_id 114235 2024-09-16T05:40:47.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 10 monclient: _renew_subs 2024-09-16T05:40:47.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:47.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd201274d0 con 0x7fdd200a4790 2024-09-16T05:40:47.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 10 monclient: _renew_subs 2024-09-16T05:40:47.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:47.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd201377b0 con 0x7fdd200a4790 2024-09-16T05:40:47.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd180046b0 con 0x7fdd200a4790 2024-09-16T05:40:47.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:47.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdd18004a00 con 0x7fdd200a4790 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd1d59a640 4 set_mon_vals no callback set 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 10 monclient: dump: 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:47.993 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdd18004ba0 con 0x7fdd200a4790 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd04005090 con 0x7fdd200a4790 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(240..240 src has 20..240) ==== 9485+0+0 (unknown 274565339 0 0) 0x7fdd18045bb0 con 0x7fdd200a4790 2024-09-16T05:40:47.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.973+0000 7fdd1e59c640 10 monclient: get_auth_request con 0x7fdd00036b20 auth_method 0 2024-09-16T05:40:47.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.977+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fdd18015710 con 0x7fdd200a4790 2024-09-16T05:40:47.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.977+0000 7fdd16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:47.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:47.977+0000 7fdd16ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:48.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.137+0000 7fdd24aab640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T05:40:48.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.137+0000 7fdd24aab640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T05:40:48.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.137+0000 7fdd24aab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:48.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.137+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) -- 0x7fdd04005380 con 0x7fdd200a4790 2024-09-16T05:40:48.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.729+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 removed item id -85 name 'h23' from crush map v241) ==== 120+0+0 (unknown 3262234622 0 0) 0x7fdd18015010 con 0x7fdd200a4790 2024-09-16T05:40:48.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.729+0000 7fdd16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T05:40:48.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.729+0000 7fdd16ffd640 10 monclient: _finish_command 2 = system:0 removed item id -85 name 'h23' from crush map 2024-09-16T05:40:48.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.833+0000 7fdd24aab640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T05:40:48.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.833+0000 7fdd24aab640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T05:40:48.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.833+0000 7fdd24aab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:48.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.833+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) -- 0x7fdd04002220 con 0x7fdd200a4790 2024-09-16T05:40:48.840 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd16ffd640 1 -- 172.21.15.63:0/826916382 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 device 'h23' does not appear in the crush map v241) ==== 120+0+0 (unknown 1414388988 0 0) 0x7fdd18019360 con 0x7fdd200a4790 2024-09-16T05:40:48.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd16ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T05:40:48.841 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd16ffd640 10 monclient: _finish_command 3 = system:0 device 'h23' does not appear in the crush map 2024-09-16T05:40:48.841 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h23' does not appear in the crush map 2024-09-16T05:40:48.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 >> v1:172.21.15.63:6812/3643207211 conn(0x7fdd00036b20 legacy=0x7fdd00038fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:48.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd24aab640 10 monclient: shutdown 2024-09-16T05:40:48.842 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd24aab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 >> v1:172.21.15.63:6789/0 conn(0x7fdd200a4790 legacy=0x7fdd201d3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 shutdown_connections 2024-09-16T05:40:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 >> 172.21.15.63:0/826916382 conn(0x7fdd200a67c0 msgr2=0x7fdd200a2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 shutdown_connections 2024-09-16T05:40:48.843 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.837+0000 7fdd24aab640 1 -- 172.21.15.63:0/826916382 wait complete. 2024-09-16T05:40:48.857 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:48.857 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h24 2024-09-16T05:40:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient: get_monmap_and_config 2024-09-16T05:40:48.978 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient: build_initial_monmap 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient: monmap: 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:48.978633+0000 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:48.978633+0000 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:48.979 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:48.980 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 1 Processor -- start 2024-09-16T05:40:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 1 -- start start 2024-09-16T05:40:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient: init 2024-09-16T05:40:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient(hunting): picked mon.noname-a con 0x7f3544136ac0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient(hunting): picked mon.noname-c con 0x7f354409fd80 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:48.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient(hunting): picked mon.noname-b con 0x7f354413d0b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3544072cb0 con 0x7f3544136ac0 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3544136670 con 0x7f354409fd80 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35440a14f0 con 0x7f354413d0b0 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354b6bf640 10 monclient(hunting): authenticate will time out at 1305.755642s 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f354945b640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f3544136ac0 legacy=0x7f3544136ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f3548c5a640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f354409fd80 0x7f354413b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56152/0 (socket says 172.21.15.63:56152) 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.973+0000 7f3548c5a640 1 -- 172.21.15.63:0/1714378680 learned_addr learned my addr 172.21.15.63:0/1714378680 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862206098 0 0) 0x7f3544136670 con 0x7f354409fd80 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:48.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient(hunting): my global_id is 114238 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3530003530 con 0x7f354409fd80 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2370496049 0 0) 0x7f3530003530 con 0x7f354409fd80 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 >> v1:172.21.15.6:6790/0 conn(0x7f354413d0b0 legacy=0x7f354413f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 >> v1:172.21.15.6:6789/0 conn(0x7f3544136ac0 legacy=0x7f3544136ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 monclient: found mon.noname-c 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35440a0630 con 0x7f354409fd80 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: _finish_auth 0 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:48.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3534001620 con 0x7f354409fd80 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: dump: 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:48.990 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 5 monclient: authenticate success, global_id 114238 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3534001980 con 0x7f354409fd80 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 1 -- 172.21.15.63:0/1714378680 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3534001d50 con 0x7f354409fd80 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 4 set_mon_vals no callback set 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353bfff640 10 monclient: dump: 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:48.991 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:48.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: shutdown 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- 172.21.15.63:0/1714378680 >> v1:172.21.15.63:6789/0 conn(0x7f354409fd80 legacy=0x7f354413b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- 172.21.15.63:0/1714378680 shutdown_connections 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- 172.21.15.63:0/1714378680 >> 172.21.15.63:0/1714378680 conn(0x7f35440a1c90 msgr2=0x7f35440a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- 172.21.15.63:0/1714378680 shutdown_connections 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- 172.21.15.63:0/1714378680 wait complete. 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: build_initial_monmap 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: monmap: 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:48.993 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 Processor -- start 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- start start 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: init 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient(hunting): picked mon.noname-a con 0x7f354409fd80 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:48.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient(hunting): picked mon.noname-b con 0x7f3544136ac0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient(hunting): picked mon.noname-c con 0x7f354413d0b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f354411f5e0 con 0x7f354409fd80 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f354411f860 con 0x7f3544136ac0 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3544120d10 con 0x7f354413d0b0 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient(hunting): authenticate will time out at 1305.758721s 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354945b640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f354409fd80 legacy=0x7f35441352b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3548c5a640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f3544136ac0 0x7f35441d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56166/0 (socket says 172.21.15.63:56166) 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3548c5a640 1 -- 172.21.15.63:0/3337269532 learned_addr learned my addr 172.21.15.63:0/3337269532 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2552966709 0 0) 0x7f354411f860 con 0x7f3544136ac0 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:48.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient(hunting): my global_id is 114241 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3528003530 con 0x7f3544136ac0 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2971035319 0 0) 0x7f3528003530 con 0x7f3544136ac0 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f354411f860 con 0x7f3544136ac0 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35340015e0 con 0x7f3544136ac0 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient(hunting): dump: 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:48.996 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2761409565 0 0) 0x7f354411f860 con 0x7f3544136ac0 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 >> v1:172.21.15.6:6790/0 conn(0x7f354413d0b0 legacy=0x7f35441d3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 >> v1:172.21.15.6:6789/0 conn(0x7f354409fd80 legacy=0x7f35441352b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:48.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 monclient: found mon.b 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354413c7b0 con 0x7f3544136ac0 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 5 monclient: authenticate success, global_id 114241 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: _renew_subs 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3544120750 con 0x7f3544136ac0 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: _renew_subs 2024-09-16T05:40:48.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f354411e070 con 0x7f3544136ac0 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3534001980 con 0x7f3544136ac0 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3534004dd0 con 0x7f3544136ac0 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f353b7fe640 4 set_mon_vals no callback set 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.977+0000 7f3539ffb640 10 monclient: dump: 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:48.999 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.981+0000 7f354b6bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.981+0000 7f354b6bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.981+0000 7f354b6bf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.981+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3524005090 con 0x7f3544136ac0 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.981+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f353400fb70 con 0x7f3544136ac0 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.981+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(241..241 src has 20..241) ==== 9272+0+0 (unknown 619797718 0 0) 0x7f3534045850 con 0x7f3544136ac0 2024-09-16T05:40:49.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.981+0000 7f354945b640 10 monclient: get_auth_request con 0x7f3528036980 auth_method 0 2024-09-16T05:40:49.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.985+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3534015570 con 0x7f3544136ac0 2024-09-16T05:40:49.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.985+0000 7f3539ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:49.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:48.985+0000 7f3539ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:49.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.149+0000 7f354b6bf640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T05:40:49.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.153+0000 7f354b6bf640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T05:40:49.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.153+0000 7f354b6bf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:49.156 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.153+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) -- 0x7f3524005380 con 0x7f3544136ac0 2024-09-16T05:40:49.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.737+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 removed item id -88 name 'h24' from crush map v242) ==== 120+0+0 (unknown 3573090770 0 0) 0x7f3534014e70 con 0x7f3544136ac0 2024-09-16T05:40:49.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.737+0000 7f3539ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T05:40:49.744 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.737+0000 7f3539ffb640 10 monclient: _finish_command 2 = system:0 removed item id -88 name 'h24' from crush map 2024-09-16T05:40:49.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.841+0000 7f354b6bf640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T05:40:49.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.841+0000 7f354b6bf640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T05:40:49.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.841+0000 7f354b6bf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:49.848 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.841+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) -- 0x7f3524002220 con 0x7f3544136ac0 2024-09-16T05:40:49.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f3539ffb640 1 -- 172.21.15.63:0/3337269532 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 device 'h24' does not appear in the crush map v242) ==== 120+0+0 (unknown 1648107242 0 0) 0x7f353401d9c0 con 0x7f3544136ac0 2024-09-16T05:40:49.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f3539ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T05:40:49.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f3539ffb640 10 monclient: _finish_command 3 = system:0 device 'h24' does not appear in the crush map 2024-09-16T05:40:49.849 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h24' does not appear in the crush map 2024-09-16T05:40:49.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3528036980 legacy=0x7f3528038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:49.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f354b6bf640 10 monclient: shutdown 2024-09-16T05:40:49.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f354b6bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:49.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 >> v1:172.21.15.63:6789/0 conn(0x7f3544136ac0 legacy=0x7f35441d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:49.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 shutdown_connections 2024-09-16T05:40:49.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 >> 172.21.15.63:0/3337269532 conn(0x7f35440a1c90 msgr2=0x7f354409cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:49.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 shutdown_connections 2024-09-16T05:40:49.852 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.845+0000 7f354b6bf640 1 -- 172.21.15.63:0/3337269532 wait complete. 2024-09-16T05:40:49.865 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:49.866 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h25 2024-09-16T05:40:49.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient: get_monmap_and_config 2024-09-16T05:40:49.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient: build_initial_monmap 2024-09-16T05:40:49.974 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient: monmap: 2024-09-16T05:40:49.974 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:49.974 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:49.974 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:49.974442+0000 2024-09-16T05:40:49.974 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:49.974442+0000 2024-09-16T05:40:49.975 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:49.975 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:49.975 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:49.975 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:49.975 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:49.975 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:49.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 1 Processor -- start 2024-09-16T05:40:49.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 1 -- start start 2024-09-16T05:40:49.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient: init 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient(hunting): picked mon.noname-c con 0x7f53d00a1920 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient(hunting): picked mon.noname-a con 0x7f53d00a2410 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient(hunting): picked mon.noname-b con 0x7f53d013a030 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d009afa0 con 0x7f53d00a2410 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d009cc50 con 0x7f53d00a1920 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d009b3b0 con 0x7f53d013a030 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d7aaa640 10 monclient(hunting): authenticate will time out at 1306.751542s 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d5846640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f53d00a1920 0x7f53d00a1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56168/0 (socket says 172.21.15.63:56168) 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d5846640 1 -- 172.21.15.63:0/3527395450 learned_addr learned my addr 172.21.15.63:0/3527395450 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:49.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d5045640 1 -- 172.21.15.63:0/3527395450 >> v1:172.21.15.6:6789/0 conn(0x7f53d00a2410 legacy=0x7f53d0138910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2967396825 0 0) 0x7f53d009cc50 con 0x7f53d00a1920 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d4844640 10 monclient(hunting): _negotiate 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d4844640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d4844640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d4844640 10 monclient(hunting): my global_id is 114244 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.969+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53bc003530 con 0x7f53d00a1920 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1850771612 0 0) 0x7f53bc003530 con 0x7f53d00a1920 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 >> v1:172.21.15.6:6790/0 conn(0x7f53d013a030 legacy=0x7f53d013c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 >> v1:172.21.15.6:6789/0 conn(0x7f53d00a2410 legacy=0x7f53d0138910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: _finish_hunting 0 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 1 monclient: found mon.noname-c 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:49.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d009dc20 con 0x7f53d00a1920 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: _finish_auth 0 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: _check_auth_tickets 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 5 monclient: authenticate success, global_id 114244 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53c80019a0 con 0x7f53d00a1920 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: dump: 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:49.984 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53c8001c00 con 0x7f53d00a1920 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 1 -- 172.21.15.63:0/3527395450 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53c8002050 con 0x7f53d00a1920 2024-09-16T05:40:49.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d4844640 10 monclient: dump: 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:49.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 4 set_mon_vals no callback set 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: shutdown 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- 172.21.15.63:0/3527395450 >> v1:172.21.15.63:6789/0 conn(0x7f53d00a1920 legacy=0x7f53d00a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- 172.21.15.63:0/3527395450 shutdown_connections 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- 172.21.15.63:0/3527395450 >> 172.21.15.63:0/3527395450 conn(0x7f53d0075470 msgr2=0x7f53d0075880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- 172.21.15.63:0/3527395450 shutdown_connections 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- 172.21.15.63:0/3527395450 wait complete. 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: build_initial_monmap 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: monmap: 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:49.987 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 Processor -- start 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- start start 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: init 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:49.988 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient(hunting): picked mon.noname-a con 0x7f53d00a1920 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient(hunting): picked mon.noname-c con 0x7f53d00a2410 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient(hunting): picked mon.noname-b con 0x7f53d013a030 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d0120090 con 0x7f53d00a1920 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d01399d0 con 0x7f53d013a030 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d009d100 con 0x7f53d00a2410 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient(hunting): authenticate will time out at 1306.754388s 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d5846640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f53d00a1920 legacy=0x7f53d01c9440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d6047640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f53d013a030 0x7f53d01d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56170/0 (socket says 172.21.15.63:56170) 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d6047640 1 -- 172.21.15.63:0/1039646075 learned_addr learned my addr 172.21.15.63:0/1039646075 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:49.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3115798483 0 0) 0x7f53d01399d0 con 0x7f53d013a030 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): my global_id is 114247 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53b4003530 con 0x7f53d013a030 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 848439471 0 0) 0x7f53b4003530 con 0x7f53d013a030 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53d01399d0 con 0x7f53d013a030 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53cc001900 con 0x7f53d013a030 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): dump: 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:49.990 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618291190 0 0) 0x7f53d009d100 con 0x7f53d00a2410 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:49.991 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient(hunting): my global_id is 6401 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53b4003530 con 0x7f53d00a2410 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 17191219 0 0) 0x7f53d01399d0 con 0x7f53d013a030 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 >> v1:172.21.15.6:6790/0 conn(0x7f53d00a2410 legacy=0x7f53d01339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 >> v1:172.21.15.6:6789/0 conn(0x7f53d00a1920 legacy=0x7f53d01c9440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 monclient: found mon.b 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d00a4ca0 con 0x7f53d013a030 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:49.992 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 5 monclient: authenticate success, global_id 114247 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: _renew_subs 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53d011fdb0 con 0x7f53d013a030 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: _renew_subs 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f53d01340b0 con 0x7f53d013a030 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53cc001d10 con 0x7f53d013a030 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53cc003af0 con 0x7f53d013a030 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c77fe640 4 set_mon_vals no callback set 2024-09-16T05:40:49.993 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.973+0000 7f53c67fc640 10 monclient: dump: 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.977+0000 7f53d7aaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:49.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.977+0000 7f53d7aaa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:49.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.977+0000 7f53d7aaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:49.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.977+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b8005090 con 0x7f53d013a030 2024-09-16T05:40:49.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.977+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f53cc003c90 con 0x7f53d013a030 2024-09-16T05:40:49.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.977+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(242..242 src has 20..242) ==== 9059+0+0 (unknown 766994473 0 0) 0x7f53cc0457f0 con 0x7f53d013a030 2024-09-16T05:40:49.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.977+0000 7f53d5846640 10 monclient: get_auth_request con 0x7f53b4036b80 auth_method 0 2024-09-16T05:40:49.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.981+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f53cc015520 con 0x7f53d013a030 2024-09-16T05:40:49.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.981+0000 7f53c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:49.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:49.981+0000 7f53c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:50.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.145+0000 7f53d7aaa640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T05:40:50.151 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.145+0000 7f53d7aaa640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T05:40:50.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.145+0000 7f53d7aaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:50.152 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.145+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) -- 0x7f53b8005380 con 0x7f53d013a030 2024-09-16T05:40:50.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.449+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 removed item id -91 name 'h25' from crush map v243) ==== 120+0+0 (unknown 1397745227 0 0) 0x7f53cc014e20 con 0x7f53d013a030 2024-09-16T05:40:50.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.449+0000 7f53c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T05:40:50.453 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.449+0000 7f53c67fc640 10 monclient: _finish_command 2 = system:0 removed item id -91 name 'h25' from crush map 2024-09-16T05:40:50.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53d7aaa640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T05:40:50.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53d7aaa640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T05:40:50.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53d7aaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:50.561 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) -- 0x7f53b80010a0 con 0x7f53d013a030 2024-09-16T05:40:50.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53c67fc640 1 -- 172.21.15.63:0/1039646075 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v243) ==== 120+0+0 (unknown 3590600261 0 0) 0x7f53cc019170 con 0x7f53d013a030 2024-09-16T05:40:50.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53c67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T05:40:50.562 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53c67fc640 10 monclient: _finish_command 3 = system:0 device 'h25' does not appear in the crush map 2024-09-16T05:40:50.562 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h25' does not appear in the crush map 2024-09-16T05:40:50.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 >> v1:172.21.15.63:6812/3643207211 conn(0x7f53b4036b80 legacy=0x7f53b4039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:50.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53d7aaa640 10 monclient: shutdown 2024-09-16T05:40:50.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53d7aaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:50.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.557+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 >> v1:172.21.15.63:6789/0 conn(0x7f53d013a030 legacy=0x7f53d01d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:50.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.561+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 shutdown_connections 2024-09-16T05:40:50.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.561+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 >> 172.21.15.63:0/1039646075 conn(0x7f53d0075470 msgr2=0x7f53d0071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:50.564 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.561+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 shutdown_connections 2024-09-16T05:40:50.565 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.561+0000 7f53d7aaa640 1 -- 172.21.15.63:0/1039646075 wait complete. 2024-09-16T05:40:50.579 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:50.580 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h26 2024-09-16T05:40:50.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient: get_monmap_and_config 2024-09-16T05:40:50.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient: build_initial_monmap 2024-09-16T05:40:50.702 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient: monmap: 2024-09-16T05:40:50.702 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:50.702 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:50.702 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:50.702402+0000 2024-09-16T05:40:50.702 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:50.702402+0000 2024-09-16T05:40:50.702 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:50.703 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:50.703 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:50.703 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:50.703 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:50.703 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 1 Processor -- start 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 1 -- start start 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient: init 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient(hunting): picked mon.noname-a con 0x7faba40a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient(hunting): picked mon.noname-c con 0x7faba40a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient(hunting): picked mon.noname-b con 0x7faba40a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:50.704 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba40a4fd0 con 0x7faba40a3ca0 2024-09-16T05:40:50.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba40a5b40 con 0x7faba40a08a0 2024-09-16T05:40:50.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba40a1a70 con 0x7faba40a4790 2024-09-16T05:40:50.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:50.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.697+0000 7fabab514640 10 monclient(hunting): authenticate will time out at 1307.479930s 2024-09-16T05:40:50.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7faba8aaf640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7faba40a08a0 0x7faba413beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56186/0 (socket says 172.21.15.63:56186) 2024-09-16T05:40:50.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7faba92b0640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7faba40a3ca0 legacy=0x7faba40a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:50.705 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7faba8aaf640 1 -- 172.21.15.63:0/1982269732 learned_addr learned my addr 172.21.15.63:0/1982269732 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:50.709 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1124847725 0 0) 0x7faba40a5b40 con 0x7faba40a08a0 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient(hunting): my global_id is 114250 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab90003530 con 0x7faba40a08a0 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3658438993 0 0) 0x7fab90003530 con 0x7faba40a08a0 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 >> v1:172.21.15.6:6790/0 conn(0x7faba40a4790 legacy=0x7faba413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 >> v1:172.21.15.6:6789/0 conn(0x7faba40a3ca0 legacy=0x7faba40a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 monclient: found mon.noname-c 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:50.710 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba40a53e0 con 0x7faba40a08a0 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: _finish_auth 0 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab94001620 con 0x7faba40a08a0 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 5 monclient: authenticate success, global_id 114250 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: dump: 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:50.711 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab94001910 con 0x7faba40a08a0 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 1 -- 172.21.15.63:0/1982269732 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab94001ce0 con 0x7faba40a08a0 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:50.712 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 4 set_mon_vals no callback set 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab9bfff640 10 monclient: dump: 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:50.713 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient: shutdown 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- 172.21.15.63:0/1982269732 >> v1:172.21.15.63:6789/0 conn(0x7faba40a08a0 legacy=0x7faba413beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- 172.21.15.63:0/1982269732 shutdown_connections 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- 172.21.15.63:0/1982269732 >> 172.21.15.63:0/1982269732 conn(0x7faba40a67c0 msgr2=0x7faba4075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- 172.21.15.63:0/1982269732 shutdown_connections 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- 172.21.15.63:0/1982269732 wait complete. 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient: build_initial_monmap 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient: monmap: 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:50.714 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 Processor -- start 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- start start 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient: init 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient(hunting): picked mon.noname-b con 0x7faba40a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient(hunting): picked mon.noname-c con 0x7faba40a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:50.715 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient(hunting): picked mon.noname-a con 0x7faba40a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba4125700 con 0x7faba40a4790 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba4122b50 con 0x7faba40a08a0 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faba4124530 con 0x7faba40a3ca0 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7faba92b0640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7faba40a08a0 0x7faba4121d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56190/0 (socket says 172.21.15.63:56190) 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7faba92b0640 1 -- 172.21.15.63:0/30304277 learned_addr learned my addr 172.21.15.63:0/30304277 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient(hunting): authenticate will time out at 1307.483127s 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7faba9ab1640 1 -- 172.21.15.63:0/30304277 >> v1:172.21.15.6:6789/0 conn(0x7faba40a4790 legacy=0x7faba41d3940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 304279098 0 0) 0x7faba4122b50 con 0x7faba40a08a0 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:50.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): my global_id is 114253 2024-09-16T05:40:50.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab88003530 con 0x7faba40a08a0 2024-09-16T05:40:50.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2586685486 0 0) 0x7fab88003530 con 0x7faba40a08a0 2024-09-16T05:40:50.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faba4122b50 con 0x7faba40a08a0 2024-09-16T05:40:50.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faba0004190 con 0x7faba40a08a0 2024-09-16T05:40:50.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:50.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:50.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): dump: 2024-09-16T05:40:50.717 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:50.718 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1832721760 0 0) 0x7faba4124530 con 0x7faba40a3ca0 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient(hunting): my global_id is 6407 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab88003530 con 0x7faba40a3ca0 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2409057582 0 0) 0x7faba4122b50 con 0x7faba40a08a0 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 >> v1:172.21.15.6:6790/0 conn(0x7faba40a3ca0 legacy=0x7faba4122440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 >> v1:172.21.15.6:6789/0 conn(0x7faba40a4790 legacy=0x7faba41d3940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 monclient: found mon.b 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:50.719 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba41242b0 con 0x7faba40a08a0 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fab99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 5 monclient: authenticate success, global_id 114253 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.701+0000 7fabab514640 10 monclient: _renew_subs 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faba40a86d0 con 0x7faba40a08a0 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 10 monclient: _renew_subs 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faba4125420 con 0x7faba40a08a0 2024-09-16T05:40:50.720 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faba00044f0 con 0x7faba40a08a0 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab99ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7faba00048c0 con 0x7faba40a08a0 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab99ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab9b7fe640 4 set_mon_vals no callback set 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab99ffb640 10 monclient: dump: 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:50.721 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab84005090 con 0x7faba40a08a0 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7faba0004a60 con 0x7faba40a08a0 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(243..243 src has 20..243) ==== 8846+0+0 (unknown 2657157819 0 0) 0x7faba0045740 con 0x7faba40a08a0 2024-09-16T05:40:50.722 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.705+0000 7faba8aaf640 10 monclient: get_auth_request con 0x7fab88036b80 auth_method 0 2024-09-16T05:40:50.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.709+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7faba0015580 con 0x7faba40a08a0 2024-09-16T05:40:50.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.709+0000 7fab99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:50.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.709+0000 7fab99ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:50.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.877+0000 7fabab514640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T05:40:50.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.877+0000 7fabab514640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T05:40:50.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.877+0000 7fabab514640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:50.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:50.877+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) -- 0x7fab84005380 con 0x7faba40a08a0 2024-09-16T05:40:51.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.749+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 removed item id -94 name 'h26' from crush map v244) ==== 120+0+0 (unknown 4022514300 0 0) 0x7faba0014e80 con 0x7faba40a08a0 2024-09-16T05:40:51.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.749+0000 7fab99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T05:40:51.754 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.749+0000 7fab99ffb640 10 monclient: _finish_command 2 = system:0 removed item id -94 name 'h26' from crush map 2024-09-16T05:40:51.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.857+0000 7fabab514640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T05:40:51.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.857+0000 7fabab514640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T05:40:51.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.857+0000 7fabab514640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:51.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.857+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) -- 0x7fab84002220 con 0x7faba40a08a0 2024-09-16T05:40:51.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.857+0000 7fab99ffb640 1 -- 172.21.15.63:0/30304277 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 device 'h26' does not appear in the crush map v244) ==== 120+0+0 (unknown 994949901 0 0) 0x7faba001da30 con 0x7faba40a08a0 2024-09-16T05:40:51.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.857+0000 7fab99ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T05:40:51.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.857+0000 7fab99ffb640 10 monclient: _finish_command 3 = system:0 device 'h26' does not appear in the crush map 2024-09-16T05:40:51.862 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h26' does not appear in the crush map 2024-09-16T05:40:51.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.857+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 >> v1:172.21.15.63:6812/3643207211 conn(0x7fab88036b80 legacy=0x7fab88039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:51.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.861+0000 7fabab514640 10 monclient: shutdown 2024-09-16T05:40:51.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.861+0000 7fabab514640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:51.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.861+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 >> v1:172.21.15.63:6789/0 conn(0x7faba40a08a0 legacy=0x7faba4121d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:51.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.861+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 shutdown_connections 2024-09-16T05:40:51.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.861+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 >> 172.21.15.63:0/30304277 conn(0x7faba40a67c0 msgr2=0x7faba41bcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:51.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.861+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 shutdown_connections 2024-09-16T05:40:51.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.861+0000 7fabab514640 1 -- 172.21.15.63:0/30304277 wait complete. 2024-09-16T05:40:51.878 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:51.878 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h27 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient: get_monmap_and_config 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient: build_initial_monmap 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient: monmap: 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:51.998156+0000 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:51.998156+0000 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:51.998 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:51.999 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:51.999 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 1 Processor -- start 2024-09-16T05:40:52.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 1 -- start start 2024-09-16T05:40:52.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient: init 2024-09-16T05:40:52.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:52.004 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient(hunting): picked mon.noname-a con 0x7fc5f0136ac0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient(hunting): picked mon.noname-b con 0x7fc5f009fd80 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient(hunting): picked mon.noname-c con 0x7fc5f013d0b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f0072cb0 con 0x7fc5f0136ac0 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f0136670 con 0x7fc5f013d0b0 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00a14f0 con 0x7fc5f009fd80 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 10 monclient(hunting): authenticate will time out at 1308.775122s 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f4d34640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fc5f0136ac0 legacy=0x7fc5f0136ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f5535640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc5f013d0b0 0x7fc5f013f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56192/0 (socket says 172.21.15.63:56192) 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f5535640 1 -- 172.21.15.63:0/4220703026 learned_addr learned my addr 172.21.15.63:0/4220703026 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:52.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988597957 0 0) 0x7fc5f0136670 con 0x7fc5f013d0b0 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient(hunting): my global_id is 114256 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5d8003530 con 0x7fc5f013d0b0 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1773165669 0 0) 0x7fc5d8003530 con 0x7fc5f013d0b0 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 >> v1:172.21.15.6:6790/0 conn(0x7fc5f009fd80 legacy=0x7fc5f013b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 >> v1:172.21.15.6:6789/0 conn(0x7fc5f0136ac0 legacy=0x7fc5f0136ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 1 monclient: found mon.noname-c 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:52.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f00a0630 con 0x7fc5f013d0b0 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc5e0001620 con 0x7fc5f013d0b0 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5ef7fe640 10 monclient: dump: 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.007 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 5 monclient: authenticate success, global_id 114256 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.993+0000 7fc5f6f98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5e0001980 con 0x7fc5f013d0b0 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 4 set_mon_vals no callback set 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ef7fe640 1 -- 172.21.15.63:0/4220703026 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc5e00048b0 con 0x7fc5f013d0b0 2024-09-16T05:40:52.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ef7fe640 10 monclient: dump: 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: shutdown 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- 172.21.15.63:0/4220703026 >> v1:172.21.15.63:6789/0 conn(0x7fc5f013d0b0 legacy=0x7fc5f013f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- 172.21.15.63:0/4220703026 shutdown_connections 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- 172.21.15.63:0/4220703026 >> 172.21.15.63:0/4220703026 conn(0x7fc5f00a1c90 msgr2=0x7fc5f00a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- 172.21.15.63:0/4220703026 shutdown_connections 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- 172.21.15.63:0/4220703026 wait complete. 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: build_initial_monmap 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: monmap: 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:52.010 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 Processor -- start 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- start start 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: init 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient(hunting): picked mon.noname-c con 0x7fc5f009fd80 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient(hunting): picked mon.noname-b con 0x7fc5f0136ac0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:52.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient(hunting): picked mon.noname-a con 0x7fc5f013d0b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f0120ad0 con 0x7fc5f013d0b0 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f011f680 con 0x7fc5f0136ac0 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f011f900 con 0x7fc5f009fd80 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient(hunting): authenticate will time out at 1308.778046s 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f5535640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fc5f013d0b0 legacy=0x7fc5f01d3db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5effff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc5f0136ac0 0x7fc5f01d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56198/0 (socket says 172.21.15.63:56198) 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5effff640 1 -- 172.21.15.63:0/1204833781 learned_addr learned my addr 172.21.15.63:0/1204833781 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1692826999 0 0) 0x7fc5f011f680 con 0x7fc5f0136ac0 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:52.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): my global_id is 114259 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5dc003530 con 0x7fc5f0136ac0 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 834885227 0 0) 0x7fc5dc003530 con 0x7fc5f0136ac0 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5f011f680 con 0x7fc5f0136ac0 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc5e4001900 con 0x7fc5f0136ac0 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): dump: 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.013 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4229538091 0 0) 0x7fc5f011f900 con 0x7fc5f009fd80 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient(hunting): my global_id is 6413 2024-09-16T05:40:52.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5dc003530 con 0x7fc5f009fd80 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1168286934 0 0) 0x7fc5f011f680 con 0x7fc5f0136ac0 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 >> v1:172.21.15.6:6790/0 conn(0x7fc5f009fd80 legacy=0x7fc5f0135250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 >> v1:172.21.15.6:6789/0 conn(0x7fc5f013d0b0 legacy=0x7fc5f01d3db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 monclient: found mon.b 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f0120db0 con 0x7fc5f0136ac0 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient: _finish_auth 0 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 5 monclient: authenticate success, global_id 114259 2024-09-16T05:40:52.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: _renew_subs 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f009d860 con 0x7fc5f0136ac0 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: _renew_subs 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5f01207f0 con 0x7fc5f0136ac0 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5e4001d10 con 0x7fc5f0136ac0 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc5e4003af0 con 0x7fc5f0136ac0 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5eeffd640 4 set_mon_vals no callback set 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 10 monclient: dump: 2024-09-16T05:40:52.016 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc5e4003c90 con 0x7fc5f0136ac0 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:52.017 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:51.997+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d0005090 con 0x7fc5f0136ac0 2024-09-16T05:40:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.001+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(244..244 src has 20..244) ==== 8633+0+0 (unknown 1614564786 0 0) 0x7fc5e4045660 con 0x7fc5f0136ac0 2024-09-16T05:40:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.001+0000 7fc5f4d34640 10 monclient: get_auth_request con 0x7fc5dc036ad0 auth_method 0 2024-09-16T05:40:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.005+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc5e4015590 con 0x7fc5f0136ac0 2024-09-16T05:40:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.005+0000 7fc5ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:52.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.005+0000 7fc5ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:52.177 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.173+0000 7fc5f6f98640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T05:40:52.273 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.173+0000 7fc5f6f98640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T05:40:52.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.173+0000 7fc5f6f98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:52.274 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.173+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) -- 0x7fc5d0005380 con 0x7fc5f0136ac0 2024-09-16T05:40:52.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.753+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 removed item id -97 name 'h27' from crush map v245) ==== 120+0+0 (unknown 1799803398 0 0) 0x7fc5e4014e90 con 0x7fc5f0136ac0 2024-09-16T05:40:52.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.753+0000 7fc5ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T05:40:52.757 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.753+0000 7fc5ed7fa640 10 monclient: _finish_command 2 = system:0 removed item id -97 name 'h27' from crush map 2024-09-16T05:40:52.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.857+0000 7fc5f6f98640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T05:40:52.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.857+0000 7fc5f6f98640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T05:40:52.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.857+0000 7fc5f6f98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:52.862 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.857+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) -- 0x7fc5d0002220 con 0x7fc5f0136ac0 2024-09-16T05:40:52.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.857+0000 7fc5ed7fa640 1 -- 172.21.15.63:0/1204833781 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 device 'h27' does not appear in the crush map v245) ==== 120+0+0 (unknown 2406847394 0 0) 0x7fc5e401da40 con 0x7fc5f0136ac0 2024-09-16T05:40:52.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.857+0000 7fc5ed7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T05:40:52.863 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.857+0000 7fc5ed7fa640 10 monclient: _finish_command 3 = system:0 device 'h27' does not appear in the crush map 2024-09-16T05:40:52.863 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h27' does not appear in the crush map 2024-09-16T05:40:52.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.861+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc5dc036ad0 legacy=0x7fc5dc038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:52.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.861+0000 7fc5f6f98640 10 monclient: shutdown 2024-09-16T05:40:52.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.861+0000 7fc5f6f98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:52.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.861+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 >> v1:172.21.15.63:6789/0 conn(0x7fc5f0136ac0 legacy=0x7fc5f01d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:52.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.861+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 shutdown_connections 2024-09-16T05:40:52.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.861+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 >> 172.21.15.63:0/1204833781 conn(0x7fc5f00a1c90 msgr2=0x7fc5f009cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:52.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.861+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 shutdown_connections 2024-09-16T05:40:52.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.861+0000 7fc5f6f98640 1 -- 172.21.15.63:0/1204833781 wait complete. 2024-09-16T05:40:52.879 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:52.880 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h28 2024-09-16T05:40:52.989 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient: get_monmap_and_config 2024-09-16T05:40:52.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient: build_initial_monmap 2024-09-16T05:40:52.990 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient: monmap: 2024-09-16T05:40:52.990 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:52.990 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:52.990 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:52.990016+0000 2024-09-16T05:40:52.990 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:52.990016+0000 2024-09-16T05:40:52.991 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:52.991 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:52.991 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:52.991 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:52.991 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:52.991 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 1 Processor -- start 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 1 -- start start 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient: init 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient(hunting): picked mon.noname-b con 0x7febd009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient(hunting): picked mon.noname-a con 0x7febd006dd30 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient(hunting): picked mon.noname-c con 0x7febd00a0210 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd009f800 con 0x7febd006dd30 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd00a0ab0 con 0x7febd00a0210 2024-09-16T05:40:52.994 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd00a0d90 con 0x7febd009ac90 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febd611d640 10 monclient(hunting): authenticate will time out at 1309.767497s 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febc7fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7febd006dd30 legacy=0x7febd01399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febcffff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7febd00a0210 0x7febd013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56206/0 (socket says 172.21.15.63:56206) 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.985+0000 7febcffff640 1 -- 172.21.15.63:0/4291987730 learned_addr learned my addr 172.21.15.63:0/4291987730 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983852079 0 0) 0x7febd00a0ab0 con 0x7febd00a0210 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient(hunting): my global_id is 114262 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febbc003530 con 0x7febd00a0210 2024-09-16T05:40:52.995 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1700076129 0 0) 0x7febbc003530 con 0x7febd00a0210 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 >> v1:172.21.15.6:6790/0 conn(0x7febd009ac90 legacy=0x7febd009b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 >> v1:172.21.15.6:6789/0 conn(0x7febd006dd30 legacy=0x7febd01399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 monclient: found mon.noname-c 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd009e770 con 0x7febd00a0210 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 5 monclient: authenticate success, global_id 114262 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febc80019a0 con 0x7febd00a0210 2024-09-16T05:40:52.996 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: dump: 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:52.997 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febc8001d00 con 0x7febd00a0210 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 1 -- 172.21.15.63:0/4291987730 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febc8004c30 con 0x7febd00a0210 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 4 set_mon_vals no callback set 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febceffd640 10 monclient: dump: 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:52.998 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: shutdown 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- 172.21.15.63:0/4291987730 >> v1:172.21.15.63:6789/0 conn(0x7febd00a0210 legacy=0x7febd013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- 172.21.15.63:0/4291987730 shutdown_connections 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- 172.21.15.63:0/4291987730 >> 172.21.15.63:0/4291987730 conn(0x7febd00a1c90 msgr2=0x7febd00a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- 172.21.15.63:0/4291987730 shutdown_connections 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- 172.21.15.63:0/4291987730 wait complete. 2024-09-16T05:40:53.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: build_initial_monmap 2024-09-16T05:40:53.000 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: monmap: 2024-09-16T05:40:53.000 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:53.000 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:53.000 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:53.000 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:53.000 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:53.000 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:53.001 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:53.001 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:53.001 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:53.001 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:53.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 Processor -- start 2024-09-16T05:40:53.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- start start 2024-09-16T05:40:53.001 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: init 2024-09-16T05:40:53.005 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient(hunting): picked mon.noname-c con 0x7febd006dd30 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient(hunting): picked mon.noname-b con 0x7febd009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient(hunting): picked mon.noname-a con 0x7febd00a0210 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd009c0f0 con 0x7febd00a0210 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd011ee40 con 0x7febd009ac90 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd0120e50 con 0x7febd006dd30 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient(hunting): authenticate will time out at 1309.770502s 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febcffff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7febd00a0210 legacy=0x7febd01d1790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febc7fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7febd009ac90 0x7febd01ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56208/0 (socket says 172.21.15.63:56208) 2024-09-16T05:40:53.006 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febc7fff640 1 -- 172.21.15.63:0/536225485 learned_addr learned my addr 172.21.15.63:0/536225485 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4222214101 0 0) 0x7febd011ee40 con 0x7febd009ac90 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): my global_id is 114265 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febb0003530 con 0x7febd009ac90 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1691398194 0 0) 0x7febb0003530 con 0x7febd009ac90 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febd011ee40 con 0x7febd009ac90 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febc0004150 con 0x7febd009ac90 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): dump: 2024-09-16T05:40:53.007 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837933990 0 0) 0x7febd0120e50 con 0x7febd006dd30 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:53.008 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient(hunting): my global_id is 6419 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febb0003530 con 0x7febd006dd30 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1870529603 0 0) 0x7febd011ee40 con 0x7febd009ac90 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 >> v1:172.21.15.6:6790/0 conn(0x7febd006dd30 legacy=0x7febd0133770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 >> v1:172.21.15.6:6789/0 conn(0x7febd00a0210 legacy=0x7febd01d1790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient: _finish_hunting 0 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 monclient: found mon.b 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd0121130 con 0x7febd009ac90 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient: _finish_auth 0 2024-09-16T05:40:53.009 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient: _check_auth_tickets 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 5 monclient: authenticate success, global_id 114265 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: _renew_subs 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7febd011f720 con 0x7febd009ac90 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: _renew_subs 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7febd0134050 con 0x7febd009ac90 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febc00043b0 con 0x7febd009ac90 2024-09-16T05:40:53.010 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febc0004700 con 0x7febd009ac90 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febcdffb640 4 set_mon_vals no callback set 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.989+0000 7febccff9640 10 monclient: dump: 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:53.011 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.993+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7febc0004920 con 0x7febd009ac90 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.993+0000 7febd611d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.993+0000 7febd611d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.993+0000 7febd611d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.993+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb4005090 con 0x7febd009ac90 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.993+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(245..245 src has 20..245) ==== 8420+0+0 (unknown 1976750310 0 0) 0x7febc00454f0 con 0x7febd009ac90 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.993+0000 7febcf7fe640 10 monclient: get_auth_request con 0x7febb0036ad0 auth_method 0 2024-09-16T05:40:53.012 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.997+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7febc00190f0 con 0x7febd009ac90 2024-09-16T05:40:53.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.997+0000 7febccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:53.013 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:52.997+0000 7febccff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:53.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.157+0000 7febd611d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T05:40:53.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.157+0000 7febd611d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T05:40:53.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.157+0000 7febd611d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:53.164 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.157+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) -- 0x7febb4005380 con 0x7febd009ac90 2024-09-16T05:40:53.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.765+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 removed item id -100 name 'h28' from crush map v246) ==== 121+0+0 (unknown 3242944796 0 0) 0x7febc001d9b0 con 0x7febd009ac90 2024-09-16T05:40:53.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.765+0000 7febccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T05:40:53.772 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.765+0000 7febccff9640 10 monclient: _finish_command 2 = system:0 removed item id -100 name 'h28' from crush map 2024-09-16T05:40:53.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.869+0000 7febd611d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T05:40:53.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.869+0000 7febd611d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T05:40:53.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.869+0000 7febd611d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:53.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.869+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) -- 0x7febb4002220 con 0x7febd009ac90 2024-09-16T05:40:53.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.869+0000 7febccff9640 1 -- 172.21.15.63:0/536225485 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 device 'h28' does not appear in the crush map v246) ==== 120+0+0 (unknown 1665383785 0 0) 0x7febc0013120 con 0x7febd009ac90 2024-09-16T05:40:53.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.869+0000 7febccff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T05:40:53.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.869+0000 7febccff9640 10 monclient: _finish_command 3 = system:0 device 'h28' does not appear in the crush map 2024-09-16T05:40:53.876 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h28' does not appear in the crush map 2024-09-16T05:40:53.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.873+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 >> v1:172.21.15.63:6812/3643207211 conn(0x7febb0036ad0 legacy=0x7febb0038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:53.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.873+0000 7febd611d640 10 monclient: shutdown 2024-09-16T05:40:53.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.873+0000 7febd611d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:53.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.873+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 >> v1:172.21.15.63:6789/0 conn(0x7febd009ac90 legacy=0x7febd01ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:53.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.873+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 shutdown_connections 2024-09-16T05:40:53.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.873+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 >> 172.21.15.63:0/536225485 conn(0x7febd00a1c90 msgr2=0x7febd00a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:53.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.873+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 shutdown_connections 2024-09-16T05:40:53.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:53.873+0000 7febd611d640 1 -- 172.21.15.63:0/536225485 wait complete. 2024-09-16T05:40:53.892 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:53.892 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h29 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient: get_monmap_and_config 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient: build_initial_monmap 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient: monmap: 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:54.014116+0000 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:54.014116+0000 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:54.014 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:54.015 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:54.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 1 Processor -- start 2024-09-16T05:40:54.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 1 -- start start 2024-09-16T05:40:54.015 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient: init 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient(hunting): picked mon.noname-c con 0x7ff9a40a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient(hunting): picked mon.noname-b con 0x7ff9a40a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient(hunting): picked mon.noname-a con 0x7ff9a40a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a40a4fd0 con 0x7ff9a40a4790 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a40a5b40 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a40a1a70 con 0x7ff9a40a08a0 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:54.018 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.009+0000 7ff9a8f3f640 10 monclient(hunting): authenticate will time out at 1310.791869s 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a259c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff9a40a3ca0 0x7ff9a40a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56220/0 (socket says 172.21.15.63:56220) 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a2d9d640 1 -- 172.21.15.63:0/2458313330 >> v1:172.21.15.6:6789/0 conn(0x7ff9a40a4790 legacy=0x7ff9a413f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a259c640 1 -- 172.21.15.63:0/2458313330 learned_addr learned my addr 172.21.15.63:0/2458313330 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3212039702 0 0) 0x7ff9a40a5b40 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient(hunting): my global_id is 114268 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff98c003530 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3018261201 0 0) 0x7ff98c003530 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 >> v1:172.21.15.6:6790/0 conn(0x7ff9a40a08a0 legacy=0x7ff9a413beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 >> v1:172.21.15.6:6789/0 conn(0x7ff9a40a4790 legacy=0x7ff9a413f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:54.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 monclient: found mon.noname-c 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a40a53e0 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: _finish_auth 0 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff994001620 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: dump: 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:54.020 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:54.021 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 5 monclient: authenticate success, global_id 114268 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff994001910 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 1 -- 172.21.15.63:0/2458313330 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff994001d20 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a159a640 10 monclient: dump: 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:54.022 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 4 set_mon_vals no callback set 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: shutdown 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/2458313330 >> v1:172.21.15.63:6789/0 conn(0x7ff9a40a3ca0 legacy=0x7ff9a40a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/2458313330 shutdown_connections 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/2458313330 >> 172.21.15.63:0/2458313330 conn(0x7ff9a40a67c0 msgr2=0x7ff9a4075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:54.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/2458313330 shutdown_connections 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/2458313330 wait complete. 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: build_initial_monmap 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: monmap: 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:54.024 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:54.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 Processor -- start 2024-09-16T05:40:54.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- start start 2024-09-16T05:40:54.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: init 2024-09-16T05:40:54.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:54.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:54.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient(hunting): picked mon.noname-a con 0x7ff9a40a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:54.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient(hunting): picked mon.noname-b con 0x7ff9a40a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient(hunting): picked mon.noname-c con 0x7ff9a40a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a4127880 con 0x7ff9a40a08a0 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a4125d10 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a4127b60 con 0x7ff9a40a4790 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient(hunting): authenticate will time out at 1310.795099s 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a1d9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff9a40a3ca0 0x7ff9a4137150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:56228/0 (socket says 172.21.15.63:56228) 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a1d9b640 1 -- 172.21.15.63:0/413434699 learned_addr learned my addr 172.21.15.63:0/413434699 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a259c640 1 -- 172.21.15.63:0/413434699 >> v1:172.21.15.6:6789/0 conn(0x7ff9a40a08a0 legacy=0x7ff9a41cb0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653772451 0 0) 0x7ff9a4125d10 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:54.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient(hunting): my global_id is 114271 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff984003530 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2128687290 0 0) 0x7ff984003530 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9a4125d10 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff9980018a0 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:54.030 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1613978958 0 0) 0x7ff9a4125d10 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 >> v1:172.21.15.6:6790/0 conn(0x7ff9a40a4790 legacy=0x7ff9a41d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 >> v1:172.21.15.6:6789/0 conn(0x7ff9a40a08a0 legacy=0x7ff9a41cb0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:54.031 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:54.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 monclient: found mon.b 2024-09-16T05:40:54.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:54.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:54.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a40a8950 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:54.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:54.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:54.032 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 5 monclient: authenticate success, global_id 114271 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: _renew_subs 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a40a8420 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: _renew_subs 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.013+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9a4137860 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff998003620 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff992ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:54.033 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff9980039f0 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff992ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff992ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff9a0d99640 4 set_mon_vals no callback set 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff992ffd640 10 monclient: dump: 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:54.034 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff9a8f3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff9a8f3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff9a8f3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff988005090 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff998003c10 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(246..246 src has 20..246) ==== 8207+0+0 (unknown 530731776 0 0) 0x7ff998045440 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.017+0000 7ff9a259c640 10 monclient: get_auth_request con 0x7ff9840369d0 auth_method 0 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.021+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff998015520 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.021+0000 7ff992ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:54.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.021+0000 7ff992ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:54.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.189+0000 7ff9a8f3f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T05:40:54.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.189+0000 7ff9a8f3f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T05:40:54.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.189+0000 7ff9a8f3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:54.196 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.189+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) -- 0x7ff988005380 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.769+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 removed item id -103 name 'h29' from crush map v247) ==== 121+0+0 (unknown 996859508 0 0) 0x7ff998014e20 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.769+0000 7ff992ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T05:40:54.775 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.769+0000 7ff992ffd640 10 monclient: _finish_command 2 = system:0 removed item id -103 name 'h29' from crush map 2024-09-16T05:40:54.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.877+0000 7ff9a8f3f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T05:40:54.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.877+0000 7ff9a8f3f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T05:40:54.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.877+0000 7ff9a8f3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:54.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.877+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) -- 0x7ff988002220 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.877+0000 7ff992ffd640 1 -- 172.21.15.63:0/413434699 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 device 'h29' does not appear in the crush map v247) ==== 120+0+0 (unknown 3615200710 0 0) 0x7ff99801d9d0 con 0x7ff9a40a3ca0 2024-09-16T05:40:54.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.877+0000 7ff992ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T05:40:54.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.877+0000 7ff992ffd640 10 monclient: _finish_command 3 = system:0 device 'h29' does not appear in the crush map 2024-09-16T05:40:54.883 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h29' does not appear in the crush map 2024-09-16T05:40:54.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.881+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff9840369d0 legacy=0x7ff984038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:54.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.881+0000 7ff9a8f3f640 10 monclient: shutdown 2024-09-16T05:40:54.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.881+0000 7ff9a8f3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:54.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.881+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 >> v1:172.21.15.63:6789/0 conn(0x7ff9a40a3ca0 legacy=0x7ff9a4137150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:54.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.881+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 shutdown_connections 2024-09-16T05:40:54.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.881+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 >> 172.21.15.63:0/413434699 conn(0x7ff9a40a67c0 msgr2=0x7ff9a4070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:54.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.881+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 shutdown_connections 2024-09-16T05:40:54.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:54.881+0000 7ff9a8f3f640 1 -- 172.21.15.63:0/413434699 wait complete. 2024-09-16T05:40:54.899 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:54.899 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h30 2024-09-16T05:40:55.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.013+0000 7f23e0c9f640 10 monclient: get_monmap_and_config 2024-09-16T05:40:55.019 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.013+0000 7f23e0c9f640 10 monclient: build_initial_monmap 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.013+0000 7f23e0c9f640 10 monclient: monmap: 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:55.019717+0000 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:55.019717+0000 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:55.020 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:55.023 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 Processor -- start 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- start start 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient: init 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f23dc0a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f23dc0a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient(hunting): picked mon.noname-a con 0x7f23dc0a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc0a4fd0 con 0x7f23dc0a4790 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc0a5b40 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc0a1a70 con 0x7f23dc0a08a0 2024-09-16T05:40:55.024 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient(hunting): authenticate will time out at 1311.797211s 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23dad9d640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f23dc0a4790 legacy=0x7f23dc13f5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23da59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f23dc0a3ca0 0x7f23dc0a4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49926/0 (socket says 172.21.15.63:49926) 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23da59c640 1 -- 172.21.15.63:0/3547301987 learned_addr learned my addr 172.21.15.63:0/3547301987 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2532852654 0 0) 0x7f23dc0a5b40 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient(hunting): my global_id is 114274 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23c4003530 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3884009479 0 0) 0x7f23c4003530 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.025 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 >> v1:172.21.15.6:6790/0 conn(0x7f23dc0a08a0 legacy=0x7f23dc13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 >> v1:172.21.15.6:6789/0 conn(0x7f23dc0a4790 legacy=0x7f23dc13f5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 monclient: found mon.noname-c 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc0a53e0 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: _finish_auth 0 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23cc001620 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:55.026 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: dump: 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 5 monclient: authenticate success, global_id 114274 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:55.027 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23cc001910 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 4 set_mon_vals no callback set 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 1 -- 172.21.15.63:0/3547301987 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23cc001d20 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23d959a640 10 monclient: dump: 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:55.028 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient: shutdown 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- 172.21.15.63:0/3547301987 >> v1:172.21.15.63:6789/0 conn(0x7f23dc0a3ca0 legacy=0x7f23dc0a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- 172.21.15.63:0/3547301987 shutdown_connections 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- 172.21.15.63:0/3547301987 >> 172.21.15.63:0/3547301987 conn(0x7f23dc0a67c0 msgr2=0x7f23dc075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- 172.21.15.63:0/3547301987 shutdown_connections 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- 172.21.15.63:0/3547301987 wait complete. 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient: build_initial_monmap 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient: monmap: 2024-09-16T05:40:55.029 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 Processor -- start 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 1 -- start start 2024-09-16T05:40:55.030 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.017+0000 7f23e0c9f640 10 monclient: init 2024-09-16T05:40:55.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:55.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:55.034 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f23dc0a08a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient(hunting): picked mon.noname-a con 0x7f23dc0a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f23dc0a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc137a30 con 0x7f23dc0a3ca0 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc1cc140 con 0x7f23dc0a4790 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc124dc0 con 0x7f23dc0a08a0 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient(hunting): authenticate will time out at 1311.800351s 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23dad9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f23dc0a4790 0x7f23dc1d3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49932/0 (socket says 172.21.15.63:49932) 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23dad9d640 1 -- 172.21.15.63:0/1022495863 learned_addr learned my addr 172.21.15.63:0/1022495863 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23d9d9b640 1 -- 172.21.15.63:0/1022495863 >> v1:172.21.15.6:6789/0 conn(0x7f23dc0a3ca0 legacy=0x7f23dc1366d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329747513 0 0) 0x7f23dc1cc140 con 0x7f23dc0a4790 2024-09-16T05:40:55.035 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): my global_id is 114277 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23bc003530 con 0x7f23dc0a4790 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3706667416 0 0) 0x7f23bc003530 con 0x7f23dc0a4790 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23dc1cc140 con 0x7f23dc0a4790 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23d40043c0 con 0x7f23dc0a4790 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): dump: 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:55.036 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2461133235 0 0) 0x7f23dc124dc0 con 0x7f23dc0a08a0 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:55.037 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient(hunting): my global_id is 6431 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23bc003530 con 0x7f23dc0a08a0 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 133570655 0 0) 0x7f23dc1cc140 con 0x7f23dc0a4790 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 >> v1:172.21.15.6:6790/0 conn(0x7f23dc0a08a0 legacy=0x7f23dc1cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 >> v1:172.21.15.6:6789/0 conn(0x7f23dc0a3ca0 legacy=0x7f23dc1366d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 monclient: found mon.b 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc1250a0 con 0x7f23dc0a4790 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:55.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 5 monclient: authenticate success, global_id 114277 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient: _renew_subs 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f23dc0a8390 con 0x7f23dc0a4790 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient: _renew_subs 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f23dc124ae0 con 0x7f23dc0a4790 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23d4004620 con 0x7f23dc0a4790 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23d4004970 con 0x7f23dc0a4790 2024-09-16T05:40:55.039 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23d8d99640 4 set_mon_vals no callback set 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 10 monclient: dump: 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:55.040 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f23d4004b90 con 0x7f23dc0a4790 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23c0005090 con 0x7f23dc0a4790 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(247..247 src has 20..247) ==== 7994+0+0 (unknown 669045639 0 0) 0x7f23d40455d0 con 0x7f23dc0a4790 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.021+0000 7f23da59c640 10 monclient: get_auth_request con 0x7f23bc036b80 auth_method 0 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.025+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f23d4015710 con 0x7f23dc0a4790 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.025+0000 7f23caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:55.041 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.025+0000 7f23caffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:55.194 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.189+0000 7f23e0c9f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T05:40:55.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.189+0000 7f23e0c9f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T05:40:55.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.189+0000 7f23e0c9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:55.195 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.189+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) -- 0x7f23c0005380 con 0x7f23dc0a4790 2024-09-16T05:40:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.785+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 removed item id -106 name 'h30' from crush map v248) ==== 121+0+0 (unknown 1230998154 0 0) 0x7f23d4015010 con 0x7f23dc0a4790 2024-09-16T05:40:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.785+0000 7f23caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T05:40:55.792 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.785+0000 7f23caffd640 10 monclient: _finish_command 2 = system:0 removed item id -106 name 'h30' from crush map 2024-09-16T05:40:55.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.893+0000 7f23e0c9f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T05:40:55.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.893+0000 7f23e0c9f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T05:40:55.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.893+0000 7f23e0c9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:55.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.893+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) -- 0x7f23c0002220 con 0x7f23dc0a4790 2024-09-16T05:40:55.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.893+0000 7f23caffd640 1 -- 172.21.15.63:0/1022495863 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 device 'h30' does not appear in the crush map v248) ==== 120+0+0 (unknown 1899368546 0 0) 0x7f23d4019360 con 0x7f23dc0a4790 2024-09-16T05:40:55.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.893+0000 7f23caffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T05:40:55.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.893+0000 7f23caffd640 10 monclient: _finish_command 3 = system:0 device 'h30' does not appear in the crush map 2024-09-16T05:40:55.900 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h30' does not appear in the crush map 2024-09-16T05:40:55.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.897+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 >> v1:172.21.15.63:6812/3643207211 conn(0x7f23bc036b80 legacy=0x7f23bc039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:55.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.897+0000 7f23e0c9f640 10 monclient: shutdown 2024-09-16T05:40:55.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.897+0000 7f23e0c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:55.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.897+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 >> v1:172.21.15.63:6789/0 conn(0x7f23dc0a4790 legacy=0x7f23dc1d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:55.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.897+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 shutdown_connections 2024-09-16T05:40:55.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.897+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 >> 172.21.15.63:0/1022495863 conn(0x7f23dc0a67c0 msgr2=0x7f23dc137320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:55.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.897+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 shutdown_connections 2024-09-16T05:40:55.902 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:55.897+0000 7f23e0c9f640 1 -- 172.21.15.63:0/1022495863 wait complete. 2024-09-16T05:40:55.916 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:55.917 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h31 2024-09-16T05:40:56.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient: get_monmap_and_config 2024-09-16T05:40:56.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient: build_initial_monmap 2024-09-16T05:40:56.038 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient: monmap: 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:56.038701+0000 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:56.038701+0000 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:56.039 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 1 Processor -- start 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 1 -- start start 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient: init 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient(hunting): picked mon.noname-a con 0x7f4e7c0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient(hunting): picked mon.noname-c con 0x7f4e7c09b450 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient(hunting): picked mon.noname-b con 0x7f4e7c09b8a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c09e970 con 0x7f4e7c0a08a0 2024-09-16T05:40:56.047 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c09c370 con 0x7f4e7c09b450 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c09d620 con 0x7f4e7c09b8a0 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e83f35640 10 monclient(hunting): authenticate will time out at 1312.815813s 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e81cd1640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f4e7c0a08a0 legacy=0x7f4e7c09ad40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e814d0640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f4e7c09b450 0x7f4e7c1389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49942/0 (socket says 172.21.15.63:49942) 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.033+0000 7f4e814d0640 1 -- 172.21.15.63:0/2636187410 learned_addr learned my addr 172.21.15.63:0/2636187410 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368868152 0 0) 0x7f4e7c09c370 con 0x7f4e7c09b450 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient(hunting): _negotiate 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:56.048 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient(hunting): my global_id is 114280 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e64003530 con 0x7f4e7c09b450 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3019099435 0 0) 0x7f4e64003530 con 0x7f4e7c09b450 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 >> v1:172.21.15.6:6790/0 conn(0x7f4e7c09b8a0 legacy=0x7f4e7c13c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 >> v1:172.21.15.6:6789/0 conn(0x7f4e7c0a08a0 legacy=0x7f4e7c09ad40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: _finish_hunting 0 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 monclient: found mon.noname-c 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c09d900 con 0x7f4e7c09b450 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: _finish_auth 0 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: _check_auth_tickets 2024-09-16T05:40:56.049 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 5 monclient: authenticate success, global_id 114280 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4e70001620 con 0x7f4e7c09b450 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: dump: 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:56.050 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4e70001910 con 0x7f4e7c09b450 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 4 set_mon_vals no callback set 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 1 -- 172.21.15.63:0/2636187410 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4e70001ce0 con 0x7f4e7c09b450 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e80ccf640 10 monclient: dump: 2024-09-16T05:40:56.051 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient: shutdown 2024-09-16T05:40:56.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- 172.21.15.63:0/2636187410 >> v1:172.21.15.63:6789/0 conn(0x7f4e7c09b450 legacy=0x7f4e7c1389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- 172.21.15.63:0/2636187410 shutdown_connections 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- 172.21.15.63:0/2636187410 >> 172.21.15.63:0/2636187410 conn(0x7f4e7c0a1c90 msgr2=0x7f4e7c0a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- 172.21.15.63:0/2636187410 shutdown_connections 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- 172.21.15.63:0/2636187410 wait complete. 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient: build_initial_monmap 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient: monmap: 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:56.053 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 Processor -- start 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- start start 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient: init 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient(hunting): picked mon.noname-b con 0x7f4e7c09b450 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient(hunting): picked mon.noname-c con 0x7f4e7c09b8a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:56.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient(hunting): picked mon.noname-a con 0x7f4e7c0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c11e830 con 0x7f4e7c0a08a0 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c11c200 con 0x7f4e7c09b450 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c13cc70 con 0x7f4e7c09b8a0 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e83f35640 10 monclient(hunting): authenticate will time out at 1312.819149s 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e81cd1640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f4e7c09b450 0x7f4e7c11b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49954/0 (socket says 172.21.15.63:49954) 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e81cd1640 1 -- 172.21.15.63:0/3840408393 learned_addr learned my addr 172.21.15.63:0/3840408393 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e824d2640 1 -- 172.21.15.63:0/3840408393 >> v1:172.21.15.6:6789/0 conn(0x7f4e7c0a08a0 legacy=0x7f4e7c1d3890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3824310659 0 0) 0x7f4e7c11c200 con 0x7f4e7c09b450 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:56.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): my global_id is 114283 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e68003560 con 0x7f4e7c09b450 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3479321697 0 0) 0x7f4e68003560 con 0x7f4e7c09b450 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e7c11c200 con 0x7f4e7c09b450 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4e6c004560 con 0x7f4e7c09b450 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): dump: 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:56.056 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767510875 0 0) 0x7f4e7c13cc70 con 0x7f4e7c09b8a0 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient(hunting): my global_id is 6437 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e68003560 con 0x7f4e7c09b8a0 2024-09-16T05:40:56.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3428500389 0 0) 0x7f4e7c11c200 con 0x7f4e7c09b450 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 >> v1:172.21.15.6:6790/0 conn(0x7f4e7c09b8a0 legacy=0x7f4e7c11baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 >> v1:172.21.15.6:6789/0 conn(0x7f4e7c0a08a0 legacy=0x7f4e7c1d3890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.037+0000 7f4e7a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 1 monclient: found mon.b 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c0a3f60 con 0x7f4e7c09b450 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:56.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 5 monclient: authenticate success, global_id 114283 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 10 monclient: _renew_subs 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7c09c0f0 con 0x7f4e7c09b450 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 10 monclient: _renew_subs 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4e7c11d100 con 0x7f4e7c09b450 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4e6c004850 con 0x7f4e7c09b450 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4e6c004c20 con 0x7f4e7c09b450 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:56.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7bfff640 4 set_mon_vals no callback set 2024-09-16T05:40:56.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:56.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 10 monclient: dump: 2024-09-16T05:40:56.060 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:56.060 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:56.060 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:56.060 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:56.060 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e5c005090 con 0x7f4e7c09b450 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4e6c004e40 con 0x7f4e7c09b450 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(248..248 src has 20..248) ==== 7781+0+0 (unknown 3290742957 0 0) 0x7f4e6c045670 con 0x7f4e7c09b450 2024-09-16T05:40:56.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.041+0000 7f4e814d0640 10 monclient: get_auth_request con 0x7f4e68036ad0 auth_method 0 2024-09-16T05:40:56.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.045+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4e6c0159c0 con 0x7f4e7c09b450 2024-09-16T05:40:56.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.045+0000 7f4e7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:56.062 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.045+0000 7f4e7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:56.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.233+0000 7f4e83f35640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T05:40:56.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.233+0000 7f4e83f35640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T05:40:56.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.233+0000 7f4e83f35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:56.237 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.233+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) -- 0x7f4e5c005380 con 0x7f4e7c09b450 2024-09-16T05:40:56.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.789+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 removed item id -109 name 'h31' from crush map v249) ==== 121+0+0 (unknown 1657264416 0 0) 0x7f4e6c0152c0 con 0x7f4e7c09b450 2024-09-16T05:40:56.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.789+0000 7f4e7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T05:40:56.795 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.789+0000 7f4e7a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -109 name 'h31' from crush map 2024-09-16T05:40:56.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T05:40:56.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T05:40:56.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:56.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) -- 0x7f4e5c002220 con 0x7f4e7c09b450 2024-09-16T05:40:56.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e7a7fc640 1 -- 172.21.15.63:0/3840408393 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 device 'h31' does not appear in the crush map v249) ==== 120+0+0 (unknown 3306043597 0 0) 0x7f4e6c019610 con 0x7f4e7c09b450 2024-09-16T05:40:56.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e7a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T05:40:56.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e7a7fc640 10 monclient: _finish_command 3 = system:0 device 'h31' does not appear in the crush map 2024-09-16T05:40:56.906 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h31' does not appear in the crush map 2024-09-16T05:40:56.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4e68036ad0 legacy=0x7f4e68038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:56.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 10 monclient: shutdown 2024-09-16T05:40:56.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:56.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 >> v1:172.21.15.63:6789/0 conn(0x7f4e7c09b450 legacy=0x7f4e7c11b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:56.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 shutdown_connections 2024-09-16T05:40:56.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 >> 172.21.15.63:0/3840408393 conn(0x7f4e7c0a1c90 msgr2=0x7f4e7c0a4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:56.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 shutdown_connections 2024-09-16T05:40:56.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:56.901+0000 7f4e83f35640 1 -- 172.21.15.63:0/3840408393 wait complete. 2024-09-16T05:40:56.922 INFO:tasks.workunit.client.0.smithi063.stderr:+ for f in `seq 1 32` 2024-09-16T05:40:56.922 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm h32 2024-09-16T05:40:57.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient: get_monmap_and_config 2024-09-16T05:40:57.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient: build_initial_monmap 2024-09-16T05:40:57.042 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient: monmap: 2024-09-16T05:40:57.042 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:57.042 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:57.042 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:57.042502+0000 2024-09-16T05:40:57.042 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:57.042502+0000 2024-09-16T05:40:57.043 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:57.043 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:57.043 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:57.043 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:57.043 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:57.043 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:57.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 1 Processor -- start 2024-09-16T05:40:57.050 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 1 -- start start 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient: init 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient(hunting): picked mon.noname-a con 0x7f20d80a3ca0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient(hunting): picked mon.noname-c con 0x7f20d80a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient(hunting): picked mon.noname-b con 0x7f20d80a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d80a4fd0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d80a5b40 con 0x7f20d80a08a0 2024-09-16T05:40:57.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d80a1a70 con 0x7f20d80a4790 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20dd4ef640 10 monclient(hunting): authenticate will time out at 1313.819563s 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20d6ffd640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f20d80a3ca0 legacy=0x7f20d80a4080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20d67fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f20d80a08a0 0x7f20d813beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49960/0 (socket says 172.21.15.63:49960) 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20d67fc640 1 -- 172.21.15.63:0/4120191186 learned_addr learned my addr 172.21.15.63:0/4120191186 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.037+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 717369533 0 0) 0x7f20d80a5b40 con 0x7f20d80a08a0 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient(hunting): my global_id is 114286 2024-09-16T05:40:57.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20c0003530 con 0x7f20d80a08a0 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3118212952 0 0) 0x7f20c0003530 con 0x7f20d80a08a0 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 >> v1:172.21.15.6:6790/0 conn(0x7f20d80a4790 legacy=0x7f20d813f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 >> v1:172.21.15.6:6789/0 conn(0x7f20d80a3ca0 legacy=0x7f20d80a4080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 monclient: found mon.noname-c 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d80a53e0 con 0x7f20d80a08a0 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20cc001620 con 0x7f20d80a08a0 2024-09-16T05:40:57.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: dump: 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:57.054 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 5 monclient: authenticate success, global_id 114286 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20cc001910 con 0x7f20d80a08a0 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 4 set_mon_vals no callback set 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 1 -- 172.21.15.63:0/4120191186 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20cc001d20 con 0x7f20d80a08a0 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d5ffb640 10 monclient: dump: 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:57.055 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient: shutdown 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- 172.21.15.63:0/4120191186 >> v1:172.21.15.63:6789/0 conn(0x7f20d80a08a0 legacy=0x7f20d813beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- 172.21.15.63:0/4120191186 shutdown_connections 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- 172.21.15.63:0/4120191186 >> 172.21.15.63:0/4120191186 conn(0x7f20d80a67c0 msgr2=0x7f20d8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:57.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- 172.21.15.63:0/4120191186 shutdown_connections 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- 172.21.15.63:0/4120191186 wait complete. 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient: build_initial_monmap 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient: monmap: 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:57.057 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:57.058 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:57.058 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:57.058 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:57.058 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:57.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 Processor -- start 2024-09-16T05:40:57.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- start start 2024-09-16T05:40:57.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient: init 2024-09-16T05:40:57.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:57.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:40:57.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient(hunting): picked mon.noname-a con 0x7f20d80a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:57.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient(hunting): picked mon.noname-b con 0x7f20d80a3ca0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:57.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient(hunting): picked mon.noname-c con 0x7f20d80a4790 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:57.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d80a85d0 con 0x7f20d80a08a0 2024-09-16T05:40:57.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d8125ad0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d80a80a0 con 0x7f20d80a4790 2024-09-16T05:40:57.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:57.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20dd4ef640 10 monclient(hunting): authenticate will time out at 1313.823072s 2024-09-16T05:40:57.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d6ffd640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f20d80a08a0 legacy=0x7f20d8137100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:57.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d67fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f20d80a3ca0 0x7f20d81d02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49970/0 (socket says 172.21.15.63:49970) 2024-09-16T05:40:57.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20d67fc640 1 -- 172.21.15.63:0/2724268222 learned_addr learned my addr 172.21.15.63:0/2724268222 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:57.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1811087479 0 0) 0x7f20d8125ad0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:57.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:57.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:57.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): my global_id is 114289 2024-09-16T05:40:57.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20c4003530 con 0x7f20d80a3ca0 2024-09-16T05:40:57.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1621652108 0 0) 0x7f20c4003530 con 0x7f20d80a3ca0 2024-09-16T05:40:57.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20d8125ad0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20cc0015e0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:57.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:57.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): dump: 2024-09-16T05:40:57.062 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:57.062 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:57.062 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:57.062 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:57.062 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:57.062 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:57.062 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:57.062 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:57.063 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:57.063 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:57.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:57.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3282870427 0 0) 0x7f20d80a80a0 con 0x7f20d80a4790 2024-09-16T05:40:57.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:40:57.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:57.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:57.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 10 monclient(hunting): my global_id is 6443 2024-09-16T05:40:57.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20c4003530 con 0x7f20d80a4790 2024-09-16T05:40:57.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.041+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3153820401 0 0) 0x7f20d8125ad0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 >> v1:172.21.15.6:6790/0 conn(0x7f20d80a4790 legacy=0x7f20d81d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:57.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 >> v1:172.21.15.6:6789/0 conn(0x7f20d80a08a0 legacy=0x7f20d8137100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:57.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:40:57.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 1 monclient: found mon.b 2024-09-16T05:40:57.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:57.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:57.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d80a0500 con 0x7f20d80a3ca0 2024-09-16T05:40:57.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:40:57.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:40:57.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:57.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 5 monclient: authenticate success, global_id 114289 2024-09-16T05:40:57.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 10 monclient: _renew_subs 2024-09-16T05:40:57.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:57.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20d81257f0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 10 monclient: _renew_subs 2024-09-16T05:40:57.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:57.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f20d8137810 con 0x7f20d80a3ca0 2024-09-16T05:40:57.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20cc004cb0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:57.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20d57fa640 4 set_mon_vals no callback set 2024-09-16T05:40:57.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f20cc005080 con 0x7f20d80a3ca0 2024-09-16T05:40:57.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:57.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:57.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 10 monclient: dump: 2024-09-16T05:40:57.067 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:57.068 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:57.068 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:57.068 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:57.068 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:57.068 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:57.068 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:57.069 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:57.069 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:57.069 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:57.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:57.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:57.069 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:57.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20b8005090 con 0x7f20d80a3ca0 2024-09-16T05:40:57.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f20cc0052a0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(249..249 src has 20..249) ==== 7568+0+0 (unknown 3336171053 0 0) 0x7f20cc0453d0 con 0x7f20d80a3ca0 2024-09-16T05:40:57.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.045+0000 7f20d6ffd640 10 monclient: get_auth_request con 0x7f20c4036b30 auth_method 0 2024-09-16T05:40:57.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.049+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f20cc015860 con 0x7f20d80a3ca0 2024-09-16T05:40:57.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.049+0000 7f20bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:57.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.049+0000 7f20bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:57.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.217+0000 7f20dd4ef640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T05:40:57.222 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.217+0000 7f20dd4ef640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T05:40:57.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.217+0000 7f20dd4ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:57.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.217+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) -- 0x7f20b8005380 con 0x7f20d80a3ca0 2024-09-16T05:40:57.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.805+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 removed item id -112 name 'h32' from crush map v250) ==== 121+0+0 (unknown 3652679579 0 0) 0x7f20cc01dd10 con 0x7f20d80a3ca0 2024-09-16T05:40:57.811 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.805+0000 7f20bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T05:40:57.812 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.805+0000 7f20bf7fe640 10 monclient: _finish_command 2 = system:0 removed item id -112 name 'h32' from crush map 2024-09-16T05:40:57.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.909+0000 7f20dd4ef640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T05:40:57.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.909+0000 7f20dd4ef640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T05:40:57.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.909+0000 7f20dd4ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:57.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.909+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) -- 0x7f20b8002220 con 0x7f20d80a3ca0 2024-09-16T05:40:57.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.909+0000 7f20bf7fe640 1 -- 172.21.15.63:0/2724268222 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 device 'h32' does not appear in the crush map v250) ==== 120+0+0 (unknown 480917453 0 0) 0x7f20cc042370 con 0x7f20d80a3ca0 2024-09-16T05:40:57.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.909+0000 7f20bf7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T05:40:57.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.909+0000 7f20bf7fe640 10 monclient: _finish_command 3 = system:0 device 'h32' does not appear in the crush map 2024-09-16T05:40:57.916 INFO:tasks.workunit.client.0.smithi063.stderr:device 'h32' does not appear in the crush map 2024-09-16T05:40:57.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.913+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 >> v1:172.21.15.63:6812/3643207211 conn(0x7f20c4036b30 legacy=0x7f20c4038ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:57.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.913+0000 7f20dd4ef640 10 monclient: shutdown 2024-09-16T05:40:57.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.913+0000 7f20dd4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:57.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.913+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 >> v1:172.21.15.63:6789/0 conn(0x7f20d80a3ca0 legacy=0x7f20d81d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:57.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.913+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 shutdown_connections 2024-09-16T05:40:57.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.913+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 >> 172.21.15.63:0/2724268222 conn(0x7f20d80a67c0 msgr2=0x7f20d81bd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:57.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.913+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 shutdown_connections 2024-09-16T05:40:57.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:57.913+0000 7f20dd4ef640 1 -- 172.21.15.63:0/2724268222 wait complete. 2024-09-16T05:40:57.932 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush rm r1 2024-09-16T05:40:58.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.045+0000 7fc8447b0640 10 monclient: get_monmap_and_config 2024-09-16T05:40:58.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.045+0000 7fc8447b0640 10 monclient: build_initial_monmap 2024-09-16T05:40:58.051 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.045+0000 7fc8447b0640 10 monclient: monmap: 2024-09-16T05:40:58.051 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:58.051744+0000 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:58.051744+0000 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:58.052 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:58.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 Processor -- start 2024-09-16T05:40:58.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- start start 2024-09-16T05:40:58.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient: init 2024-09-16T05:40:58.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:58.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:58.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient(hunting): picked mon.noname-b con 0x7fc83c0a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:58.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient(hunting): picked mon.noname-a con 0x7fc83c0a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:58.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient(hunting): picked mon.noname-c con 0x7fc83c0a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:58.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc83c0a4fd0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc83c0a5b40 con 0x7fc83c0a4790 2024-09-16T05:40:58.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc83c0a1a70 con 0x7fc83c0a3ca0 2024-09-16T05:40:58.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:58.057 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient(hunting): authenticate will time out at 1314.828846s 2024-09-16T05:40:58.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc842d4d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc83c0a4790 0x7fc83c13f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49982/0 (socket says 172.21.15.63:49982) 2024-09-16T05:40:58.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc842d4d640 1 -- 172.21.15.63:0/1491272509 learned_addr learned my addr 172.21.15.63:0/1491272509 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:58.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc841d4b640 1 -- 172.21.15.63:0/1491272509 >> v1:172.21.15.6:6789/0 conn(0x7fc83c0a08a0 legacy=0x7fc83c13beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:58.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234846743 0 0) 0x7fc83c0a5b40 con 0x7fc83c0a4790 2024-09-16T05:40:58.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient(hunting): _negotiate 2024-09-16T05:40:58.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:58.058 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:58.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient(hunting): my global_id is 114292 2024-09-16T05:40:58.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc828003530 con 0x7fc83c0a4790 2024-09-16T05:40:58.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1917586198 0 0) 0x7fc828003530 con 0x7fc83c0a4790 2024-09-16T05:40:58.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 >> v1:172.21.15.6:6790/0 conn(0x7fc83c0a3ca0 legacy=0x7fc83c0a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:58.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 >> v1:172.21.15.6:6789/0 conn(0x7fc83c0a08a0 legacy=0x7fc83c13beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:58.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: _finish_hunting 0 2024-09-16T05:40:58.059 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 monclient: found mon.noname-c 2024-09-16T05:40:58.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:58.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:58.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c0a53e0 con 0x7fc83c0a4790 2024-09-16T05:40:58.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: _finish_auth 0 2024-09-16T05:40:58.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: _check_auth_tickets 2024-09-16T05:40:58.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:58.060 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc8340019a0 con 0x7fc83c0a4790 2024-09-16T05:40:58.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 5 monclient: authenticate success, global_id 114292 2024-09-16T05:40:58.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:58.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:58.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:58.061 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: dump: 2024-09-16T05:40:58.061 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:58.061 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:58.062 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:58.062 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:58.062 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:58.062 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:58.062 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:58.062 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:58.063 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:58.063 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:58.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:58.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc834001c10 con 0x7fc83c0a4790 2024-09-16T05:40:58.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:58.063 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:58.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 1 -- 172.21.15.63:0/1491272509 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc834002020 con 0x7fc83c0a4790 2024-09-16T05:40:58.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 4 set_mon_vals no callback set 2024-09-16T05:40:58.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:58.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:58.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc84154a640 10 monclient: dump: 2024-09-16T05:40:58.064 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:58.064 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:58.064 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient: shutdown 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- 172.21.15.63:0/1491272509 >> v1:172.21.15.63:6789/0 conn(0x7fc83c0a4790 legacy=0x7fc83c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- 172.21.15.63:0/1491272509 shutdown_connections 2024-09-16T05:40:58.065 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- 172.21.15.63:0/1491272509 >> 172.21.15.63:0/1491272509 conn(0x7fc83c0a67c0 msgr2=0x7fc83c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- 172.21.15.63:0/1491272509 shutdown_connections 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- 172.21.15.63:0/1491272509 wait complete. 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient: build_initial_monmap 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient: monmap: 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:58.066 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:58.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 Processor -- start 2024-09-16T05:40:58.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 1 -- start start 2024-09-16T05:40:58.067 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.049+0000 7fc8447b0640 10 monclient: init 2024-09-16T05:40:58.070 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient(hunting): picked mon.noname-b con 0x7fc83c0a08a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient(hunting): picked mon.noname-c con 0x7fc83c0a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient(hunting): picked mon.noname-a con 0x7fc83c0a4790 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc83c0a7d90 con 0x7fc83c0a4790 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc83c0a82e0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc83c0a0500 con 0x7fc83c0a3ca0 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient(hunting): authenticate will time out at 1314.832197s 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc84254c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc83c0a08a0 0x7fc83c1cad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49984/0 (socket says 172.21.15.63:49984) 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc84254c640 1 -- 172.21.15.63:0/2383510689 learned_addr learned my addr 172.21.15.63:0/2383510689 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:58.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc842d4d640 1 -- 172.21.15.63:0/2383510689 >> v1:172.21.15.6:6789/0 conn(0x7fc83c0a4790 legacy=0x7fc83c1d30c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2258551236 0 0) 0x7fc83c0a82e0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): my global_id is 114295 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc820003530 con 0x7fc83c0a08a0 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1657955295 0 0) 0x7fc820003530 con 0x7fc83c0a08a0 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc83c0a82e0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc8380019c0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:58.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): dump: 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2877836911 0 0) 0x7fc83c0a0500 con 0x7fc83c0a3ca0 2024-09-16T05:40:58.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient(hunting): my global_id is 6449 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc820003530 con 0x7fc83c0a3ca0 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4100306895 0 0) 0x7fc83c0a82e0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 >> v1:172.21.15.6:6790/0 conn(0x7fc83c0a3ca0 legacy=0x7fc83c136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 >> v1:172.21.15.6:6789/0 conn(0x7fc83c0a4790 legacy=0x7fc83c1d30c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 monclient: found mon.b 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:58.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c127160 con 0x7fc83c0a08a0 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient: _finish_auth 0 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 5 monclient: authenticate success, global_id 114295 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient: _renew_subs 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c127440 con 0x7fc83c0a08a0 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient: _renew_subs 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc83c137420 con 0x7fc83c0a08a0 2024-09-16T05:40:58.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc838003950 con 0x7fc83c0a08a0 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc838003ca0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc840d49640 4 set_mon_vals no callback set 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 10 monclient: dump: 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:58.076 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc824005090 con 0x7fc83c0a08a0 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc838003ec0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(250..250 src has 20..250) ==== 7355+0+0 (unknown 508688065 0 0) 0x7fc838045300 con 0x7fc83c0a08a0 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.053+0000 7fc841d4b640 10 monclient: get_auth_request con 0x7fc820036a50 auth_method 0 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.057+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc838015660 con 0x7fc83c0a08a0 2024-09-16T05:40:58.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.057+0000 7fc832ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:58.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.057+0000 7fc832ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:58.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.217+0000 7fc8447b0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T05:40:58.223 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.217+0000 7fc8447b0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T05:40:58.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.217+0000 7fc8447b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:58.224 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.217+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) -- 0x7fc824005380 con 0x7fc83c0a08a0 2024-09-16T05:40:58.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.813+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 removed item id -16 name 'r1' from crush map v251) ==== 118+0+0 (unknown 177091629 0 0) 0x7fc838019310 con 0x7fc83c0a08a0 2024-09-16T05:40:58.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.813+0000 7fc832ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T05:40:58.820 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.813+0000 7fc832ffd640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'r1' from crush map 2024-09-16T05:40:58.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.917+0000 7fc8447b0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T05:40:58.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.917+0000 7fc8447b0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T05:40:58.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.917+0000 7fc8447b0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:58.923 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.917+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) -- 0x7fc824002220 con 0x7fc83c0a08a0 2024-09-16T05:40:58.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.917+0000 7fc832ffd640 1 -- 172.21.15.63:0/2383510689 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 device 'r1' does not appear in the crush map v251) ==== 118+0+0 (unknown 1241059631 0 0) 0x7fc83801dbd0 con 0x7fc83c0a08a0 2024-09-16T05:40:58.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.917+0000 7fc832ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T05:40:58.924 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.917+0000 7fc832ffd640 10 monclient: _finish_command 3 = system:0 device 'r1' does not appear in the crush map 2024-09-16T05:40:58.924 INFO:tasks.workunit.client.0.smithi063.stderr:device 'r1' does not appear in the crush map 2024-09-16T05:40:58.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.921+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc820036a50 legacy=0x7fc820038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:58.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.921+0000 7fc8447b0640 10 monclient: shutdown 2024-09-16T05:40:58.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.921+0000 7fc8447b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:58.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.921+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 >> v1:172.21.15.63:6789/0 conn(0x7fc83c0a08a0 legacy=0x7fc83c1cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:58.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.921+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 shutdown_connections 2024-09-16T05:40:58.926 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.921+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 >> 172.21.15.63:0/2383510689 conn(0x7fc83c0a67c0 msgr2=0x7fc83c0a8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:58.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.921+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 shutdown_connections 2024-09-16T05:40:58.927 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:58.921+0000 7fc8447b0640 1 -- 172.21.15.63:0/2383510689 wait complete. 2024-09-16T05:40:58.941 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T05:40:59.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.057+0000 7f299c32f640 10 monclient: get_monmap_and_config 2024-09-16T05:40:59.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.057+0000 7f299c32f640 10 monclient: build_initial_monmap 2024-09-16T05:40:59.064 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.057+0000 7f299c32f640 10 monclient: monmap: 2024-09-16T05:40:59.064 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:40:59.064370+0000 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:40:59.064370+0000 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:40:59.065 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:59.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 Processor -- start 2024-09-16T05:40:59.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- start start 2024-09-16T05:40:59.071 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient: init 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient(hunting): picked mon.noname-b con 0x7f29940a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient(hunting): picked mon.noname-a con 0x7f29940a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient(hunting): picked mon.noname-c con 0x7f29940a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29940a4fd0 con 0x7f29940a08a0 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29940a5b40 con 0x7f29940a4790 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29940a1a70 con 0x7f29940a3ca0 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient(hunting): authenticate will time out at 1315.841370s 2024-09-16T05:40:59.072 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299a8cc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f29940a4790 0x7f299413f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49986/0 (socket says 172.21.15.63:49986) 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299a8cc640 1 -- 172.21.15.63:0/2574304302 learned_addr learned my addr 172.21.15.63:0/2574304302 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29998ca640 1 -- 172.21.15.63:0/2574304302 >> v1:172.21.15.6:6789/0 conn(0x7f29940a08a0 legacy=0x7f299413beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130441966 0 0) 0x7f29940a5b40 con 0x7f29940a4790 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient(hunting): _negotiate 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient(hunting): my global_id is 114298 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2980003530 con 0x7f29940a4790 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3279024911 0 0) 0x7f2980003530 con 0x7f29940a4790 2024-09-16T05:40:59.073 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 >> v1:172.21.15.6:6790/0 conn(0x7f29940a3ca0 legacy=0x7f29940a4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 >> v1:172.21.15.6:6789/0 conn(0x7f29940a08a0 legacy=0x7f299413beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: _finish_hunting 0 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 monclient: found mon.noname-c 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29940a53e0 con 0x7f29940a4790 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: _finish_auth 0 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: _check_auth_tickets 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 5 monclient: authenticate success, global_id 114298 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f298c0019a0 con 0x7f29940a4790 2024-09-16T05:40:59.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: dump: 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:40:59.075 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f298c001c90 con 0x7f29940a4790 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 1 -- 172.21.15.63:0/2574304302 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f298c0020a0 con 0x7f29940a4790 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient: get_monmap_and_config success 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 4 set_mon_vals no callback set 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f29990c9640 10 monclient: dump: 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:59.076 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient: shutdown 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- 172.21.15.63:0/2574304302 >> v1:172.21.15.63:6789/0 conn(0x7f29940a4790 legacy=0x7f299413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- 172.21.15.63:0/2574304302 shutdown_connections 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- 172.21.15.63:0/2574304302 >> 172.21.15.63:0/2574304302 conn(0x7f29940a67c0 msgr2=0x7f2994075470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- 172.21.15.63:0/2574304302 shutdown_connections 2024-09-16T05:40:59.077 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- 172.21.15.63:0/2574304302 wait complete. 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient: build_initial_monmap 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient: monmap: 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:59.078 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 Processor -- start 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 1 -- start start 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.061+0000 7f299c32f640 10 monclient: init 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient: _reopen_session rank -1 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient(hunting): picked mon.noname-a con 0x7f29940a08a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient(hunting): picked mon.noname-c con 0x7f29940a3ca0 addr v1:172.21.15.6:6790/0 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient(hunting): picked mon.noname-b con 0x7f29940a4790 addr v1:172.21.15.63:6789/0 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2994127520 con 0x7f29940a08a0 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2994125c90 con 0x7f29940a4790 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29940a8940 con 0x7f29940a3ca0 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient(hunting): _renew_subs 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient(hunting): authenticate will time out at 1315.844196s 2024-09-16T05:40:59.079 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299a8cc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f29940a4790 0x7f29941d3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49988/0 (socket says 172.21.15.63:49988) 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299a8cc640 1 -- 172.21.15.63:0/15329140 learned_addr learned my addr 172.21.15.63:0/15329140 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299a0cb640 1 -- 172.21.15.63:0/15329140 >> v1:172.21.15.6:6789/0 conn(0x7f29940a08a0 legacy=0x7f29941cb070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2916679059 0 0) 0x7f2994125c90 con 0x7f29940a4790 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): my global_id is 114301 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2978003530 con 0x7f29940a4790 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 777451800 0 0) 0x7f2978003530 con 0x7f29940a4790 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2994125c90 con 0x7f29940a4790 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f298c0017c0 con 0x7f29940a4790 2024-09-16T05:40:59.080 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): dump: 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:59.081 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 294783078 0 0) 0x7f29940a8940 con 0x7f29940a3ca0 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): _negotiate 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient(hunting): my global_id is 6455 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2978003530 con 0x7f29940a3ca0 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3232231139 0 0) 0x7f2994125c90 con 0x7f29940a4790 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 >> v1:172.21.15.6:6790/0 conn(0x7f29940a3ca0 legacy=0x7f29941370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 >> v1:172.21.15.6:6789/0 conn(0x7f29940a08a0 legacy=0x7f29941cb070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient: _finish_hunting 0 2024-09-16T05:40:59.082 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 monclient: found mon.b 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29940a8430 con 0x7f29940a4790 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient: _finish_auth 0 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient: _check_auth_tickets 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 5 monclient: authenticate success, global_id 114301 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient: _renew_subs 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2994137800 con 0x7f29940a4790 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient: _renew_subs 2024-09-16T05:40:59.083 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29941d4d00 con 0x7f29940a4790 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f298c001c90 con 0x7f29940a4790 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f298c0051f0 con 0x7f29940a4790 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f29988c8640 4 set_mon_vals no callback set 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 10 monclient: dump: 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:40:59.084 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f298c00ff90 con 0x7f29940a4790 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f297c005090 con 0x7f29940a4790 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(251..251 src has 20..251) ==== 7145+0+0 (unknown 3354586240 0 0) 0x7f298c045510 con 0x7f29940a4790 2024-09-16T05:40:59.085 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.065+0000 7f299a0cb640 10 monclient: get_auth_request con 0x7f2978036b30 auth_method 0 2024-09-16T05:40:59.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.069+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f298c015a60 con 0x7f29940a4790 2024-09-16T05:40:59.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.069+0000 7f298affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:40:59.086 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.069+0000 7f298affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:40:59.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.233+0000 7f299c32f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:40:59.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.233+0000 7f299c32f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:40:59.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.233+0000 7f299c32f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:59.239 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.233+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7f297c005380 con 0x7f29940a4790 2024-09-16T05:40:59.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.825+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v252) ==== 78+0+0 (unknown 4188627000 0 0) 0x7f298c015360 con 0x7f29940a4790 2024-09-16T05:40:59.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.825+0000 7f298affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:40:59.828 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.825+0000 7f298affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:40:59.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.929+0000 7f299c32f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:40:59.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.929+0000 7f299c32f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:40:59.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.929+0000 7f299c32f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:40:59.934 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:40:59.929+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7f297c002220 con 0x7f29940a4790 2024-09-16T05:41:00.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f298affd640 1 -- 172.21.15.63:0/15329140 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v253) ==== 78+0+0 (unknown 3093834269 0 0) 0x7f298c01deb0 con 0x7f29940a4790 2024-09-16T05:41:00.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f298affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T05:41:00.457 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f298affd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:41:00.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2978036b30 legacy=0x7f2978038ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:00.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f299c32f640 10 monclient: shutdown 2024-09-16T05:41:00.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f299c32f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:00.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 >> v1:172.21.15.63:6789/0 conn(0x7f29940a4790 legacy=0x7f29941d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:00.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 shutdown_connections 2024-09-16T05:41:00.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 >> 172.21.15.63:0/15329140 conn(0x7f29940a67c0 msgr2=0x7f29940a29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:00.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 shutdown_connections 2024-09-16T05:41:00.460 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.453+0000 7f299c32f640 1 -- 172.21.15.63:0/15329140 wait complete. 2024-09-16T05:41:00.480 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo OK 2024-09-16T05:41:00.481 INFO:tasks.workunit.client.0.smithi063.stdout:OK 2024-09-16T05:41:00.481 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:41:00.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:41:00.537 INFO:tasks.workunit:Running workunits matching mon/osd.sh on client.0... 2024-09-16T05:41:00.538 INFO:tasks.workunit:Running workunit mon/osd.sh... 2024-09-16T05:41:00.538 DEBUG:teuthology.orchestra.run.smithi063:workunit test mon/osd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-09-16T05:41:00.590 INFO:tasks.workunit.client.0.smithi063.stderr:+ set -e 2024-09-16T05:41:00.590 INFO:tasks.workunit.client.0.smithi063.stderr:+ uuidgen 2024-09-16T05:41:00.591 INFO:tasks.workunit.client.0.smithi063.stderr:+ ua=5a2f0584-0de7-4238-aba4-c51878b788e8 2024-09-16T05:41:00.592 INFO:tasks.workunit.client.0.smithi063.stderr:+ uuidgen 2024-09-16T05:41:00.592 INFO:tasks.workunit.client.0.smithi063.stderr:+ ub=83a33391-0f81-4765-a441-83e73ba7cfdd 2024-09-16T05:41:00.592 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd create 5a2f0584-0de7-4238-aba4-c51878b788e8 2024-09-16T05:41:00.716 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient: get_monmap_and_config 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient: build_initial_monmap 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient: monmap: 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:00.716831+0000 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:00.716831+0000 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:00.717 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:00.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 Processor -- start 2024-09-16T05:41:00.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- start start 2024-09-16T05:41:00.723 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient: init 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient(hunting): picked mon.noname-b con 0x7f5d2809b520 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2809eff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient(hunting): picked mon.noname-a con 0x7f5d28139640 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d280a0d10 con 0x7f5d28139640 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d2809c820 con 0x7f5d2809eff0 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d2809cb00 con 0x7f5d2809b520 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient(hunting): authenticate will time out at 1317.493928s 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2d65a640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f5d28139640 legacy=0x7f5d2813ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:00.724 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d27fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f5d2809eff0 0x7f5d28137f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49992/0 (socket says 172.21.15.63:49992) 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d27fff640 1 -- 172.21.15.63:0/1352769938 learned_addr learned my addr 172.21.15.63:0/1352769938 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1623569862 0 0) 0x7f5d2809c820 con 0x7f5d2809eff0 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient(hunting): _negotiate 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient(hunting): my global_id is 114304 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d10003530 con 0x7f5d2809eff0 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3201067308 0 0) 0x7f5d10003530 con 0x7f5d2809eff0 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 >> v1:172.21.15.6:6790/0 conn(0x7f5d2809b520 legacy=0x7f5d2809e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 >> v1:172.21.15.6:6789/0 conn(0x7f5d28139640 legacy=0x7f5d2813ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: _finish_hunting 0 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 monclient: found mon.noname-c 2024-09-16T05:41:00.725 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2812fbe0 con 0x7f5d2809eff0 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: _finish_auth 0 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: _check_auth_tickets 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d1c001620 con 0x7f5d2809eff0 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: dump: 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:00.726 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 5 monclient: authenticate success, global_id 114304 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d1c001880 con 0x7f5d2809eff0 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 1 -- 172.21.15.63:0/1352769938 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d1c001c50 con 0x7f5d2809eff0 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:00.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d277fe640 10 monclient: dump: 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:00.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 4 set_mon_vals no callback set 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 10 monclient: shutdown 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1352769938 >> v1:172.21.15.63:6789/0 conn(0x7f5d2809eff0 legacy=0x7f5d28137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1352769938 shutdown_connections 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1352769938 >> 172.21.15.63:0/1352769938 conn(0x7f5d280a1bf0 msgr2=0x7f5d280a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1352769938 shutdown_connections 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.713+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1352769938 wait complete. 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: build_initial_monmap 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: monmap: 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:00.729 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 1 Processor -- start 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 1 -- start start 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: init 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient(hunting): picked mon.noname-b con 0x7f5d2809b520 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient(hunting): picked mon.noname-a con 0x7f5d28139640 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:00.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2806e930 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d28137260 con 0x7f5d28139640 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d28072f10 con 0x7f5d2809b520 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d28136400 con 0x7f5d2806e930 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient(hunting): authenticate will time out at 1317.496993s 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2ce59640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f5d2809b520 0x7f5d2806db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49998/0 (socket says 172.21.15.63:49998) 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2ce59640 1 -- 172.21.15.63:0/1221935096 learned_addr learned my addr 172.21.15.63:0/1221935096 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d27fff640 1 -- 172.21.15.63:0/1221935096 >> v1:172.21.15.6:6789/0 conn(0x7f5d28139640 legacy=0x7f5d2806e220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 116662216 0 0) 0x7f5d28072f10 con 0x7f5d2809b520 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): _negotiate 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:00.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): my global_id is 114307 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d14003530 con 0x7f5d2809b520 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4243029422 0 0) 0x7f5d14003530 con 0x7f5d2809b520 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d28072f10 con 0x7f5d2809b520 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d18001980 con 0x7f5d2809b520 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): dump: 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:00.732 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2808671702 0 0) 0x7f5d28136400 con 0x7f5d2806e930 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): _negotiate 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient(hunting): my global_id is 6461 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d14003530 con 0x7f5d2806e930 2024-09-16T05:41:00.733 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2111623975 0 0) 0x7f5d28072f10 con 0x7f5d2809b520 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 >> v1:172.21.15.6:6790/0 conn(0x7f5d2806e930 legacy=0x7f5d281d3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 >> v1:172.21.15.6:6789/0 conn(0x7f5d28139640 legacy=0x7f5d2806e220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient: _finish_hunting 0 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 monclient: found mon.b 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d281366e0 con 0x7f5d2809b520 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient: _finish_auth 0 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient: _check_auth_tickets 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 5 monclient: authenticate success, global_id 114307 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: _renew_subs 2024-09-16T05:41:00.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d28135210 con 0x7f5d2809b520 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: _renew_subs 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d2806ede0 con 0x7f5d2809b520 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d18003950 con 0x7f5d2809b520 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d18003ca0 con 0x7f5d2809b520 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d26ffd640 4 set_mon_vals no callback set 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 10 monclient: dump: 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:00.735 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5d18003e40 con 0x7f5d2809b520 2024-09-16T05:41:00.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d08005090 con 0x7f5d2809b520 2024-09-16T05:41:00.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(253..253 src has 20..253) ==== 6817+0+0 (unknown 46030466 0 0) 0x7f5d18044fe0 con 0x7f5d2809b520 2024-09-16T05:41:00.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.717+0000 7f5d27fff640 10 monclient: get_auth_request con 0x7f5d14036b30 auth_method 0 2024-09-16T05:41:00.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.721+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5d18015550 con 0x7f5d2809b520 2024-09-16T05:41:00.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.721+0000 7f5d257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:00.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.721+0000 7f5d257fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:00.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.893+0000 7f5d2f0bd640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:00.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.893+0000 7f5d2f0bd640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:00.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.893+0000 7f5d2f0bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:00.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:00.893+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"} v 0) -- 0x7f5d08005380 con 0x7f5d2809b520 2024-09-16T05:41:01.838 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.833+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}]=0 v254) ==== 106+0+3 (unknown 797326506 0 944643765) 0x7f5d18019200 con 0x7f5d2809b520 2024-09-16T05:41:01.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.833+0000 7f5d257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:01.839 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.833+0000 7f5d257fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:41:01.951 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.945+0000 7f5d2f0bd640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:01.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.945+0000 7f5d2f0bd640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:01.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.945+0000 7f5d2f0bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:01.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.945+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"} v 0) -- 0x7f5d08005be0 con 0x7f5d2809b520 2024-09-16T05:41:01.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d257fa640 1 -- 172.21.15.63:0/1221935096 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}]=0 v254) ==== 106+0+3 (unknown 797326506 0 944643765) 0x7f5d1801dac0 con 0x7f5d2809b520 2024-09-16T05:41:01.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d257fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:01.952 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d257fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:41:01.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5d14036b30 legacy=0x7f5d14038ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:01.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d2f0bd640 10 monclient: shutdown 2024-09-16T05:41:01.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d2f0bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:01.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 >> v1:172.21.15.63:6789/0 conn(0x7f5d2809b520 legacy=0x7f5d2806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:01.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 shutdown_connections 2024-09-16T05:41:01.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 >> 172.21.15.63:0/1221935096 conn(0x7f5d280a1bf0 msgr2=0x7f5d2809d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:01.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 shutdown_connections 2024-09-16T05:41:01.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:01.949+0000 7f5d2f0bd640 1 -- 172.21.15.63:0/1221935096 wait complete. 2024-09-16T05:41:01.969 INFO:tasks.workunit.client.0.smithi063.stderr:+ na=12 2024-09-16T05:41:01.969 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd create 5a2f0584-0de7-4238-aba4-c51878b788e8 2024-09-16T05:41:02.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient: get_monmap_and_config 2024-09-16T05:41:02.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient: build_initial_monmap 2024-09-16T05:41:02.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient: monmap: 2024-09-16T05:41:02.089 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:02.089 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.089 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:02.089169+0000 2024-09-16T05:41:02.089 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:02.089169+0000 2024-09-16T05:41:02.089 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:02.090 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.090 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:02.090 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:02.090 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:02.090 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 1 Processor -- start 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 1 -- start start 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient: init 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient(hunting): picked mon.noname-a con 0x7f739009ac90 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient(hunting): picked mon.noname-c con 0x7f739009b070 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient(hunting): picked mon.noname-b con 0x7f739013a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73900722a0 con 0x7f739009ac90 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73900a0510 con 0x7f739009b070 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f739009e660 con 0x7f739013a9a0 2024-09-16T05:41:02.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient(hunting): authenticate will time out at 1318.866972s 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7394884640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f739009b070 0x7f739009b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50002/0 (socket says 172.21.15.63:50002) 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7394884640 1 -- 172.21.15.63:0/4158776906 learned_addr learned my addr 172.21.15.63:0/4158776906 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7395085640 1 -- 172.21.15.63:0/4158776906 >> v1:172.21.15.6:6789/0 conn(0x7f739009ac90 legacy=0x7f739009f6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317696818 0 0) 0x7f73900a0510 con 0x7f739009b070 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient(hunting): _negotiate 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:02.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient(hunting): my global_id is 114310 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f737c003530 con 0x7f739009b070 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 991943568 0 0) 0x7f737c003530 con 0x7f739009b070 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 >> v1:172.21.15.6:6790/0 conn(0x7f739013a9a0 legacy=0x7f739013cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 >> v1:172.21.15.6:6789/0 conn(0x7f739009ac90 legacy=0x7f739009f6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: _finish_hunting 0 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 monclient: found mon.noname-c 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73901414e0 con 0x7f739009b070 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: _finish_auth 0 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: _check_auth_tickets 2024-09-16T05:41:02.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7380001620 con 0x7f739009b070 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 5 monclient: authenticate success, global_id 114310 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: dump: 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.096 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7380001910 con 0x7f739009b070 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 1 -- 172.21.15.63:0/4158776906 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7380001ce0 con 0x7f739009b070 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:02.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f73972e9640 4 set_mon_vals no callback set 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.085+0000 7f7387fff640 10 monclient: dump: 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:02.098 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: shutdown 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- 172.21.15.63:0/4158776906 >> v1:172.21.15.63:6789/0 conn(0x7f739009b070 legacy=0x7f739009b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- 172.21.15.63:0/4158776906 shutdown_connections 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- 172.21.15.63:0/4158776906 >> 172.21.15.63:0/4158776906 conn(0x7f73900a1bf0 msgr2=0x7f73900a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- 172.21.15.63:0/4158776906 shutdown_connections 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- 172.21.15.63:0/4158776906 wait complete. 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: build_initial_monmap 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: monmap: 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:02.099 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:02.100 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:02.100 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:02.100 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.100 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:02.100 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:02.100 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:02.100 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 Processor -- start 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- start start 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: init 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient(hunting): picked mon.noname-b con 0x7f739009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient(hunting): picked mon.noname-a con 0x7f739009b070 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient(hunting): picked mon.noname-c con 0x7f739013a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7390136790 con 0x7f739009b070 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7390137af0 con 0x7f739009ac90 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7390138fa0 con 0x7f739013a9a0 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient(hunting): authenticate will time out at 1318.869958s 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7395085640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f739009ac90 0x7f7390135970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50006/0 (socket says 172.21.15.63:50006) 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7395085640 1 -- 172.21.15.63:0/1468309464 learned_addr learned my addr 172.21.15.63:0/1468309464 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7394884640 1 -- 172.21.15.63:0/1468309464 >> v1:172.21.15.6:6789/0 conn(0x7f739009b070 legacy=0x7f7390136080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:02.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 372638692 0 0) 0x7f7390137af0 con 0x7f739009ac90 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): my global_id is 114313 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7374003530 con 0x7f739009ac90 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 890619063 0 0) 0x7f7374003530 con 0x7f739009ac90 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7390137af0 con 0x7f739009ac90 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f738c004190 con 0x7f739009ac90 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): dump: 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:02.106 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109669645 0 0) 0x7f7390138fa0 con 0x7f739013a9a0 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:41:02.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient(hunting): my global_id is 6467 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7374003530 con 0x7f739013a9a0 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3890496299 0 0) 0x7f7390137af0 con 0x7f739009ac90 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 >> v1:172.21.15.6:6790/0 conn(0x7f739013a9a0 legacy=0x7f73901d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 >> v1:172.21.15.6:6789/0 conn(0x7f739009b070 legacy=0x7f7390136080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 monclient: found mon.b 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739013e170 con 0x7f739009ac90 2024-09-16T05:41:02.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient: _finish_auth 0 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 5 monclient: authenticate success, global_id 114313 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: _renew_subs 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73901389e0 con 0x7f739009ac90 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: _renew_subs 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7390137870 con 0x7f739009ac90 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f738c004480 con 0x7f739009ac90 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f738c004850 con 0x7f739009ac90 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:02.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73877fe640 4 set_mon_vals no callback set 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 10 monclient: dump: 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f738c004a70 con 0x7f739009ac90 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7370005090 con 0x7f739009ac90 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.089+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(254..254 src has 20..254) ==== 6817+0+0 (unknown 1878403883 0 0) 0x7f738c044ed0 con 0x7f739009ac90 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.093+0000 7f7394884640 10 monclient: get_auth_request con 0x7f7374036a80 auth_method 0 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.097+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f738c015580 con 0x7f739009ac90 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.097+0000 7f7385ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:02.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.097+0000 7f7385ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:02.269 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.265+0000 7f73972e9640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:02.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.265+0000 7f73972e9640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:02.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.265+0000 7f73972e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.265+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"} v 0) -- 0x7f7370005380 con 0x7f739009ac90 2024-09-16T05:41:02.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.265+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}]=0 v254) ==== 106+0+3 (unknown 797326506 0 944643765) 0x7f738c019230 con 0x7f739009ac90 2024-09-16T05:41:02.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.265+0000 7f7385ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:02.270 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.265+0000 7f7385ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:41:02.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:02.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:02.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"} v 0) -- 0x7f7370005be0 con 0x7f739009ac90 2024-09-16T05:41:02.377 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f7385ffb640 1 -- 172.21.15.63:0/1468309464 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}]=0 v254) ==== 106+0+3 (unknown 797326506 0 944643765) 0x7f738c01daf0 con 0x7f739009ac90 2024-09-16T05:41:02.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f7385ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:02.378 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f7385ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:41:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7374036a80 legacy=0x7f7374038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 10 monclient: shutdown 2024-09-16T05:41:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 >> v1:172.21.15.63:6789/0 conn(0x7f739009ac90 legacy=0x7f7390135970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 shutdown_connections 2024-09-16T05:41:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 >> 172.21.15.63:0/1468309464 conn(0x7f73900a1bf0 msgr2=0x7f739013fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 shutdown_connections 2024-09-16T05:41:02.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.373+0000 7f73972e9640 1 -- 172.21.15.63:0/1468309464 wait complete. 2024-09-16T05:41:02.394 INFO:tasks.workunit.client.0.smithi063.stderr:+ test 12 -eq 12 2024-09-16T05:41:02.394 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd create 83a33391-0f81-4765-a441-83e73ba7cfdd 2024-09-16T05:41:02.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient: get_monmap_and_config 2024-09-16T05:41:02.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient: build_initial_monmap 2024-09-16T05:41:02.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient: monmap: 2024-09-16T05:41:02.514 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:02.514 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.514 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:02.514330+0000 2024-09-16T05:41:02.514 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:02.514330+0000 2024-09-16T05:41:02.514 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:02.515 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.515 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:02.515 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:02.515 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:02.515 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 1 Processor -- start 2024-09-16T05:41:02.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 1 -- start start 2024-09-16T05:41:02.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient: init 2024-09-16T05:41:02.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:02.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:41:02.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient(hunting): picked mon.noname-a con 0x7f73ec133810 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:02.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient(hunting): picked mon.noname-c con 0x7f73ec133bf0 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient(hunting): picked mon.noname-b con 0x7f73ec134020 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ec09d3c0 con 0x7f73ec133810 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ec09d910 con 0x7f73ec133bf0 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ec0a1560 con 0x7f73ec134020 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73f25d6640 10 monclient(hunting): authenticate will time out at 1319.291497s 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73ebfff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f73ec133810 legacy=0x7f73ec09dd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73eb7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f73ec133bf0 0x7f73ec138700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50008/0 (socket says 172.21.15.63:50008) 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73eb7fe640 1 -- 172.21.15.63:0/3266096297 learned_addr learned my addr 172.21.15.63:0/3266096297 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.509+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2784787837 0 0) 0x7f73ec09d910 con 0x7f73ec133bf0 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:02.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient(hunting): my global_id is 114316 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73d4003530 con 0x7f73ec133bf0 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2954681964 0 0) 0x7f73d4003530 con 0x7f73ec133bf0 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 >> v1:172.21.15.6:6790/0 conn(0x7f73ec134020 legacy=0x7f73ec13be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 >> v1:172.21.15.6:6789/0 conn(0x7f73ec133810 legacy=0x7f73ec09dd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 monclient: found mon.noname-c 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ec0a0a60 con 0x7f73ec133bf0 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: _finish_auth 0 2024-09-16T05:41:02.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f73e0001620 con 0x7f73ec133bf0 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 5 monclient: authenticate success, global_id 114316 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: dump: 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:02.521 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73e0001980 con 0x7f73ec133bf0 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 4 set_mon_vals no callback set 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 1 -- 172.21.15.63:0/3266096297 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f73e00048b0 con 0x7f73ec133bf0 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:02.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73eaffd640 10 monclient: dump: 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: shutdown 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:02.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- 172.21.15.63:0/3266096297 >> v1:172.21.15.63:6789/0 conn(0x7f73ec133bf0 legacy=0x7f73ec138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- 172.21.15.63:0/3266096297 shutdown_connections 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- 172.21.15.63:0/3266096297 >> 172.21.15.63:0/3266096297 conn(0x7f73ec0a1bf0 msgr2=0x7f73ec0a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- 172.21.15.63:0/3266096297 shutdown_connections 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- 172.21.15.63:0/3266096297 wait complete. 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: build_initial_monmap 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: monmap: 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:02.525 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:02.525 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:02.525 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 Processor -- start 2024-09-16T05:41:02.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- start start 2024-09-16T05:41:02.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: init 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient(hunting): picked mon.noname-a con 0x7f73ec133810 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient(hunting): picked mon.noname-c con 0x7f73ec133bf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient(hunting): picked mon.noname-b con 0x7f73ec134020 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ec13d090 con 0x7f73ec133810 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ec130390 con 0x7f73ec134020 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ec131540 con 0x7f73ec133bf0 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient(hunting): authenticate will time out at 1319.294763s 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73ebfff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f73ec133810 legacy=0x7f73ec06db10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f0b73640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f73ec134020 0x7f73ec06e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50018/0 (socket says 172.21.15.63:50018) 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f0b73640 1 -- 172.21.15.63:0/27204021 learned_addr learned my addr 172.21.15.63:0/27204021 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4239968846 0 0) 0x7f73ec130390 con 0x7f73ec134020 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): my global_id is 114319 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73d8003530 con 0x7f73ec134020 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1553465671 0 0) 0x7f73d8003530 con 0x7f73ec134020 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73ec130390 con 0x7f73ec134020 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f73e4001cd0 con 0x7f73ec134020 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): dump: 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 484745300 0 0) 0x7f73ec131540 con 0x7f73ec133bf0 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient(hunting): my global_id is 6473 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73d8003530 con 0x7f73ec133bf0 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2779454661 0 0) 0x7f73ec130390 con 0x7f73ec134020 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 >> v1:172.21.15.6:6790/0 conn(0x7f73ec133bf0 legacy=0x7f73ec06e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 >> v1:172.21.15.6:6789/0 conn(0x7f73ec133810 legacy=0x7f73ec06db10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 monclient: found mon.b 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ec130ca0 con 0x7f73ec134020 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient: _finish_auth 0 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 5 monclient: authenticate success, global_id 114319 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: _renew_subs 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73ec130f80 con 0x7f73ec134020 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: _renew_subs 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f73ec06f810 con 0x7f73ec134020 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73e40020c0 con 0x7f73ec134020 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:02.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f73e4003e70 con 0x7f73ec134020 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73ea7fc640 4 set_mon_vals no callback set 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.513+0000 7f73e8ff9640 10 monclient: dump: 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:02.534 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.517+0000 7f73f25d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.517+0000 7f73f25d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.517+0000 7f73f25d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.517+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73cc005090 con 0x7f73ec134020 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.517+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f73e4004090 con 0x7f73ec134020 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.517+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(254..254 src has 20..254) ==== 6817+0+0 (unknown 1878403883 0 0) 0x7f73e4045340 con 0x7f73ec134020 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.517+0000 7f73ebfff640 10 monclient: get_auth_request con 0x7f73d8036a80 auth_method 0 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.521+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f73ec06f810 con 0x7f73ec134020 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.521+0000 7f73e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:02.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.521+0000 7f73e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:02.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.693+0000 7f73f25d6640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:02.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.693+0000 7f73f25d6640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:02.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.693+0000 7f73f25d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.700 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.693+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"} v 0) -- 0x7f73cc005380 con 0x7f73ec134020 2024-09-16T05:41:02.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.845+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}]=0 v255) ==== 106+0+3 (unknown 1593470610 0 1367428999) 0x7f73e4019560 con 0x7f73ec134020 2024-09-16T05:41:02.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.845+0000 7f73e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:02.849 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.845+0000 7f73e8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:41:02.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.949+0000 7f73f25d6640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:02.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.949+0000 7f73f25d6640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:02.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.949+0000 7f73f25d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:02.954 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.949+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"} v 0) -- 0x7f73cc005be0 con 0x7f73ec134020 2024-09-16T05:41:02.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.949+0000 7f73e8ff9640 1 -- 172.21.15.63:0/27204021 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}]=0 v255) ==== 106+0+3 (unknown 1593470610 0 1367428999) 0x7f73e401de20 con 0x7f73ec134020 2024-09-16T05:41:02.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.949+0000 7f73e8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:02.955 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.949+0000 7f73e8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:41:02.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.953+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 >> v1:172.21.15.63:6812/3643207211 conn(0x7f73d8036a80 legacy=0x7f73d8038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.953+0000 7f73f25d6640 10 monclient: shutdown 2024-09-16T05:41:02.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.953+0000 7f73f25d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:02.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.953+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 >> v1:172.21.15.63:6789/0 conn(0x7f73ec134020 legacy=0x7f73ec06e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:02.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.953+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 shutdown_connections 2024-09-16T05:41:02.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.953+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 >> 172.21.15.63:0/27204021 conn(0x7f73ec0a1bf0 msgr2=0x7f73ec0a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:02.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.953+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 shutdown_connections 2024-09-16T05:41:02.957 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:02.953+0000 7f73f25d6640 1 -- 172.21.15.63:0/27204021 wait complete. 2024-09-16T05:41:02.970 INFO:tasks.workunit.client.0.smithi063.stderr:+ nb=14 2024-09-16T05:41:02.971 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd create 83a33391-0f81-4765-a441-83e73ba7cfdd 2024-09-16T05:41:03.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: get_monmap_and_config 2024-09-16T05:41:03.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: build_initial_monmap 2024-09-16T05:41:03.088 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: monmap: 2024-09-16T05:41:03.088 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:03.088661+0000 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:03.088661+0000 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:03.089 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 Processor -- start 2024-09-16T05:41:03.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- start start 2024-09-16T05:41:03.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: init 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient(hunting): picked mon.noname-c con 0x7fe048136da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient(hunting): picked mon.noname-a con 0x7fe0480a4ea0 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient(hunting): picked mon.noname-b con 0x7fe04809fe80 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0480a48e0 con 0x7fe0480a4ea0 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0480a4440 con 0x7fe048136da0 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0480a4b60 con 0x7fe04809fe80 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient(hunting): authenticate will time out at 1319.865767s 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04659c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe048136da0 0x7fe048137180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50032/0 (socket says 172.21.15.63:50032) 2024-09-16T05:41:03.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04659c640 1 -- 172.21.15.63:0/3722242072 learned_addr learned my addr 172.21.15.63:0/3722242072 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe045d9b640 1 -- 172.21.15.63:0/3722242072 >> v1:172.21.15.6:6789/0 conn(0x7fe0480a4ea0 legacy=0x7fe04813bd80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1646529360 0 0) 0x7fe0480a4440 con 0x7fe048136da0 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient(hunting): _negotiate 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient(hunting): my global_id is 114322 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe030003530 con 0x7fe048136da0 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3643126118 0 0) 0x7fe030003530 con 0x7fe048136da0 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 >> v1:172.21.15.6:6790/0 conn(0x7fe04809fe80 legacy=0x7fe04813f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 >> v1:172.21.15.6:6789/0 conn(0x7fe0480a4ea0 legacy=0x7fe04813bd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: _finish_hunting 0 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 monclient: found mon.noname-c 2024-09-16T05:41:03.099 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe048137890 con 0x7fe048136da0 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: _finish_auth 0 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: _check_auth_tickets 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0400019a0 con 0x7fe048136da0 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: dump: 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.100 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 5 monclient: authenticate success, global_id 114322 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe040001d00 con 0x7fe048136da0 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 1 -- 172.21.15.63:0/3722242072 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0400020d0 con 0x7fe048136da0 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:03.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 4 set_mon_vals no callback set 2024-09-16T05:41:03.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:03.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:03.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04559a640 10 monclient: dump: 2024-09-16T05:41:03.102 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:03.102 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.102 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.102 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: shutdown 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- 172.21.15.63:0/3722242072 >> v1:172.21.15.63:6789/0 conn(0x7fe048136da0 legacy=0x7fe048137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- 172.21.15.63:0/3722242072 shutdown_connections 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- 172.21.15.63:0/3722242072 >> 172.21.15.63:0/3722242072 conn(0x7fe0480a5e80 msgr2=0x7fe0480a6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- 172.21.15.63:0/3722242072 shutdown_connections 2024-09-16T05:41:03.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 1 -- 172.21.15.63:0/3722242072 wait complete. 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: build_initial_monmap 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.085+0000 7fe04c979640 10 monclient: monmap: 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 1 Processor -- start 2024-09-16T05:41:03.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 1 -- start start 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: init 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient(hunting): picked mon.noname-a con 0x7fe04809fe80 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient(hunting): picked mon.noname-c con 0x7fe0480a4ea0 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient(hunting): picked mon.noname-b con 0x7fe048136da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe048122230 con 0x7fe04809fe80 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe048122510 con 0x7fe048136da0 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0481cbe40 con 0x7fe0480a4ea0 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient(hunting): authenticate will time out at 1319.868771s 2024-09-16T05:41:03.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe046d9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe048136da0 0x7fe0481d35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50048/0 (socket says 172.21.15.63:50048) 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04659c640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fe04809fe80 legacy=0x7fe0481cb020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe046d9d640 1 -- 172.21.15.63:0/3415051825 learned_addr learned my addr 172.21.15.63:0/3415051825 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2870917728 0 0) 0x7fe048122510 con 0x7fe048136da0 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): my global_id is 114325 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe028003530 con 0x7fe048136da0 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2854396749 0 0) 0x7fe028003530 con 0x7fe048136da0 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe048122510 con 0x7fe048136da0 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe038001900 con 0x7fe048136da0 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:03.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): dump: 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4267557484 0 0) 0x7fe0481cbe40 con 0x7fe0480a4ea0 2024-09-16T05:41:03.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient(hunting): my global_id is 6479 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe028003530 con 0x7fe0480a4ea0 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4019030659 0 0) 0x7fe048122510 con 0x7fe048136da0 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 >> v1:172.21.15.6:6790/0 conn(0x7fe0480a4ea0 legacy=0x7fe0481cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 >> v1:172.21.15.6:6789/0 conn(0x7fe04809fe80 legacy=0x7fe0481cb020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 monclient: found mon.b 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0480a0570 con 0x7fe048136da0 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient: _finish_auth 0 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 5 monclient: authenticate success, global_id 114325 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: _renew_subs 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0480a7280 con 0x7fe048136da0 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: _renew_subs 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0481357a0 con 0x7fe048136da0 2024-09-16T05:41:03.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe038001d10 con 0x7fe048136da0 2024-09-16T05:41:03.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:03.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe038003af0 con 0x7fe048136da0 2024-09-16T05:41:03.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:03.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:03.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 10 monclient: dump: 2024-09-16T05:41:03.111 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:03.111 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.111 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe044d99640 4 set_mon_vals no callback set 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe038003c90 con 0x7fe048136da0 2024-09-16T05:41:03.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe02c005090 con 0x7fe048136da0 2024-09-16T05:41:03.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(255..255 src has 20..255) ==== 6817+0+0 (unknown 3080050147 0 0) 0x7fe038044f20 con 0x7fe048136da0 2024-09-16T05:41:03.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.089+0000 7fe04659c640 10 monclient: get_auth_request con 0x7fe028036ad0 auth_method 0 2024-09-16T05:41:03.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.093+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe038015590 con 0x7fe048136da0 2024-09-16T05:41:03.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.093+0000 7fe036ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:03.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.093+0000 7fe036ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:03.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.269+0000 7fe04c979640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:03.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.269+0000 7fe04c979640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:03.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.269+0000 7fe04c979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.269+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"} v 0) -- 0x7fe02c005380 con 0x7fe048136da0 2024-09-16T05:41:03.275 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.269+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}]=0 v255) ==== 106+0+3 (unknown 1593470610 0 1367428999) 0x7fe038014e90 con 0x7fe048136da0 2024-09-16T05:41:03.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.269+0000 7fe036ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:03.276 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.269+0000 7fe036ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:41:03.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.373+0000 7fe04c979640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:03.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.373+0000 7fe04c979640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:03.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.373+0000 7fe04c979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.373+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"} v 0) -- 0x7fe02c005be0 con 0x7fe048136da0 2024-09-16T05:41:03.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.373+0000 7fe036ffd640 1 -- 172.21.15.63:0/3415051825 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}]=0 v255) ==== 106+0+3 (unknown 1593470610 0 1367428999) 0x7fe03801da40 con 0x7fe048136da0 2024-09-16T05:41:03.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe036ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "83a33391-0f81-4765-a441-83e73ba7cfdd"}] 2024-09-16T05:41:03.380 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe036ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:41:03.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe028036ad0 legacy=0x7fe028038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe04c979640 10 monclient: shutdown 2024-09-16T05:41:03.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe04c979640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:03.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 >> v1:172.21.15.63:6789/0 conn(0x7fe048136da0 legacy=0x7fe0481d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 shutdown_connections 2024-09-16T05:41:03.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 >> 172.21.15.63:0/3415051825 conn(0x7fe0480a5e80 msgr2=0x7fe048135dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:03.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 shutdown_connections 2024-09-16T05:41:03.382 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.377+0000 7fe04c979640 1 -- 172.21.15.63:0/3415051825 wait complete. 2024-09-16T05:41:03.395 INFO:tasks.workunit.client.0.smithi063.stderr:+ test 14 -eq 14 2024-09-16T05:41:03.395 INFO:tasks.workunit.client.0.smithi063.stderr:+ test 14 -ne 12 2024-09-16T05:41:03.395 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd rm 12 2024-09-16T05:41:03.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.505+0000 7f821bf69640 10 monclient: get_monmap_and_config 2024-09-16T05:41:03.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.505+0000 7f821bf69640 10 monclient: build_initial_monmap 2024-09-16T05:41:03.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.505+0000 7f821bf69640 10 monclient: monmap: 2024-09-16T05:41:03.512 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:03.512 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.513 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:03.512395+0000 2024-09-16T05:41:03.513 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:03.512395+0000 2024-09-16T05:41:03.513 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:03.513 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.513 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:03.513 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:03.513 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:03.513 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 Processor -- start 2024-09-16T05:41:03.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- start start 2024-09-16T05:41:03.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient: init 2024-09-16T05:41:03.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:03.521 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:41:03.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient(hunting): picked mon.noname-b con 0x7f821409ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:03.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient(hunting): picked mon.noname-c con 0x7f821409b070 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:03.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient(hunting): picked mon.noname-a con 0x7f821413a9a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:03.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82140722a0 con 0x7f821413a9a0 2024-09-16T05:41:03.522 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82140a0510 con 0x7f821409b070 2024-09-16T05:41:03.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821409e660 con 0x7f821409ac90 2024-09-16T05:41:03.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:03.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient(hunting): authenticate will time out at 1320.289320s 2024-09-16T05:41:03.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8219504640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f821409b070 0x7f821409b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50050/0 (socket says 172.21.15.63:50050) 2024-09-16T05:41:03.523 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821a506640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f821413a9a0 legacy=0x7f821413cd90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:03.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8219504640 1 -- 172.21.15.63:0/3734151125 learned_addr learned my addr 172.21.15.63:0/3734151125 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:03.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744020347 0 0) 0x7f82140a0510 con 0x7f821409b070 2024-09-16T05:41:03.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient(hunting): _negotiate 2024-09-16T05:41:03.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:03.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:03.524 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient(hunting): my global_id is 114328 2024-09-16T05:41:03.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8200003530 con 0x7f821409b070 2024-09-16T05:41:03.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3790191141 0 0) 0x7f8200003530 con 0x7f821409b070 2024-09-16T05:41:03.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 >> v1:172.21.15.6:6790/0 conn(0x7f821409ac90 legacy=0x7f821409f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 >> v1:172.21.15.6:6789/0 conn(0x7f821413a9a0 legacy=0x7f821413cd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:03.525 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: _finish_hunting 0 2024-09-16T05:41:03.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 monclient: found mon.noname-c 2024-09-16T05:41:03.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:03.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82141414e0 con 0x7f821409b070 2024-09-16T05:41:03.526 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: _finish_auth 0 2024-09-16T05:41:03.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: _check_auth_tickets 2024-09-16T05:41:03.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:03.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8204001620 con 0x7f821409b070 2024-09-16T05:41:03.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 5 monclient: authenticate success, global_id 114328 2024-09-16T05:41:03.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:03.527 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:03.528 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: dump: 2024-09-16T05:41:03.528 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:03.528 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.528 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.528 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.528 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:03.529 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.529 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:03.529 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:03.529 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:03.529 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:03.529 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8204001910 con 0x7f821409b070 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 1 -- 172.21.15.63:0/3734151125 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8204001ce0 con 0x7f821409b070 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 4 set_mon_vals no callback set 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f8218d03640 10 monclient: dump: 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:03.530 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient: shutdown 2024-09-16T05:41:03.531 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:03.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- 172.21.15.63:0/3734151125 >> v1:172.21.15.63:6789/0 conn(0x7f821409b070 legacy=0x7f821409b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- 172.21.15.63:0/3734151125 shutdown_connections 2024-09-16T05:41:03.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- 172.21.15.63:0/3734151125 >> 172.21.15.63:0/3734151125 conn(0x7f82140a1bf0 msgr2=0x7f82140a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:03.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- 172.21.15.63:0/3734151125 shutdown_connections 2024-09-16T05:41:03.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 -- 172.21.15.63:0/3734151125 wait complete. 2024-09-16T05:41:03.532 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient: build_initial_monmap 2024-09-16T05:41:03.533 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 10 monclient: monmap: 2024-09-16T05:41:03.533 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:03.533 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:03.533 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:03.533 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:03.534 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:03.534 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.534 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:03.534 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:03.535 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:03.535 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.509+0000 7f821bf69640 1 Processor -- start 2024-09-16T05:41:03.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 1 -- start start 2024-09-16T05:41:03.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: init 2024-09-16T05:41:03.535 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:03.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:41:03.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient(hunting): picked mon.noname-b con 0x7f821409ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:03.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient(hunting): picked mon.noname-a con 0x7f821409b070 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:03.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient(hunting): picked mon.noname-c con 0x7f821413a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:03.536 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821413a100 con 0x7f821409b070 2024-09-16T05:41:03.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821409d010 con 0x7f821409ac90 2024-09-16T05:41:03.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821413e0b0 con 0x7f821413a9a0 2024-09-16T05:41:03.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:03.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f8219504640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f821409b070 legacy=0x7f821409c900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:03.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient(hunting): authenticate will time out at 1320.292579s 2024-09-16T05:41:03.537 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f8219d05640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f821409ac90 0x7f821409c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50052/0 (socket says 172.21.15.63:50052) 2024-09-16T05:41:03.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f8219d05640 1 -- 172.21.15.63:0/1902086868 learned_addr learned my addr 172.21.15.63:0/1902086868 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:03.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3889755220 0 0) 0x7f821409d010 con 0x7f821409ac90 2024-09-16T05:41:03.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:41:03.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:03.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:03.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): my global_id is 114331 2024-09-16T05:41:03.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81f8003530 con 0x7f821409ac90 2024-09-16T05:41:03.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2718556058 0 0) 0x7f81f8003530 con 0x7f821409ac90 2024-09-16T05:41:03.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f821409d010 con 0x7f821409ac90 2024-09-16T05:41:03.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f82100019d0 con 0x7f821409ac90 2024-09-16T05:41:03.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:03.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:03.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): dump: 2024-09-16T05:41:03.540 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:03.540 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.540 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.540 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.540 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:03.541 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.541 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:03.541 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:03.541 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:03.541 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:03.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3652764210 0 0) 0x7f821413e0b0 con 0x7f821413a9a0 2024-09-16T05:41:03.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:41:03.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:03.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:03.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient(hunting): my global_id is 6485 2024-09-16T05:41:03.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81f8003530 con 0x7f821413a9a0 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2398209993 0 0) 0x7f821409d010 con 0x7f821409ac90 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 >> v1:172.21.15.6:6790/0 conn(0x7f821413a9a0 legacy=0x7f82141d3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 >> v1:172.21.15.6:6789/0 conn(0x7f821409b070 legacy=0x7f821409c900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 monclient: found mon.b 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8214141260 con 0x7f821409ac90 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:41:03.543 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:41:03.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:03.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 5 monclient: authenticate success, global_id 114331 2024-09-16T05:41:03.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: _renew_subs 2024-09-16T05:41:03.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f821411b720 con 0x7f821409ac90 2024-09-16T05:41:03.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: _renew_subs 2024-09-16T05:41:03.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f821409d270 con 0x7f821409ac90 2024-09-16T05:41:03.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8210003950 con 0x7f821409ac90 2024-09-16T05:41:03.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:03.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820bfff640 4 set_mon_vals no callback set 2024-09-16T05:41:03.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8210003d20 con 0x7f821409ac90 2024-09-16T05:41:03.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:03.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:03.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 10 monclient: dump: 2024-09-16T05:41:03.547 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:03.548 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:03.548 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.548 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:03.548 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:03.548 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:03.549 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:03.549 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:03.549 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:03.549 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:03.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:03.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:03.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81fc005090 con 0x7f821409ac90 2024-09-16T05:41:03.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8210003f40 con 0x7f821409ac90 2024-09-16T05:41:03.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(255..255 src has 20..255) ==== 6817+0+0 (unknown 3080050147 0 0) 0x7f82100450d0 con 0x7f821409ac90 2024-09-16T05:41:03.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.513+0000 7f8219504640 10 monclient: get_auth_request con 0x7f81f8036a80 auth_method 0 2024-09-16T05:41:03.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.517+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f821411b720 con 0x7f821409ac90 2024-09-16T05:41:03.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.517+0000 7f820a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:03.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.517+0000 7f820a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:03.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.685+0000 7f821bf69640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:03.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.685+0000 7f821bf69640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:03.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.685+0000 7f821bf69640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.689 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.685+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) -- 0x7f81fc005380 con 0x7f821409ac90 2024-09-16T05:41:03.690 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:41:03.685+0000 7ff77aa64640 -1 mon.b@1(leader).osd e255 definitely_dead 0 2024-09-16T05:41:03.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.845+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 removed osd.12 v256) ==== 83+0+0 (unknown 2709489590 0 0) 0x7f8210015700 con 0x7f821409ac90 2024-09-16T05:41:03.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.845+0000 7f820a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:03.851 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.845+0000 7f820a7fc640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-16T05:41:03.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.953+0000 7f821bf69640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:03.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.953+0000 7f821bf69640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:03.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.953+0000 7f821bf69640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:03.959 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.953+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) -- 0x7f81fc000bd0 con 0x7f821409ac90 2024-09-16T05:41:03.960 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:41:03.953+0000 7ff77aa64640 -1 mon.b@1(leader).osd e256 definitely_dead 0 2024-09-16T05:41:03.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.953+0000 7f820a7fc640 1 -- 172.21.15.63:0/1902086868 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v256) ==== 92+0+0 (unknown 350706680 0 0) 0x7f8210015000 con 0x7f821409ac90 2024-09-16T05:41:03.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.953+0000 7f820a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:03.960 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.953+0000 7f820a7fc640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T05:41:03.960 INFO:tasks.workunit.client.0.smithi063.stderr:osd.12 does not exist. 2024-09-16T05:41:03.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.957+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 >> v1:172.21.15.63:6812/3643207211 conn(0x7f81f8036a80 legacy=0x7f81f8038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.957+0000 7f821bf69640 10 monclient: shutdown 2024-09-16T05:41:03.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.957+0000 7f821bf69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:03.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.957+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 >> v1:172.21.15.63:6789/0 conn(0x7f821409ac90 legacy=0x7f821409c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:03.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.957+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 shutdown_connections 2024-09-16T05:41:03.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.957+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 >> 172.21.15.63:0/1902086868 conn(0x7f82140a1bf0 msgr2=0x7f82140a38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:03.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.957+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 shutdown_connections 2024-09-16T05:41:03.962 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:03.957+0000 7f821bf69640 1 -- 172.21.15.63:0/1902086868 wait complete. 2024-09-16T05:41:03.976 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd rm 12 2024-09-16T05:41:04.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient: get_monmap_and_config 2024-09-16T05:41:04.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient: build_initial_monmap 2024-09-16T05:41:04.084 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient: monmap: 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:04.084844+0000 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:04.084844+0000 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:04.085 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:04.086 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 1 Processor -- start 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 1 -- start start 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient: init 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient(hunting): picked mon.noname-b con 0x7f16100a1030 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient(hunting): picked mon.noname-c con 0x7f1610099ae0 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient(hunting): picked mon.noname-a con 0x7f16100a1b20 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16100a2bf0 con 0x7f16100a1b20 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f161009f620 con 0x7f1610099ae0 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16100a2e30 con 0x7f16100a1030 2024-09-16T05:41:04.089 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient(hunting): authenticate will time out at 1320.862489s 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f16156fd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f1610099ae0 0x7f1610138a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50062/0 (socket says 172.21.15.63:50062) 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f16156fd640 1 -- 172.21.15.63:0/963401749 learned_addr learned my addr 172.21.15.63:0/963401749 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f16166ff640 1 -- 172.21.15.63:0/963401749 >> v1:172.21.15.6:6789/0 conn(0x7f16100a1b20 legacy=0x7f161013c1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353973620 0 0) 0x7f161009f620 con 0x7f1610099ae0 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient(hunting): _negotiate 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient(hunting): my global_id is 114334 2024-09-16T05:41:04.090 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15f8003530 con 0x7f1610099ae0 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3139692956 0 0) 0x7f15f8003530 con 0x7f1610099ae0 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 >> v1:172.21.15.6:6790/0 conn(0x7f16100a1030 legacy=0x7f16100a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 >> v1:172.21.15.6:6789/0 conn(0x7f16100a1b20 legacy=0x7f161013c1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: _finish_hunting 0 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 monclient: found mon.noname-c 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161009ed80 con 0x7f1610099ae0 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: _finish_auth 0 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: _check_auth_tickets 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1604001620 con 0x7f1610099ae0 2024-09-16T05:41:04.091 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: dump: 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:04.092 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:04.093 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 5 monclient: authenticate success, global_id 114334 2024-09-16T05:41:04.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:04.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1604001910 con 0x7f1610099ae0 2024-09-16T05:41:04.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:04.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:04.093 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 1 -- 172.21.15.63:0/963401749 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1604001ce0 con 0x7f1610099ae0 2024-09-16T05:41:04.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 4 set_mon_vals no callback set 2024-09-16T05:41:04.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:04.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:04.094 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1614efc640 10 monclient: dump: 2024-09-16T05:41:04.094 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:04.094 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 10 monclient: shutdown 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 1 -- 172.21.15.63:0/963401749 >> v1:172.21.15.63:6789/0 conn(0x7f1610099ae0 legacy=0x7f1610138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.095 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 1 -- 172.21.15.63:0/963401749 shutdown_connections 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.081+0000 7f1618162640 1 -- 172.21.15.63:0/963401749 >> 172.21.15.63:0/963401749 conn(0x7f16100a5c20 msgr2=0x7f16100a2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- 172.21.15.63:0/963401749 shutdown_connections 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- 172.21.15.63:0/963401749 wait complete. 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: build_initial_monmap 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: monmap: 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:04.096 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.097 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:04.097 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:04.097 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:04.097 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 Processor -- start 2024-09-16T05:41:04.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- start start 2024-09-16T05:41:04.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: init 2024-09-16T05:41:04.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:04.101 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:41:04.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient(hunting): picked mon.noname-b con 0x7f1610099ae0 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:04.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient(hunting): picked mon.noname-c con 0x7f16100a1030 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:04.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient(hunting): picked mon.noname-a con 0x7f16100a1b20 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:04.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1610132bb0 con 0x7f16100a1b20 2024-09-16T05:41:04.102 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f161012cb20 con 0x7f1610099ae0 2024-09-16T05:41:04.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1610132e90 con 0x7f16100a1030 2024-09-16T05:41:04.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:04.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient(hunting): authenticate will time out at 1320.865565s 2024-09-16T05:41:04.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1615efe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f1610099ae0 0x7f161012bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:50070/0 (socket says 172.21.15.63:50070) 2024-09-16T05:41:04.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1615efe640 1 -- 172.21.15.63:0/956044455 learned_addr learned my addr 172.21.15.63:0/956044455 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:04.103 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f16166ff640 1 -- 172.21.15.63:0/956044455 >> v1:172.21.15.6:6789/0 conn(0x7f16100a1b20 legacy=0x7f16101d3940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:04.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964395098 0 0) 0x7f161012cb20 con 0x7f1610099ae0 2024-09-16T05:41:04.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:41:04.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:04.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:04.104 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): my global_id is 114337 2024-09-16T05:41:04.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15fc003530 con 0x7f1610099ae0 2024-09-16T05:41:04.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3265142555 0 0) 0x7f15fc003530 con 0x7f1610099ae0 2024-09-16T05:41:04.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f161012cb20 con 0x7f1610099ae0 2024-09-16T05:41:04.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1600001d00 con 0x7f1610099ae0 2024-09-16T05:41:04.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:04.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:04.106 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): dump: 2024-09-16T05:41:04.106 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:04.106 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.106 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.106 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.106 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:04.107 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.107 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:04.107 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:04.107 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:04.107 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.107 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:04.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360167355 0 0) 0x7f1610132e90 con 0x7f16100a1030 2024-09-16T05:41:04.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:41:04.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:04.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:04.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient(hunting): my global_id is 6491 2024-09-16T05:41:04.108 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15fc003530 con 0x7f16100a1030 2024-09-16T05:41:04.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2287152198 0 0) 0x7f161012cb20 con 0x7f1610099ae0 2024-09-16T05:41:04.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 >> v1:172.21.15.6:6790/0 conn(0x7f16100a1030 legacy=0x7f161012c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 >> v1:172.21.15.6:6789/0 conn(0x7f16100a1b20 legacy=0x7f16101d3940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:04.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:41:04.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 monclient: found mon.b 2024-09-16T05:41:04.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:04.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.109 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161009a3a0 con 0x7f1610099ae0 2024-09-16T05:41:04.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:41:04.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:41:04.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:04.110 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 5 monclient: authenticate success, global_id 114337 2024-09-16T05:41:04.111 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: _renew_subs 2024-09-16T05:41:04.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1610099ec0 con 0x7f1610099ae0 2024-09-16T05:41:04.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: _renew_subs 2024-09-16T05:41:04.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.112 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f161012ce00 con 0x7f1610099ae0 2024-09-16T05:41:04.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1600003cd0 con 0x7f1610099ae0 2024-09-16T05:41:04.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:04.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f16000040a0 con 0x7f1610099ae0 2024-09-16T05:41:04.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:04.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160ffff640 4 set_mon_vals no callback set 2024-09-16T05:41:04.113 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:04.114 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 10 monclient: dump: 2024-09-16T05:41:04.114 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:04.114 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.114 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.114 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.114 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:04.115 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.115 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:04.115 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:04.115 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:04.115 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:04.115 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:04.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1600004240 con 0x7f1610099ae0 2024-09-16T05:41:04.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f0005090 con 0x7f1610099ae0 2024-09-16T05:41:04.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(256..256 src has 20..256) ==== 6817+0+0 (unknown 507185705 0 0) 0x7f1600045350 con 0x7f1610099ae0 2024-09-16T05:41:04.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.085+0000 7f16156fd640 10 monclient: get_auth_request con 0x7f15fc036ad0 auth_method 0 2024-09-16T05:41:04.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.093+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1600015920 con 0x7f1610099ae0 2024-09-16T05:41:04.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.093+0000 7f160e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:04.117 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.093+0000 7f160e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:04.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.257+0000 7f1618162640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:04.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.257+0000 7f1618162640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:04.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.257+0000 7f1618162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.257+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) -- 0x7f15f0005380 con 0x7f1610099ae0 2024-09-16T05:41:04.264 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:41:04.257+0000 7ff77aa64640 -1 mon.b@1(leader).osd e256 definitely_dead 0 2024-09-16T05:41:04.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.261+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v256) ==== 92+0+0 (unknown 350706680 0 0) 0x7f1600015220 con 0x7f1610099ae0 2024-09-16T05:41:04.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.261+0000 7f160e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:04.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.261+0000 7f160e7fc640 10 monclient: _finish_command 2 = system:0 osd.12 does not exist. 2024-09-16T05:41:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.361+0000 7f1618162640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.361+0000 7f1618162640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.361+0000 7f1618162640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.366 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.361+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) -- 0x7f15f0000bd0 con 0x7f1610099ae0 2024-09-16T05:41:04.367 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:41:04.361+0000 7ff77aa64640 -1 mon.b@1(leader).osd e256 definitely_dead 0 2024-09-16T05:41:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.361+0000 7f160e7fc640 1 -- 172.21.15.63:0/956044455 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v256) ==== 92+0+0 (unknown 350706680 0 0) 0x7f1600019570 con 0x7f1610099ae0 2024-09-16T05:41:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.361+0000 7f160e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T05:41:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.361+0000 7f160e7fc640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T05:41:04.367 INFO:tasks.workunit.client.0.smithi063.stderr:osd.12 does not exist. 2024-09-16T05:41:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.365+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 >> v1:172.21.15.63:6812/3643207211 conn(0x7f15fc036ad0 legacy=0x7f15fc038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.365+0000 7f1618162640 10 monclient: shutdown 2024-09-16T05:41:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.365+0000 7f1618162640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.365+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 >> v1:172.21.15.63:6789/0 conn(0x7f1610099ae0 legacy=0x7f161012bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.365+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 shutdown_connections 2024-09-16T05:41:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.365+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 >> 172.21.15.63:0/956044455 conn(0x7f16100a5c20 msgr2=0x7f16101b9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.365+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 shutdown_connections 2024-09-16T05:41:04.369 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.365+0000 7f1618162640 1 -- 172.21.15.63:0/956044455 wait complete. 2024-09-16T05:41:04.382 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd rm 14 2024-09-16T05:41:04.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.493+0000 7fe301956640 10 monclient: get_monmap_and_config 2024-09-16T05:41:04.500 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.493+0000 7fe301956640 10 monclient: build_initial_monmap 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.493+0000 7fe301956640 10 monclient: monmap: 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:04.500424+0000 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:04.500424+0000 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 Processor -- start 2024-09-16T05:41:04.501 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- start start 2024-09-16T05:41:04.502 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient: init 2024-09-16T05:41:04.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:04.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:41:04.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient(hunting): picked mon.noname-a con 0x7fe2fc0726a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:04.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient(hunting): picked mon.noname-b con 0x7fe2fc072a80 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:04.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient(hunting): picked mon.noname-c con 0x7fe2fc13d5d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:04.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc099d70 con 0x7fe2fc0726a0 2024-09-16T05:41:04.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc072ed0 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.507 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc1372d0 con 0x7fe2fc072a80 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient(hunting): authenticate will time out at 1321.277320s 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2fb7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe2fc13d5d0 0x7fe2fc13f9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52486/0 (socket says 172.21.15.63:52486) 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2faffd640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fe2fc0726a0 legacy=0x7fe2fc09b7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2fb7fe640 1 -- 172.21.15.63:0/365946401 learned_addr learned my addr 172.21.15.63:0/365946401 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767416106 0 0) 0x7fe2fc072ed0 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient(hunting): my global_id is 114340 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2e4003530 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1807355998 0 0) 0x7fe2e4003530 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.508 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 >> v1:172.21.15.6:6790/0 conn(0x7fe2fc072a80 legacy=0x7fe2fc13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 >> v1:172.21.15.6:6789/0 conn(0x7fe2fc0726a0 legacy=0x7fe2fc09b7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 monclient: found mon.noname-c 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc099fe0 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 5 monclient: authenticate success, global_id 114340 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2f40019a0 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:04.509 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: dump: 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:04.510 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2f4001c00 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 1 -- 172.21.15.63:0/365946401 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2f4002050 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 4 set_mon_vals no callback set 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe2f9ffb640 10 monclient: dump: 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:04.511 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient: shutdown 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- 172.21.15.63:0/365946401 >> v1:172.21.15.63:6789/0 conn(0x7fe2fc13d5d0 legacy=0x7fe2fc13f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- 172.21.15.63:0/365946401 shutdown_connections 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- 172.21.15.63:0/365946401 >> 172.21.15.63:0/365946401 conn(0x7fe2fc0a1ed0 msgr2=0x7fe2fc0a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- 172.21.15.63:0/365946401 shutdown_connections 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- 172.21.15.63:0/365946401 wait complete. 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient: build_initial_monmap 2024-09-16T05:41:04.512 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient: monmap: 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 Processor -- start 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 1 -- start start 2024-09-16T05:41:04.513 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.497+0000 7fe301956640 10 monclient: init 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient(hunting): picked mon.noname-b con 0x7fe2fc0726a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient(hunting): picked mon.noname-c con 0x7fe2fc072a80 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient(hunting): picked mon.noname-a con 0x7fe2fc13d5d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc138a60 con 0x7fe2fc13d5d0 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc139250 con 0x7fe2fc0726a0 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc13b520 con 0x7fe2fc072a80 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient(hunting): authenticate will time out at 1321.280477s 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2faffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe2fc0726a0 0x7fe2fc137c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52488/0 (socket says 172.21.15.63:52488) 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2faffd640 1 -- 172.21.15.63:0/3602757325 learned_addr learned my addr 172.21.15.63:0/3602757325 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2fb7fe640 1 -- 172.21.15.63:0/3602757325 >> v1:172.21.15.6:6789/0 conn(0x7fe2fc13d5d0 legacy=0x7fe2fc1d39b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:04.514 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2126104545 0 0) 0x7fe2fc139250 con 0x7fe2fc0726a0 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient(hunting): _negotiate 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient(hunting): my global_id is 114343 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2e8003530 con 0x7fe2fc0726a0 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3637965922 0 0) 0x7fe2e8003530 con 0x7fe2fc0726a0 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2fc139250 con 0x7fe2fc0726a0 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2f0004190 con 0x7fe2fc0726a0 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient(hunting): dump: 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:04.515 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 289970254 0 0) 0x7fe2fc139250 con 0x7fe2fc0726a0 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 >> v1:172.21.15.6:6790/0 conn(0x7fe2fc072a80 legacy=0x7fe2fc138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 >> v1:172.21.15.6:6789/0 conn(0x7fe2fc13d5d0 legacy=0x7fe2fc1d39b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:04.516 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient: _finish_hunting 0 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 monclient: found mon.b 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc13d2e0 con 0x7fe2fc0726a0 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient: _finish_auth 0 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient: _check_auth_tickets 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 5 monclient: authenticate success, global_id 114343 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient: _renew_subs 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2fc13cdb0 con 0x7fe2fc0726a0 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient: _renew_subs 2024-09-16T05:41:04.517 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2fc13af60 con 0x7fe2fc0726a0 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2f00044f0 con 0x7fe2fc0726a0 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2f00048c0 con 0x7fe2fc0726a0 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2f97fa640 4 set_mon_vals no callback set 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 10 monclient: dump: 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:04.518 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2dc005090 con 0x7fe2fc0726a0 2024-09-16T05:41:04.519 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe2f0004a60 con 0x7fe2fc0726a0 2024-09-16T05:41:04.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(256..256 src has 20..256) ==== 6817+0+0 (unknown 507185705 0 0) 0x7fe2f0044ed0 con 0x7fe2fc0726a0 2024-09-16T05:41:04.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.501+0000 7fe2fa7fc640 10 monclient: get_auth_request con 0x7fe2e80369a0 auth_method 0 2024-09-16T05:41:04.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.505+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe2f0015500 con 0x7fe2fc0726a0 2024-09-16T05:41:04.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.505+0000 7fe2e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:04.520 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.505+0000 7fe2e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:04.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.677+0000 7fe301956640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T05:41:04.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.677+0000 7fe301956640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T05:41:04.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.677+0000 7fe301956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.684 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.677+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) -- 0x7fe2dc005380 con 0x7fe2fc0726a0 2024-09-16T05:41:04.685 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:41:04.681+0000 7ff77aa64640 -1 mon.b@1(leader).osd e256 definitely_dead 0 2024-09-16T05:41:04.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.865+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 removed osd.14 v257) ==== 83+0+0 (unknown 100629483 0 0) 0x7fe2f0014e00 con 0x7fe2fc0726a0 2024-09-16T05:41:04.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.865+0000 7fe2e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T05:41:04.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.865+0000 7fe2e3fff640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-16T05:41:04.979 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.973+0000 7fe301956640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T05:41:04.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.973+0000 7fe301956640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T05:41:04.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.973+0000 7fe301956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:04.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.973+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) -- 0x7fe2dc000bd0 con 0x7fe2fc0726a0 2024-09-16T05:41:04.980 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:41:04.973+0000 7ff77aa64640 -1 mon.b@1(leader).osd e257 definitely_dead 0 2024-09-16T05:41:04.980 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe2e3fff640 1 -- 172.21.15.63:0/3602757325 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 osd.14 does not exist. v257) ==== 92+0+0 (unknown 1523264938 0 0) 0x7fe2f001d9b0 con 0x7fe2fc0726a0 2024-09-16T05:41:04.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe2e3fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T05:41:04.981 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe2e3fff640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-16T05:41:04.981 INFO:tasks.workunit.client.0.smithi063.stderr:osd.14 does not exist. 2024-09-16T05:41:04.982 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe2e80369a0 legacy=0x7fe2e8038e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe301956640 10 monclient: shutdown 2024-09-16T05:41:04.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe301956640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:04.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 >> v1:172.21.15.63:6789/0 conn(0x7fe2fc0726a0 legacy=0x7fe2fc137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:04.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 shutdown_connections 2024-09-16T05:41:04.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 >> 172.21.15.63:0/3602757325 conn(0x7fe2fc0a1ed0 msgr2=0x7fe2fc136460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:04.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 shutdown_connections 2024-09-16T05:41:04.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:04.977+0000 7fe301956640 1 -- 172.21.15.63:0/3602757325 wait complete. 2024-09-16T05:41:04.997 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd rm 1000 2024-09-16T05:41:05.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.109+0000 7f6d111fe640 10 monclient: get_monmap_and_config 2024-09-16T05:41:05.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.109+0000 7f6d111fe640 10 monclient: build_initial_monmap 2024-09-16T05:41:05.116 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: monmap: 2024-09-16T05:41:05.116 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:05.116 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.116 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:05.116522+0000 2024-09-16T05:41:05.116 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:05.116522+0000 2024-09-16T05:41:05.117 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:05.117 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.117 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:05.117 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:05.117 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:05.117 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 Processor -- start 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- start start 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: init 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6d0c09ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6d0c09b070 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient(hunting): picked mon.noname-b con 0x7f6d0c13a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:05.123 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c0722a0 con 0x7f6d0c09b070 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c0a0510 con 0x7f6d0c09ac90 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c09e660 con 0x7f6d0c13a9a0 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient(hunting): authenticate will time out at 1321.893586s 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0ad9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f6d0c09ac90 0x7f6d0c09f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52490/0 (socket says 172.21.15.63:52490) 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0ad9d640 1 -- 172.21.15.63:0/3335890936 learned_addr learned my addr 172.21.15.63:0/3335890936 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0259c640 1 -- 172.21.15.63:0/3335890936 >> v1:172.21.15.6:6789/0 conn(0x7f6d0c09b070 legacy=0x7f6d0c09b480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154462459 0 0) 0x7f6d0c0a0510 con 0x7f6d0c09ac90 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient(hunting): _negotiate 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient(hunting): my global_id is 114346 2024-09-16T05:41:05.124 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cf8003530 con 0x7f6d0c09ac90 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2824705222 0 0) 0x7f6cf8003530 con 0x7f6d0c09ac90 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 >> v1:172.21.15.6:6790/0 conn(0x7f6d0c13a9a0 legacy=0x7f6d0c13cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 >> v1:172.21.15.6:6789/0 conn(0x7f6d0c09b070 legacy=0x7f6d0c09b480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: _finish_hunting 0 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 monclient: found mon.noname-c 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c1414e0 con 0x7f6d0c09ac90 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: _finish_auth 0 2024-09-16T05:41:05.125 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: _check_auth_tickets 2024-09-16T05:41:05.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:05.126 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6cfc001620 con 0x7f6d0c09ac90 2024-09-16T05:41:05.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 5 monclient: authenticate success, global_id 114346 2024-09-16T05:41:05.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:05.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:05.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:05.127 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: dump: 2024-09-16T05:41:05.128 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:05.128 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.128 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.128 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.128 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:05.128 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6cfc001910 con 0x7f6d0c09ac90 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 1 -- 172.21.15.63:0/3335890936 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6cfc001ce0 con 0x7f6d0c09ac90 2024-09-16T05:41:05.129 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:05.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 4 set_mon_vals no callback set 2024-09-16T05:41:05.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:05.130 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d0a59c640 10 monclient: dump: 2024-09-16T05:41:05.130 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:05.130 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.131 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.131 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.131 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:05.131 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.131 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:05.131 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:05.132 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:05.132 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: shutdown 2024-09-16T05:41:05.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:05.132 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- 172.21.15.63:0/3335890936 >> v1:172.21.15.63:6789/0 conn(0x7f6d0c09ac90 legacy=0x7f6d0c09f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- 172.21.15.63:0/3335890936 shutdown_connections 2024-09-16T05:41:05.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- 172.21.15.63:0/3335890936 >> 172.21.15.63:0/3335890936 conn(0x7f6d0c0a1bf0 msgr2=0x7f6d0c0a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:05.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- 172.21.15.63:0/3335890936 shutdown_connections 2024-09-16T05:41:05.135 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- 172.21.15.63:0/3335890936 wait complete. 2024-09-16T05:41:05.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: build_initial_monmap 2024-09-16T05:41:05.136 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: monmap: 2024-09-16T05:41:05.136 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:05.136 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:05.136 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:05.136 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:05.136 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 Processor -- start 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 1 -- start start 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.113+0000 7f6d111fe640 10 monclient: init 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:41:05.137 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6d0c09ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient(hunting): picked mon.noname-b con 0x7f6d0c09b070 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6d0c13a9a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c09d010 con 0x7f6d0c13a9a0 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c09d4f0 con 0x7f6d0c09b070 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0c135530 con 0x7f6d0c09ac90 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient(hunting): authenticate will time out at 1321.896430s 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d0259c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f6d0c09b070 0x7f6d0c09c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52506/0 (socket says 172.21.15.63:52506) 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d0259c640 1 -- 172.21.15.63:0/508856998 learned_addr learned my addr 172.21.15.63:0/508856998 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d0b59e640 1 -- 172.21.15.63:0/508856998 >> v1:172.21.15.6:6789/0 conn(0x7f6d0c13a9a0 legacy=0x7f6d0c1d3670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642883688 0 0) 0x7f6d0c09d4f0 con 0x7f6d0c09b070 2024-09-16T05:41:05.138 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): _negotiate 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): my global_id is 114349 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cec003530 con 0x7f6d0c09b070 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 706493559 0 0) 0x7f6cec003530 con 0x7f6d0c09b070 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d0c09d4f0 con 0x7f6d0c09b070 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6cf40041e0 con 0x7f6d0c09b070 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): dump: 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.139 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2123482333 0 0) 0x7f6d0c135530 con 0x7f6d0c09ac90 2024-09-16T05:41:05.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): _negotiate 2024-09-16T05:41:05.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:05.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:05.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient(hunting): my global_id is 6503 2024-09-16T05:41:05.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cec003530 con 0x7f6d0c09ac90 2024-09-16T05:41:05.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3352783290 0 0) 0x7f6d0c09d4f0 con 0x7f6d0c09b070 2024-09-16T05:41:05.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 >> v1:172.21.15.6:6790/0 conn(0x7f6d0c09ac90 legacy=0x7f6d0c09c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 >> v1:172.21.15.6:6789/0 conn(0x7f6d0c13a9a0 legacy=0x7f6d0c1d3670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:05.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient: _finish_hunting 0 2024-09-16T05:41:05.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 monclient: found mon.b 2024-09-16T05:41:05.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:05.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c09d270 con 0x7f6d0c09b070 2024-09-16T05:41:05.142 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient: _finish_auth 0 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient: _check_auth_tickets 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 5 monclient: authenticate success, global_id 114349 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient: _renew_subs 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0c11b600 con 0x7f6d0c09b070 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient: _renew_subs 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d0c071440 con 0x7f6d0c09b070 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6cf4004440 con 0x7f6d0c09b070 2024-09-16T05:41:05.143 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:05.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6cf4004790 con 0x7f6d0c09b070 2024-09-16T05:41:05.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:05.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:05.144 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d09d9b640 4 set_mon_vals no callback set 2024-09-16T05:41:05.145 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 10 monclient: dump: 2024-09-16T05:41:05.145 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:05.145 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.145 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.146 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.146 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:05.146 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.146 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:05.146 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:05.146 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:05.147 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6cf40049f0 con 0x7f6d0c09b070 2024-09-16T05:41:05.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:05.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:05.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf0005090 con 0x7f6d0c09b070 2024-09-16T05:41:05.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(257..257 src has 20..257) ==== 6817+0+0 (unknown 3327718625 0 0) 0x7f6cf4044f40 con 0x7f6d0c09b070 2024-09-16T05:41:05.147 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.117+0000 7f6d0ad9d640 10 monclient: get_auth_request con 0x7f6cec036ad0 auth_method 0 2024-09-16T05:41:05.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.121+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6cf40154b0 con 0x7f6d0c09b070 2024-09-16T05:41:05.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.121+0000 7f6d03fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:05.148 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.121+0000 7f6d03fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:05.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.293+0000 7f6d111fe640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T05:41:05.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.293+0000 7f6d111fe640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T05:41:05.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.293+0000 7f6d111fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.293+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) -- 0x7f6cf0005380 con 0x7f6d0c09b070 2024-09-16T05:41:05.298 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:41:05.293+0000 7ff77aa64640 -1 mon.b@1(leader).osd e257 definitely_dead 0 2024-09-16T05:41:05.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.293+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v257) ==== 96+0+0 (unknown 259342859 0 0) 0x7f6cf4019160 con 0x7f6d0c09b070 2024-09-16T05:41:05.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.293+0000 7f6d03fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T05:41:05.298 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.293+0000 7f6d03fff640 10 monclient: _finish_command 2 = system:0 osd.1000 does not exist. 2024-09-16T05:41:05.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.393+0000 7f6d111fe640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T05:41:05.399 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.393+0000 7f6d111fe640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T05:41:05.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.393+0000 7f6d111fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.393+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) -- 0x7f6cf0000bd0 con 0x7f6d0c09b070 2024-09-16T05:41:05.400 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:41:05.393+0000 7ff77aa64640 -1 mon.b@1(leader).osd e257 definitely_dead 0 2024-09-16T05:41:05.400 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.393+0000 7f6d03fff640 1 -- 172.21.15.63:0/508856998 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v257) ==== 96+0+0 (unknown 259342859 0 0) 0x7f6cf401da20 con 0x7f6d0c09b070 2024-09-16T05:41:05.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.393+0000 7f6d03fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T05:41:05.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.393+0000 7f6d03fff640 10 monclient: _finish_command 3 = system:0 osd.1000 does not exist. 2024-09-16T05:41:05.401 INFO:tasks.workunit.client.0.smithi063.stderr:osd.1000 does not exist. 2024-09-16T05:41:05.401 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.397+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6cec036ad0 legacy=0x7f6cec038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.397+0000 7f6d111fe640 10 monclient: shutdown 2024-09-16T05:41:05.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.397+0000 7f6d111fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:05.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.397+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 >> v1:172.21.15.63:6789/0 conn(0x7f6d0c09b070 legacy=0x7f6d0c09c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.397+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 shutdown_connections 2024-09-16T05:41:05.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.397+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 >> 172.21.15.63:0/508856998 conn(0x7f6d0c0a1bf0 msgr2=0x7f6d0c075d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:05.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.397+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 shutdown_connections 2024-09-16T05:41:05.402 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.397+0000 7f6d111fe640 1 -- 172.21.15.63:0/508856998 wait complete. 2024-09-16T05:41:05.415 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph osd create 5a2f0584-0de7-4238-aba4-c51878b788e8 2024-09-16T05:41:05.538 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient: get_monmap_and_config 2024-09-16T05:41:05.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient: build_initial_monmap 2024-09-16T05:41:05.539 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient: monmap: 2024-09-16T05:41:05.539 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:05.539 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.539 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:41:05.530167+0000 2024-09-16T05:41:05.539 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:41:05.530167+0000 2024-09-16T05:41:05.539 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:05.539 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.540 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:05.540 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:41:05.540 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:41:05.540 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 1 Processor -- start 2024-09-16T05:41:05.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 1 -- start start 2024-09-16T05:41:05.540 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient: init 2024-09-16T05:41:05.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:05.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:41:05.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient(hunting): picked mon.noname-c con 0x7f262809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:05.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient(hunting): picked mon.noname-b con 0x7f262809b070 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:05.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient(hunting): picked mon.noname-a con 0x7f262813a9a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:05.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26280722a0 con 0x7f262813a9a0 2024-09-16T05:41:05.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26280a0510 con 0x7f262809ac90 2024-09-16T05:41:05.541 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f262809e660 con 0x7f262809b070 2024-09-16T05:41:05.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:05.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 10 monclient(hunting): authenticate will time out at 1322.307064s 2024-09-16T05:41:05.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262d6b4640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f262809ac90 0x7f262809f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52518/0 (socket says 172.21.15.63:52518) 2024-09-16T05:41:05.542 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262d6b4640 1 -- 172.21.15.63:0/445787050 learned_addr learned my addr 172.21.15.63:0/445787050 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:05.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262deb5640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f262813a9a0 legacy=0x7f262813cd90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:05.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 915376045 0 0) 0x7f26280a0510 con 0x7f262809ac90 2024-09-16T05:41:05.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:41:05.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:05.544 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient(hunting): my global_id is 114352 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2614003530 con 0x7f262809ac90 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3492521963 0 0) 0x7f2614003530 con 0x7f262809ac90 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 >> v1:172.21.15.6:6790/0 conn(0x7f262809b070 legacy=0x7f262809b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 >> v1:172.21.15.6:6789/0 conn(0x7f262813a9a0 legacy=0x7f262813cd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 1 monclient: found mon.noname-c 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26281414e0 con 0x7f262809ac90 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient: _finish_auth 0 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:41:05.545 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2624001620 con 0x7f262809ac90 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f261ffff640 10 monclient: dump: 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:05.546 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 5 monclient: authenticate success, global_id 114352 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.525+0000 7f262f918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2624001910 con 0x7f262809ac90 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261ffff640 1 -- 172.21.15.63:0/445787050 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2624001d20 con 0x7f262809ac90 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: get_monmap_and_config success 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 4 set_mon_vals no callback set 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261ffff640 10 monclient: dump: 2024-09-16T05:41:05.547 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:05.548 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: shutdown 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- 172.21.15.63:0/445787050 >> v1:172.21.15.63:6789/0 conn(0x7f262809ac90 legacy=0x7f262809f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- 172.21.15.63:0/445787050 shutdown_connections 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- 172.21.15.63:0/445787050 >> 172.21.15.63:0/445787050 conn(0x7f26280a1bf0 msgr2=0x7f26280a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- 172.21.15.63:0/445787050 shutdown_connections 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- 172.21.15.63:0/445787050 wait complete. 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: build_initial_monmap 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: monmap: 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:41:05.549 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 Processor -- start 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- start start 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: init 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: _reopen_session rank -1 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:41:05.550 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient(hunting): picked mon.noname-b con 0x7f262809ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient(hunting): picked mon.noname-a con 0x7f262809b070 addr v1:172.21.15.6:6789/0 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient(hunting): picked mon.noname-c con 0x7f262813a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f262811b7c0 con 0x7f262809b070 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f262811baa0 con 0x7f262809ac90 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f262809d010 con 0x7f262813a9a0 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient(hunting): _renew_subs 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient(hunting): authenticate will time out at 1322.310344s 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262d6b4640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f262809ac90 0x7f262809c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52522/0 (socket says 172.21.15.63:52522) 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262d6b4640 1 -- 172.21.15.63:0/2125085761 learned_addr learned my addr 172.21.15.63:0/2125085761 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262ceb3640 1 -- 172.21.15.63:0/2125085761 >> v1:172.21.15.6:6789/0 conn(0x7f262809b070 legacy=0x7f262809c900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1295439425 0 0) 0x7f262811baa0 con 0x7f262809ac90 2024-09-16T05:41:05.551 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): my global_id is 114355 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f260c003530 con 0x7f262809ac90 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1451043553 0 0) 0x7f260c003530 con 0x7f262809ac90 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f262811baa0 con 0x7f262809ac90 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f26240015e0 con 0x7f262809ac90 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): dump: 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:05.552 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1871832883 0 0) 0x7f262809d010 con 0x7f262813a9a0 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:41:05.553 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient(hunting): my global_id is 6509 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f260c003530 con 0x7f262813a9a0 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1776113903 0 0) 0x7f262811baa0 con 0x7f262809ac90 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 >> v1:172.21.15.6:6790/0 conn(0x7f262813a9a0 legacy=0x7f26281d3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 >> v1:172.21.15.6:6789/0 conn(0x7f262809b070 legacy=0x7f262809c900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 monclient: found mon.b 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26281411d0 con 0x7f262809ac90 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient: _finish_auth 0 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:41:05.554 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 5 monclient: authenticate success, global_id 114355 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: _renew_subs 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2628072f10 con 0x7f262809ac90 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: _renew_subs 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f262811bec0 con 0x7f262809ac90 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2624001910 con 0x7f262809ac90 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2624004ea0 con 0x7f262809ac90 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:41:05.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261f7fe640 4 set_mon_vals no callback set 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 10 monclient: dump: 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.529+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2624005040 con 0x7f262809ac90 2024-09-16T05:41:05.556 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.533+0000 7f262f918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:05.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.533+0000 7f262f918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:05.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.533+0000 7f262f918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.533+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2608005090 con 0x7f262809ac90 2024-09-16T05:41:05.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.533+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(257..257 src has 20..257) ==== 6817+0+0 (unknown 3327718625 0 0) 0x7f26240450a0 con 0x7f262809ac90 2024-09-16T05:41:05.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.533+0000 7f262ceb3640 10 monclient: get_auth_request con 0x7f260c036a80 auth_method 0 2024-09-16T05:41:05.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.537+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2624015790 con 0x7f262809ac90 2024-09-16T05:41:05.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.537+0000 7f261dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:41:05.557 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.537+0000 7f261dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:41:05.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.709+0000 7f262f918640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:05.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.709+0000 7f262f918640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:05.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.709+0000 7f262f918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.713 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.709+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"} v 0) -- 0x7f2608005380 con 0x7f262809ac90 2024-09-16T05:41:05.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.873+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}]=0 v258) ==== 106+0+3 (unknown 2636009573 0 944643765) 0x7f2624015090 con 0x7f262809ac90 2024-09-16T05:41:05.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.873+0000 7f261dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:05.877 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.873+0000 7f261dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:41:05.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.977+0000 7f262f918640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:05.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.977+0000 7f262f918640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:05.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.977+0000 7f262f918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:41:05.983 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.977+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"} v 0) -- 0x7f2608005b60 con 0x7f262809ac90 2024-09-16T05:41:05.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.977+0000 7f261dffb640 1 -- 172.21.15.63:0/2125085761 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}]=0 v258) ==== 106+0+3 (unknown 2636009573 0 944643765) 0x7f2624033020 con 0x7f262809ac90 2024-09-16T05:41:05.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.977+0000 7f261dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "5a2f0584-0de7-4238-aba4-c51878b788e8"}] 2024-09-16T05:41:05.984 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.977+0000 7f261dffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:41:05.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.981+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 >> v1:172.21.15.63:6812/3643207211 conn(0x7f260c036a80 legacy=0x7f260c038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.985 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.981+0000 7f262f918640 10 monclient: shutdown 2024-09-16T05:41:05.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.981+0000 7f262f918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:41:05.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.981+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 >> v1:172.21.15.63:6789/0 conn(0x7f262809ac90 legacy=0x7f262809c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:41:05.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.981+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 shutdown_connections 2024-09-16T05:41:05.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.981+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 >> 172.21.15.63:0/2125085761 conn(0x7f26280a1bf0 msgr2=0x7f2628139ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:41:05.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.981+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 shutdown_connections 2024-09-16T05:41:05.986 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:41:05.981+0000 7f262f918640 1 -- 172.21.15.63:0/2125085761 wait complete. 2024-09-16T05:41:05.999 INFO:tasks.workunit.client.0.smithi063.stderr:+ na2=12 2024-09-16T05:41:05.999 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo OK 2024-09-16T05:41:06.000 INFO:tasks.workunit.client.0.smithi063.stdout:OK 2024-09-16T05:41:06.000 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:41:06.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:41:06.056 INFO:tasks.workunit:Running workunits matching mon/caps.sh on client.0... 2024-09-16T05:41:06.057 INFO:tasks.workunit:Running workunit mon/caps.sh... 2024-09-16T05:41:06.057 DEBUG:teuthology.orchestra.run.smithi063:workunit test mon/caps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-09-16T05:41:06.109 INFO:tasks.workunit.client.0.smithi063.stderr:+ tmp=/tmp/cephtest-mon-caps-madness 2024-09-16T05:41:06.109 INFO:tasks.workunit.client.0.smithi063.stderr:+ exit_on_error=1 2024-09-16T05:41:06.109 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ ! -z '' ]] 2024-09-16T05:41:06.110 INFO:tasks.workunit.client.0.smithi063.stderr:++ uname 2024-09-16T05:41:06.110 INFO:tasks.workunit.client.0.smithi063.stderr:+ '[' Linux = FreeBSD ']' 2024-09-16T05:41:06.110 INFO:tasks.workunit.client.0.smithi063.stderr:+ ETIMEDOUT=110 2024-09-16T05:41:06.110 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 0 2024-09-16T05:41:06.110 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 2024-09-16T05:41:06.111 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=0 2024-09-16T05:41:06.111 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T05:41:06.111 INFO:tasks.workunit.client.0.smithi063.stdout:ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T05:41:06.111 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T05:41:06.637 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=0 2024-09-16T05:41:06.637 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T05:41:06.637 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:41:06.637 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T05:41:06.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 13 2024-09-16T05:41:06.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 2024-09-16T05:41:06.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:41:06.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T05:41:06.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T05:42:14.220 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:42:14.221 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T05:42:14.221 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T05:42:14.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T05:42:14.226 INFO:tasks.ceph.mon.a:Started 2024-09-16T05:42:14.226 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:42:14.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:42:14.326 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:14.323+0000 7f0dab18fe00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 285 -> 0 (sync_force? mon rebuild?) 2024-09-16T05:42:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient: get_monmap_and_config 2024-09-16T05:42:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient: build_initial_monmap 2024-09-16T05:42:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient: monmap: 2024-09-16T05:42:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:14.348115+0000 2024-09-16T05:42:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:14.348115+0000 2024-09-16T05:42:14.347 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:14.348 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:14.348 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 1 Processor -- start 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 1 -- start start 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient: init 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient(hunting): picked mon.noname-b con 0x7f838c072460 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient(hunting): picked mon.noname-c con 0x7f838c0a0ed0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient(hunting): picked mon.noname-a con 0x7f838c0a29f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:14.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838c0a0210 con 0x7f838c0a29f0 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838c072f50 con 0x7f838c0a0ed0 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838c09f7c0 con 0x7f838c072460 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient(hunting): authenticate will time out at 1396.154515s 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f838affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f838c072460 0x7f838c072840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46320/0 (socket says 172.21.15.6:46320) 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f838affd640 1 -- 172.21.15.6:0/2803574241 learned_addr learned my addr 172.21.15.6:0/2803574241 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2917512276 0 0) 0x7f838c09f7c0 con 0x7f838c072460 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient(hunting): my global_id is 6587 2024-09-16T05:42:14.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f836c003530 con 0x7f838c072460 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2424817360 0 0) 0x7f836c003530 con 0x7f838c072460 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 >> v1:172.21.15.63:6789/0 conn(0x7f838c0a0ed0 legacy=0x7f838c139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 >> v1:172.21.15.6:6789/0 conn(0x7f838c0a29f0 legacy=0x7f838c13ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 monclient: found mon.noname-b 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838c141600 con 0x7f838c072460 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: _finish_auth 0 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:14.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f83740019a0 con 0x7f838c072460 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: dump: 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.355 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 5 monclient: authenticate success, global_id 6587 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8374001c90 con 0x7f838c072460 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8391385640 4 set_mon_vals no callback set 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 1 -- 172.21.15.6:0/2803574241 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8374002020 con 0x7f838c072460 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.343+0000 7f8389ffb640 10 monclient: dump: 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: shutdown 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- 172.21.15.6:0/2803574241 >> v1:172.21.15.6:6790/0 conn(0x7f838c072460 legacy=0x7f838c072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- 172.21.15.6:0/2803574241 shutdown_connections 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- 172.21.15.6:0/2803574241 >> 172.21.15.6:0/2803574241 conn(0x7f838c0a3880 msgr2=0x7f838c0a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- 172.21.15.6:0/2803574241 shutdown_connections 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- 172.21.15.6:0/2803574241 wait complete. 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: build_initial_monmap 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: monmap: 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 Processor -- start 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- start start 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: init 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient(hunting): picked mon.noname-a con 0x7f838c072460 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient(hunting): picked mon.noname-b con 0x7f838c0a0ed0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient(hunting): picked mon.noname-c con 0x7f838c0a29f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838c124990 con 0x7f838c072460 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838c124c70 con 0x7f838c0a0ed0 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838c122600 con 0x7f838c0a29f0 2024-09-16T05:42:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient(hunting): authenticate will time out at 1396.158021s 2024-09-16T05:42:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f838b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f838c0a29f0 0x7f838c1d3a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46332/0 (socket says 172.21.15.6:46332) 2024-09-16T05:42:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f838b7fe640 1 -- 172.21.15.6:0/337777909 learned_addr learned my addr 172.21.15.6:0/337777909 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2825606956 0 0) 0x7f838c124c70 con 0x7f838c0a0ed0 2024-09-16T05:42:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient(hunting): my global_id is 114436 2024-09-16T05:42:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8370003530 con 0x7f838c0a0ed0 2024-09-16T05:42:14.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1543821498 0 0) 0x7f8370003530 con 0x7f838c0a0ed0 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f838c124c70 con 0x7f838c0a0ed0 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f837c001880 con 0x7f838c0a0ed0 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient(hunting): dump: 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.366 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1039429799 0 0) 0x7f838c124c70 con 0x7f838c0a0ed0 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.6:6790/0 conn(0x7f838c0a29f0 legacy=0x7f838c1d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.6:6789/0 conn(0x7f838c072460 legacy=0x7f838c1217e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient: _finish_hunting 0 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 monclient: found mon.b 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838c1239e0 con 0x7f838c0a0ed0 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient: _finish_auth 0 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 10 monclient: _check_auth_tickets 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f83837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 5 monclient: authenticate success, global_id 114436 2024-09-16T05:42:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: _renew_subs 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f838c0a5e90 con 0x7f838c0a0ed0 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: _renew_subs 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.347+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f838c123540 con 0x7f838c0a0ed0 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f837c003620 con 0x7f838c0a0ed0 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83897fa640 4 set_mon_vals no callback set 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f838a7fc640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.63:6789/0 conn(0x7f838c0a0ed0 legacy=0x7f838c121ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f838a7fc640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.63:6789/0 conn(0x7f838c0a0ed0 legacy=0x7f838c121ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f838a7fc640 1 --1- 172.21.15.6:0/337777909 >> v1:172.21.15.63:6789/0 conn(0x7f838c0a0ed0 0x7f838c121ef0 :-1 s=OPENED pgs=168 cs=1 l=1).handle_message read tag failed 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f838a7fc640 1 --1- 172.21.15.6:0/337777909 >> v1:172.21.15.63:6789/0 conn(0x7f838c0a0ed0 0x7f838c121ef0 :-1 s=OPENED pgs=168 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:42:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.63:6789/0 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.63:6789/0 conn(0x7f838c0a0ed0 legacy=0x7f838c121ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f8391385640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient(hunting): picked mon.a con 0x7f838c072460 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient(hunting): picked mon.c con 0x7f838c0a29f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient(hunting): picked mon.b con 0x7f837000c0f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f838c124c70 con 0x7f838c072460 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8370003530 con 0x7f837000c0f0 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8370011fa0 con 0x7f838c0a29f0 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f8391385640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:14.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132512578 0 0) 0x7f8370011fa0 con 0x7f838c0a29f0 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.6:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f837000ff40 con 0x7f838c0a29f0 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 3426289088 0 0) 0x7f837000ff40 con 0x7f838c0a29f0 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.63:6789/0 conn(0x7f837000c0f0 legacy=0x7f837000e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.6:6789/0 conn(0x7f838c072460 legacy=0x7f8370004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _finish_hunting 0 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 monclient: found mon.c 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f8370013320 con 0x7f838c0a29f0 2024-09-16T05:42:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8364005090 con 0x7f838c0a29f0 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8370011fa0 con 0x7f838c0a29f0 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _finish_auth 0 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: _check_auth_tickets 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f837c003cc0 con 0x7f838c0a29f0 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: dump: 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f837c003eb0 con 0x7f838c0a29f0 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f8388ff9640 4 set_mon_vals no callback set 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f837c0040e0 con 0x7f838c0a29f0 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 6 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f837c045230 con 0x7f838c0a29f0 2024-09-16T05:42:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.351+0000 7f838affd640 10 monclient: get_auth_request con 0x7f838c0a0ed0 auth_method 0 2024-09-16T05:42:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.355+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f837c015980 con 0x7f838c0a29f0 2024-09-16T05:42:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.355+0000 7f83837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.355+0000 7f83837fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.363+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f837c015280 con 0x7f838c0a29f0 2024-09-16T05:42:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.363+0000 7f83837fe640 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T05:42:14.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:42:14.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:14.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8364004150 con 0x7f838c0a29f0 2024-09-16T05:42:14.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f83837fe640 1 -- 172.21.15.6:0/337777909 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3906881845) 0x7f837c040ad0 con 0x7f838c0a29f0 2024-09-16T05:42:14.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f83837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:14.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f83837fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:14.613 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:42:14.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.63:6812/3643207211 conn(0x7f838c0a0ed0 legacy=0x7f8370044b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 10 monclient: shutdown 2024-09-16T05:42:14.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:14.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 >> v1:172.21.15.6:6790/0 conn(0x7f838c0a29f0 legacy=0x7f837000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 shutdown_connections 2024-09-16T05:42:14.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.607+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 >> 172.21.15.6:0/337777909 conn(0x7f838c0a3880 msgr2=0x7f838c0a3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:14.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.611+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 shutdown_connections 2024-09-16T05:42:14.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.611+0000 7f8391385640 1 -- 172.21.15.6:0/337777909 wait complete. 2024-09-16T05:42:14.629 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:42:14.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:42:14.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:42:14.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient: get_monmap_and_config 2024-09-16T05:42:14.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient: build_initial_monmap 2024-09-16T05:42:14.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient: monmap: 2024-09-16T05:42:14.746 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:14.746 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.747 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:14.747245+0000 2024-09-16T05:42:14.747 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:14.747245+0000 2024-09-16T05:42:14.747 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:14.747 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.747 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:14.747 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:14.747 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:14.747 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 Processor -- start 2024-09-16T05:42:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- start start 2024-09-16T05:42:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient: init 2024-09-16T05:42:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient(hunting): picked mon.noname-c con 0x7f44e0072300 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient(hunting): picked mon.noname-b con 0x7f44e0072df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient(hunting): picked mon.noname-a con 0x7f44e013d320 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e009b330 con 0x7f44e013d320 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e009b050 con 0x7f44e0072300 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e009b610 con 0x7f44e0072df0 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient(hunting): authenticate will time out at 1396.553859s 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e5cb8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44e013d320 0x7f44e013f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36014/0 (socket says 172.21.15.6:36014) 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e5cb8640 1 -- 172.21.15.6:0/968586675 learned_addr learned my addr 172.21.15.6:0/968586675 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2528603590 0 0) 0x7f44d0001120 con 0x7f44e013d320 2024-09-16T05:42:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): my global_id is 124098 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44c8003530 con 0x7f44e013d320 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342613772 0 0) 0x7f44e009b610 con 0x7f44e0072df0 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): my global_id is 6593 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44d0001120 con 0x7f44e0072df0 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3287384093 0 0) 0x7f44e009b050 con 0x7f44e0072300 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient(hunting): my global_id is 114439 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e009b610 con 0x7f44e0072300 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2035624740 0 0) 0x7f44c8003530 con 0x7f44e013d320 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 >> v1:172.21.15.6:6790/0 conn(0x7f44e0072df0 legacy=0x7f44e013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 >> v1:172.21.15.63:6789/0 conn(0x7f44e0072300 legacy=0x7f44e00726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: _finish_hunting 0 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 monclient: found mon.noname-a 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e009c070 con 0x7f44e013d320 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: _finish_auth 0 2024-09-16T05:42:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: _check_auth_tickets 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f44d00017f0 con 0x7f44e013d320 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: dump: 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 5 monclient: authenticate success, global_id 124098 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f44d0001a50 con 0x7f44e013d320 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 1 -- 172.21.15.6:0/968586675 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f44d00048b0 con 0x7f44e013d320 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44dffff640 10 monclient: dump: 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.754 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 4 set_mon_vals no callback set 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 10 monclient: shutdown 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- 172.21.15.6:0/968586675 >> v1:172.21.15.6:6789/0 conn(0x7f44e013d320 legacy=0x7f44e013f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- 172.21.15.6:0/968586675 shutdown_connections 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- 172.21.15.6:0/968586675 >> 172.21.15.6:0/968586675 conn(0x7f44e00a1650 msgr2=0x7f44e00a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:14.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- 172.21.15.6:0/968586675 shutdown_connections 2024-09-16T05:42:14.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.743+0000 7f44e771b640 1 -- 172.21.15.6:0/968586675 wait complete. 2024-09-16T05:42:14.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: build_initial_monmap 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: monmap: 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 1 Processor -- start 2024-09-16T05:42:14.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 1 -- start start 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: init 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient(hunting): picked mon.noname-b con 0x7f44e0072300 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient(hunting): picked mon.noname-a con 0x7f44e0072df0 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient(hunting): picked mon.noname-c con 0x7f44e013d320 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e011e4a0 con 0x7f44e0072df0 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e011bfc0 con 0x7f44e0072300 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e011d2d0 con 0x7f44e013d320 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient(hunting): authenticate will time out at 1396.557171s 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e5cb8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f44e013d320 0x7f44e01d3d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46362/0 (socket says 172.21.15.6:46362) 2024-09-16T05:42:14.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e5cb8640 1 -- 172.21.15.6:0/231959389 learned_addr learned my addr 172.21.15.6:0/231959389 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e4cb6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44e0072df0 0x7f44e011b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36030/0 (socket says 172.21.15.6:36030) 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062642300 0 0) 0x7f44e011e4a0 con 0x7f44e0072df0 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): my global_id is 124101 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44cc003530 con 0x7f44e0072df0 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174237263 0 0) 0x7f44e011d2d0 con 0x7f44e013d320 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): my global_id is 6596 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e011e4a0 con 0x7f44e013d320 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1252717163 0 0) 0x7f44e011bfc0 con 0x7f44e0072300 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): my global_id is 114442 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e011d2d0 con 0x7f44e0072300 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 893619372 0 0) 0x7f44cc003530 con 0x7f44e0072df0 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44e011bfc0 con 0x7f44e0072df0 2024-09-16T05:42:14.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f44d4001de0 con 0x7f44e0072df0 2024-09-16T05:42:14.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:14.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:14.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): dump: 2024-09-16T05:42:14.766 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.766 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.766 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.766 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.766 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.767 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.767 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2661334004 0 0) 0x7f44e011e4a0 con 0x7f44e013d320 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44cc003530 con 0x7f44e013d320 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f44d0001550 con 0x7f44e013d320 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): dump: 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.768 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3894197707 0 0) 0x7f44e011bfc0 con 0x7f44e0072df0 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 >> v1:172.21.15.6:6790/0 conn(0x7f44e013d320 legacy=0x7f44e01d3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 >> v1:172.21.15.63:6789/0 conn(0x7f44e0072300 legacy=0x7f44e011b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: _finish_hunting 0 2024-09-16T05:42:14.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 monclient: found mon.a 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e011ea60 con 0x7f44e0072df0 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: _finish_auth 0 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: _check_auth_tickets 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 5 monclient: authenticate success, global_id 124101 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: _renew_subs 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f44e011e1c0 con 0x7f44e0072df0 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: _renew_subs 2024-09-16T05:42:14.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f44e011d050 con 0x7f44e0072df0 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f44d4002040 con 0x7f44e0072df0 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44df7fe640 4 set_mon_vals no callback set 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f44d4003dd0 con 0x7f44e0072df0 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: dump: 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.771 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f44c00065d0 con 0x7f44c0002870 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f44d4003f70 con 0x7f44e0072df0 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f44d40451b0 con 0x7f44e0072df0 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763858105 0 0) 0x7f44c00065d0 con 0x7f44c0002870 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient(hunting): my global_id is 124104 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44e011bfc0 con 0x7f44c0002870 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44e5cb8640 10 monclient: get_auth_request con 0x7f44cc036b80 auth_method 0 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1985634102 0 0) 0x7f44e011bfc0 con 0x7f44c0002870 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44c00065d0 con 0x7f44c0002870 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f44d80017e0 con 0x7f44c0002870 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.747+0000 7f44ddffb640 10 monclient: dump: 2024-09-16T05:42:14.773 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.751+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3422869753 0 0) 0x7f44c00065d0 con 0x7f44c0002870 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.751+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44c000a160 con 0x7f44c0002870 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.751+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f44d8001ae0 con 0x7f44c0002870 2024-09-16T05:42:14.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.751+0000 7f44ddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.751+0000 7f44ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.751+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 >> v1:172.21.15.6:6789/0 conn(0x7f44c0002870 legacy=0x7f44c0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.751+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 >> v1:172.21.15.6:6789/0 conn(0x7f44c0002870 legacy=0x7f44c0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f44c0009e30 con 0x7f44c00090f0 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608963393 0 0) 0x7f44c0009e30 con 0x7f44c00090f0 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient(hunting): my global_id is 124107 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44c00065d0 con 0x7f44c00090f0 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2293255265 0 0) 0x7f44c00065d0 con 0x7f44c00090f0 2024-09-16T05:42:14.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44c0009e30 con 0x7f44c00090f0 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f44d80037c0 con 0x7f44c00090f0 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient: dump: 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 312138251 0 0) 0x7f44c0009e30 con 0x7f44c00090f0 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f44c000c3a0 con 0x7f44c00090f0 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3961 (unknown 0 0 3713977687) 0x7f44d8005a80 con 0x7f44c00090f0 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 >> v1:172.21.15.6:6789/0 conn(0x7f44c00090f0 legacy=0x7f44c0009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44ddffb640 1 -- 172.21.15.6:0/231959389 >> v1:172.21.15.6:6789/0 conn(0x7f44c00090f0 legacy=0x7f44c0009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:14.777 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 34, 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 447, 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:42:14.778 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:14.779 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:42:14.779 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:42:14.779 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:14.779 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:14.779 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:14.779 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:14.779 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:14.779 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:14.780 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:42:14.781 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:14.782 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.783 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.784 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:42:14.785 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:14.786 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:14.787 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.788 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:14.789 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:14.789 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:42:14.789 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.789 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.789 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:42:14.789 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:14.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 >> v1:172.21.15.63:6812/3643207211 conn(0x7f44cc036b80 legacy=0x7f44cc039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 10 monclient: shutdown 2024-09-16T05:42:14.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:14.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 >> v1:172.21.15.6:6789/0 conn(0x7f44e0072df0 legacy=0x7f44e011b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e5cb8640 1 -- 172.21.15.6:0/231959389 reap_dead start 2024-09-16T05:42:14.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e5cb8640 1 -- 172.21.15.6:0/231959389 reap_dead start 2024-09-16T05:42:14.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 shutdown_connections 2024-09-16T05:42:14.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 >> 172.21.15.6:0/231959389 conn(0x7f44e00a1650 msgr2=0x7f44e00a3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:14.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 shutdown_connections 2024-09-16T05:42:14.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.755+0000 7f44e771b640 1 -- 172.21.15.6:0/231959389 wait complete. 2024-09-16T05:42:14.791 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:42:14.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:42:14.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:42:14.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient: get_monmap_and_config 2024-09-16T05:42:14.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient: build_initial_monmap 2024-09-16T05:42:14.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient: monmap: 2024-09-16T05:42:14.910 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:14.911361+0000 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:14.911361+0000 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:14.911 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 Processor -- start 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- start start 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient: init 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient(hunting): picked mon.noname-b con 0x7f13380a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient(hunting): picked mon.noname-a con 0x7f13380a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient(hunting): picked mon.noname-c con 0x7f13380a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f133806e520 con 0x7f13380a4290 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13380a0780 con 0x7f13380a1a00 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13380a4c70 con 0x7f13380a37a0 2024-09-16T05:42:14.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient(hunting): authenticate will time out at 1396.717933s 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f13377fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f13380a4290 0x7f133813bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36070/0 (socket says 172.21.15.6:36070) 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f13377fe640 1 -- 172.21.15.6:0/4276392515 learned_addr learned my addr 172.21.15.6:0/4276392515 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1337fff640 1 --1- 172.21.15.6:0/4276392515 >> v1:172.21.15.6:6790/0 conn(0x7f13380a37a0 0x7f13380a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46374/0 (socket says 172.21.15.6:46374) 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 118408735 0 0) 0x7f13380a4c70 con 0x7f13380a37a0 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): my global_id is 6599 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1320003530 con 0x7f13380a37a0 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951107639 0 0) 0x7f133806e520 con 0x7f13380a4290 2024-09-16T05:42:14.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): my global_id is 124110 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13380a4c70 con 0x7f13380a4290 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133ce55640 1 -- 172.21.15.6:0/4276392515 >> v1:172.21.15.63:6789/0 conn(0x7f13380a1a00 legacy=0x7f133813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133ce55640 1 -- 172.21.15.6:0/4276392515 >> v1:172.21.15.63:6789/0 conn(0x7f13380a1a00 legacy=0x7f133813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133ce55640 1 --1- 172.21.15.6:0/4276392515 >> v1:172.21.15.63:6789/0 conn(0x7f13380a1a00 0x7f133813f1e0 :-1 s=OPENED pgs=173 cs=1 l=1).handle_message read tag failed 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133ce55640 1 --1- 172.21.15.6:0/4276392515 >> v1:172.21.15.63:6789/0 conn(0x7f13380a1a00 0x7f133813f1e0 :-1 s=OPENED pgs=173 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6789/0 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1921543698 0 0) 0x7f1320003530 con 0x7f13380a37a0 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 >> v1:172.21.15.63:6789/0 conn(0x7f13380a1a00 legacy=0x7f133813f1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:14.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 >> v1:172.21.15.6:6789/0 conn(0x7f13380a4290 legacy=0x7f133813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 monclient: found mon.noname-b 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13380a65f0 con 0x7f13380a37a0 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: _finish_auth 0 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1328001620 con 0x7f13380a37a0 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: dump: 2024-09-16T05:42:14.925 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 5 monclient: authenticate success, global_id 6599 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1328001880 con 0x7f13380a37a0 2024-09-16T05:42:14.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 1 -- 172.21.15.6:0/4276392515 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1328001c90 con 0x7f13380a37a0 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 4 set_mon_vals no callback set 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f1336ffd640 10 monclient: dump: 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.927 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 10 monclient: shutdown 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- 172.21.15.6:0/4276392515 >> v1:172.21.15.6:6790/0 conn(0x7f13380a37a0 legacy=0x7f13380a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- 172.21.15.6:0/4276392515 shutdown_connections 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- 172.21.15.6:0/4276392515 >> 172.21.15.6:0/4276392515 conn(0x7f13380a58a0 msgr2=0x7f13380a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- 172.21.15.6:0/4276392515 shutdown_connections 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.907+0000 7f133e8b8640 1 -- 172.21.15.6:0/4276392515 wait complete. 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: build_initial_monmap 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: monmap: 2024-09-16T05:42:14.928 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 1 Processor -- start 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 1 -- start start 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: init 2024-09-16T05:42:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient(hunting): picked mon.noname-c con 0x7f13380a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient(hunting): picked mon.noname-b con 0x7f13380a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient(hunting): picked mon.noname-a con 0x7f13380a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13380a0100 con 0x7f13380a4290 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13380a7e40 con 0x7f13380a37a0 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1338135980 con 0x7f13380a1a00 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient(hunting): authenticate will time out at 1396.721147s 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1337fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f13380a1a00 0x7f1338136eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46386/0 (socket says 172.21.15.6:46386) 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1337fff640 1 -- 172.21.15.6:0/84435433 learned_addr learned my addr 172.21.15.6:0/84435433 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 790455182 0 0) 0x7f1338135980 con 0x7f13380a1a00 2024-09-16T05:42:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): my global_id is 6602 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1324003530 con 0x7f13380a1a00 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529937581 0 0) 0x7f13380a0780 con 0x7f13380a4290 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): my global_id is 124113 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1338135980 con 0x7f13380a4290 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4027265635 0 0) 0x7f1324003530 con 0x7f13380a1a00 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13380a0780 con 0x7f13380a1a00 2024-09-16T05:42:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1328001620 con 0x7f13380a1a00 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): dump: 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.932 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 438868464 0 0) 0x7f1338135980 con 0x7f13380a4290 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1324003530 con 0x7f13380a4290 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1330001b80 con 0x7f13380a4290 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): dump: 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.933 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.934 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.934 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.934 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185333414 0 0) 0x7f13380a7e40 con 0x7f13380a37a0 2024-09-16T05:42:14.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): my global_id is 114445 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1338135980 con 0x7f13380a37a0 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3388749979 0 0) 0x7f13380a0780 con 0x7f13380a1a00 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 >> v1:172.21.15.63:6789/0 conn(0x7f13380a37a0 legacy=0x7f13381d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 >> v1:172.21.15.6:6789/0 conn(0x7f13380a4290 legacy=0x7f13381d3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 monclient: found mon.c 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13381375c0 con 0x7f13380a1a00 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: _finish_auth 0 2024-09-16T05:42:14.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 5 monclient: authenticate success, global_id 6602 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: _renew_subs 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1338134430 con 0x7f13380a1a00 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: _renew_subs 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f13381c8960 con 0x7f13380a1a00 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1328001880 con 0x7f13380a1a00 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1328004da0 con 0x7f13380a1a00 2024-09-16T05:42:14.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: dump: 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f13367fc640 4 set_mon_vals no callback set 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f13180065d0 con 0x7f1318002870 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f132800fb40 con 0x7f13380a1a00 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f1328044e30 con 0x7f13380a1a00 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234269632 0 0) 0x7f13180065d0 con 0x7f1318002870 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient(hunting): my global_id is 6605 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13380a0780 con 0x7f1318002870 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3508646669 0 0) 0x7f13380a0780 con 0x7f1318002870 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13180065d0 con 0x7f1318002870 2024-09-16T05:42:14.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f132c001d50 con 0x7f1318002870 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f133ce55640 10 monclient: get_auth_request con 0x7f1324036c20 auth_method 0 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 10 monclient: dump: 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.939 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.911+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3966564637 0 0) 0x7f13180065d0 con 0x7f1318002870 2024-09-16T05:42:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.915+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f131800a160 con 0x7f1318002870 2024-09-16T05:42:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.915+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f132c001270 con 0x7f1318002870 2024-09-16T05:42:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.915+0000 7f1334ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.915+0000 7f1334ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.915+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 >> v1:172.21.15.6:6790/0 conn(0x7f1318002870 legacy=0x7f1318004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.915+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 >> v1:172.21.15.6:6790/0 conn(0x7f1318002870 legacy=0x7f1318004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:14.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f133e8b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1318009e50 con 0x7f1318009110 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 326808086 0 0) 0x7f1318009e50 con 0x7f1318009110 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient(hunting): my global_id is 6608 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13180065d0 con 0x7f1318009110 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3984741338 0 0) 0x7f13180065d0 con 0x7f1318009110 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1318009e50 con 0x7f1318009110 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f132c004990 con 0x7f1318009110 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:14.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient: dump: 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3969391334 0 0) 0x7f1318009e50 con 0x7f1318009110 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f131800c3e0 con 0x7f1318009110 2024-09-16T05:42:14.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 3795705250) 0x7f132c006b90 con 0x7f1318009110 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 >> v1:172.21.15.6:6790/0 conn(0x7f1318009110 legacy=0x7f13180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.919+0000 7f1334ff9640 1 -- 172.21.15.6:0/84435433 >> v1:172.21.15.6:6790/0 conn(0x7f1318009110 legacy=0x7f13180095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 34, 2024-09-16T05:42:14.944 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 463976, 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:14.945 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:14.946 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:42:14.947 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:14.948 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.949 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.950 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.951 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.952 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:14.953 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.954 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:14.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:14.956 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:42:14.956 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1324036c20 legacy=0x7f13240390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133e8b8640 10 monclient: shutdown 2024-09-16T05:42:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133e8b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 >> v1:172.21.15.6:6790/0 conn(0x7f13380a1a00 legacy=0x7f1338136eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:14.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133ce55640 1 -- 172.21.15.6:0/84435433 reap_dead start 2024-09-16T05:42:14.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133ce55640 1 -- 172.21.15.6:0/84435433 reap_dead start 2024-09-16T05:42:14.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 shutdown_connections 2024-09-16T05:42:14.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 >> 172.21.15.6:0/84435433 conn(0x7f13380a58a0 msgr2=0x7f13380a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:14.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 shutdown_connections 2024-09-16T05:42:14.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:14.923+0000 7f133e8b8640 1 -- 172.21.15.6:0/84435433 wait complete. 2024-09-16T05:42:14.958 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:42:14.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:42:14.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient: get_monmap_and_config 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient: build_initial_monmap 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient: monmap: 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:15.076008+0000 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:15.076008+0000 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.076 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:15.076 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:15.076 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:15.076 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 1 Processor -- start 2024-09-16T05:42:15.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 1 -- start start 2024-09-16T05:42:15.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient: init 2024-09-16T05:42:15.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:15.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:42:15.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f680a0df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:15.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient(hunting): picked mon.noname-b con 0x7f4f68135330 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:15.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient(hunting): picked mon.noname-a con 0x7f4f68138e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6809d2f0 con 0x7f4f68138e40 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6813d960 con 0x7f4f680a0df0 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6813eb40 con 0x7f4f68135330 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d82a640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4f68135330 0x7f4f68137720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46424/0 (socket says 172.21.15.6:46424) 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d82a640 1 -- 172.21.15.6:0/1018804411 learned_addr learned my addr 172.21.15.6:0/1018804411 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f7028f640 10 monclient(hunting): authenticate will time out at 1396.883482s 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339702885 0 0) 0x7f4f6813eb40 con 0x7f4f68135330 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d029640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d029640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d029640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d029640 10 monclient(hunting): my global_id is 6611 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f54004640 con 0x7f4f68135330 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6e82c640 1 -- 172.21.15.6:0/1018804411 >> v1:172.21.15.6:6789/0 conn(0x7f4f68138e40 legacy=0x7f4f6813b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6e82c640 1 -- 172.21.15.6:0/1018804411 >> v1:172.21.15.6:6789/0 conn(0x7f4f68138e40 legacy=0x7f4f6813b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6e82c640 1 --1- 172.21.15.6:0/1018804411 >> v1:172.21.15.6:6789/0 conn(0x7f4f68138e40 0x7f4f6813b230 :-1 s=OPENED pgs=836 cs=1 l=1).handle_message read tag failed 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6e82c640 1 --1- 172.21.15.6:0/1018804411 >> v1:172.21.15.6:6789/0 conn(0x7f4f68138e40 0x7f4f6813b230 :-1 s=OPENED pgs=836 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.071+0000 7f4f6d029640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1989006911 0 0) 0x7f4f54004640 con 0x7f4f68135330 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 >> v1:172.21.15.63:6789/0 conn(0x7f4f680a0df0 legacy=0x7f4f6809ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 >> v1:172.21.15.6:6789/0 conn(0x7f4f68138e40 legacy=0x7f4f6813b230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: _finish_hunting 0 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 1 monclient: found mon.noname-b 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6813fd20 con 0x7f4f68135330 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: _finish_auth 0 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: _check_auth_tickets 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4f58001620 con 0x7f4f68135330 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 5 monclient: authenticate success, global_id 6611 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: dump: 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f580018c0 con 0x7f4f68135330 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 1 -- 172.21.15.6:0/1018804411 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4f58001cd0 con 0x7f4f68135330 2024-09-16T05:42:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 4 set_mon_vals no callback set 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d029640 10 monclient: dump: 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.084 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: shutdown 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- 172.21.15.6:0/1018804411 >> v1:172.21.15.6:6790/0 conn(0x7f4f68135330 legacy=0x7f4f68137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- 172.21.15.6:0/1018804411 shutdown_connections 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- 172.21.15.6:0/1018804411 >> 172.21.15.6:0/1018804411 conn(0x7f4f680a1650 msgr2=0x7f4f680a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- 172.21.15.6:0/1018804411 shutdown_connections 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- 172.21.15.6:0/1018804411 wait complete. 2024-09-16T05:42:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: build_initial_monmap 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: monmap: 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 Processor -- start 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- start start 2024-09-16T05:42:15.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: init 2024-09-16T05:42:15.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:15.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:42:15.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient(hunting): picked mon.noname-b con 0x7f4f680a0df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:15.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient(hunting): picked mon.noname-a con 0x7f4f68135330 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f68138e40 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6813eb90 con 0x7f4f68135330 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6813ee70 con 0x7f4f680a0df0 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6813bd40 con 0x7f4f68138e40 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient(hunting): authenticate will time out at 1396.886364s 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6e82c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4f68138e40 0x7f4f681d1270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46436/0 (socket says 172.21.15.6:46436) 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6e82c640 1 -- 172.21.15.6:0/3540989115 learned_addr learned my addr 172.21.15.6:0/3540989115 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6d82a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f68135330 0x7f4f6813b630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36084/0 (socket says 172.21.15.6:36084) 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3211225894 0 0) 0x7f4f6813bd40 con 0x7f4f68138e40 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): my global_id is 6614 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f4c003530 con 0x7f4f68138e40 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1112842664 0 0) 0x7f4f6813eb90 con 0x7f4f68135330 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): my global_id is 124119 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f6813bd40 con 0x7f4f68135330 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1220853727 0 0) 0x7f4f4c003530 con 0x7f4f68138e40 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f6813eb90 con 0x7f4f68138e40 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1631855125 0 0) 0x7f4f6813bd40 con 0x7f4f68135330 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f4c003530 con 0x7f4f68135330 2024-09-16T05:42:15.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4f600019e0 con 0x7f4f68138e40 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): dump: 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.093 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4f58001520 con 0x7f4f68135330 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): dump: 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.094 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009398845 0 0) 0x7f4f6813ee70 con 0x7f4f680a0df0 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient(hunting): my global_id is 114451 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f6813bd40 con 0x7f4f680a0df0 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 800093366 0 0) 0x7f4f6813eb90 con 0x7f4f68138e40 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 >> v1:172.21.15.63:6789/0 conn(0x7f4f680a0df0 legacy=0x7f4f6813af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 >> v1:172.21.15.6:6789/0 conn(0x7f4f68135330 legacy=0x7f4f6813b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient: _finish_hunting 0 2024-09-16T05:42:15.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 monclient: found mon.c 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f680740c0 con 0x7f4f68138e40 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient: _finish_auth 0 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient: _check_auth_tickets 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 5 monclient: authenticate success, global_id 6614 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: _renew_subs 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f6809ff30 con 0x7f4f68138e40 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: _renew_subs 2024-09-16T05:42:15.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f68134ff0 con 0x7f4f68138e40 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f60001d20 con 0x7f4f68138e40 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4f60003a70 con 0x7f4f68138e40 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f6c828640 4 set_mon_vals no callback set 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f5effd640 10 monclient: dump: 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.097 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.075+0000 7f4f7028f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4f500065d0 con 0x7f4f50002870 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4f60003c10 con 0x7f4f68138e40 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f4f60044ee0 con 0x7f4f68138e40 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1222198376 0 0) 0x7f4f500065d0 con 0x7f4f50002870 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient(hunting): my global_id is 114454 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f6813eb90 con 0x7f4f50002870 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f6d82a640 10 monclient: get_auth_request con 0x7f4f4c036c20 auth_method 0 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2488146590 0 0) 0x7f4f6813eb90 con 0x7f4f50002870 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f500065d0 con 0x7f4f50002870 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4f64001a60 con 0x7f4f50002870 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient: dump: 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.099 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2445861653 0 0) 0x7f4f500065d0 con 0x7f4f50002870 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f5000a160 con 0x7f4f50002870 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4f64001eb0 con 0x7f4f50002870 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 >> v1:172.21.15.63:6789/0 conn(0x7f4f50002870 legacy=0x7f4f50004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.079+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 >> v1:172.21.15.63:6789/0 conn(0x7f4f50002870 legacy=0x7f4f50004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:15.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:42:15.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4f50009e30 con 0x7f4f500090f0 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2202544450 0 0) 0x7f4f50009e30 con 0x7f4f500090f0 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient(hunting): my global_id is 114457 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f500065d0 con 0x7f4f500090f0 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3756487310 0 0) 0x7f4f500065d0 con 0x7f4f500090f0 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f50009e30 con 0x7f4f500090f0 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4f64003d10 con 0x7f4f500090f0 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient: dump: 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3842988315 0 0) 0x7f4f50009e30 con 0x7f4f500090f0 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4f5000c3a0 con 0x7f4f500090f0 2024-09-16T05:42:15.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 319825972) 0x7f4f64005030 con 0x7f4f500090f0 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 >> v1:172.21.15.63:6789/0 conn(0x7f4f500090f0 legacy=0x7f4f50009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f5effd640 1 -- 172.21.15.6:0/3540989115 >> v1:172.21.15.63:6789/0 conn(0x7f4f500090f0 legacy=0x7f4f50009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:42:15.104 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 34, 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 109654, 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:42:15.105 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:42:15.106 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:15.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:42:15.108 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:15.109 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:15.110 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:15.112 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:15.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:15.114 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:15.115 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:15.116 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:42:15.116 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:15.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4f4c036c20 legacy=0x7f4f4c0390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 10 monclient: shutdown 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 >> v1:172.21.15.6:6790/0 conn(0x7f4f68138e40 legacy=0x7f4f681d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f6e82c640 1 -- 172.21.15.6:0/3540989115 reap_dead start 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f6e82c640 1 -- 172.21.15.6:0/3540989115 reap_dead start 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 shutdown_connections 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 >> 172.21.15.6:0/3540989115 conn(0x7f4f680a1650 msgr2=0x7f4f680a36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 shutdown_connections 2024-09-16T05:42:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.087+0000 7f4f7028f640 1 -- 172.21.15.6:0/3540989115 wait complete. 2024-09-16T05:42:15.123 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:42:15.123 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:42:15.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:42:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.235+0000 7f01c439f640 10 monclient: get_monmap_and_config 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.235+0000 7f01c439f640 10 monclient: build_initial_monmap 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.235+0000 7f01c439f640 10 monclient: monmap: 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:15.242556+0000 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:15.242556+0000 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:15.243 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:15.243 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:15.243 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 Processor -- start 2024-09-16T05:42:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- start start 2024-09-16T05:42:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient: init 2024-09-16T05:42:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:42:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient(hunting): picked mon.noname-a con 0x7f01bc09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient(hunting): picked mon.noname-b con 0x7f01bc09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient(hunting): picked mon.noname-c con 0x7f01bc0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01bc0a1370 con 0x7f01bc09aa30 2024-09-16T05:42:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01bc09ead0 con 0x7f01bc0a0da0 2024-09-16T05:42:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01bc09edb0 con 0x7f01bc09e230 2024-09-16T05:42:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient(hunting): authenticate will time out at 1397.049189s 2024-09-16T05:42:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c193a640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f01bc09e230 0x7f01bc1399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46450/0 (socket says 172.21.15.6:46450) 2024-09-16T05:42:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c193a640 1 -- 172.21.15.6:0/3174901009 learned_addr learned my addr 172.21.15.6:0/3174901009 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4089131287 0 0) 0x7f01bc09edb0 con 0x7f01bc09e230 2024-09-16T05:42:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient(hunting): my global_id is 6617 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01a8003530 con 0x7f01bc09e230 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2890871406 0 0) 0x7f01bc0a1370 con 0x7f01bc09aa30 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient(hunting): my global_id is 124122 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01bc09edb0 con 0x7f01bc09aa30 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4096870953 0 0) 0x7f01a8003530 con 0x7f01bc09e230 2024-09-16T05:42:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 >> v1:172.21.15.63:6789/0 conn(0x7f01bc0a0da0 legacy=0x7f01bc13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 >> v1:172.21.15.6:6789/0 conn(0x7f01bc09aa30 legacy=0x7f01bc09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: _finish_hunting 0 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 monclient: found mon.noname-b 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01bc09dd50 con 0x7f01bc09e230 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: _finish_auth 0 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: _check_auth_tickets 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01ac001620 con 0x7f01bc09e230 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 5 monclient: authenticate success, global_id 6617 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:15.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: dump: 2024-09-16T05:42:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01ac001850 con 0x7f01bc09e230 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 1 -- 172.21.15.6:0/3174901009 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01ac001c60 con 0x7f01bc09e230 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c1139640 10 monclient: dump: 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.257 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.258 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.258 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.258 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.258 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.258 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.258 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.258 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 4 set_mon_vals no callback set 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient: shutdown 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- 172.21.15.6:0/3174901009 >> v1:172.21.15.6:6790/0 conn(0x7f01bc09e230 legacy=0x7f01bc1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- 172.21.15.6:0/3174901009 shutdown_connections 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- 172.21.15.6:0/3174901009 >> 172.21.15.6:0/3174901009 conn(0x7f01bc0a1650 msgr2=0x7f01bc0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- 172.21.15.6:0/3174901009 shutdown_connections 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 -- 172.21.15.6:0/3174901009 wait complete. 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient: build_initial_monmap 2024-09-16T05:42:15.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 10 monclient: monmap: 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.239+0000 7f01c439f640 1 Processor -- start 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 1 -- start start 2024-09-16T05:42:15.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: init 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient(hunting): picked mon.noname-a con 0x7f01bc09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient(hunting): picked mon.noname-c con 0x7f01bc09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient(hunting): picked mon.noname-b con 0x7f01bc0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01bc13a870 con 0x7f01bc09aa30 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01bc09c960 con 0x7f01bc0a0da0 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01bc09d5e0 con 0x7f01bc09e230 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient(hunting): authenticate will time out at 1397.052589s 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c193a640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f01bc09e230 0x7f01bc09c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46466/0 (socket says 172.21.15.6:46466) 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c193a640 1 -- 172.21.15.6:0/3239911284 learned_addr learned my addr 172.21.15.6:0/3239911284 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:15.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c213b640 1 --1- 172.21.15.6:0/3239911284 >> v1:172.21.15.6:6789/0 conn(0x7f01bc09aa30 0x7f01bc09bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36088/0 (socket says 172.21.15.6:36088) 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3753672201 0 0) 0x7f01bc09d5e0 con 0x7f01bc09e230 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): my global_id is 6620 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01a0003530 con 0x7f01bc09e230 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741531181 0 0) 0x7f01bc13a870 con 0x7f01bc09aa30 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): my global_id is 124125 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01bc09d5e0 con 0x7f01bc09aa30 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2964152533 0 0) 0x7f01bc09c960 con 0x7f01bc0a0da0 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): my global_id is 114463 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01bc13a870 con 0x7f01bc0a0da0 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3983028869 0 0) 0x7f01bc09d5e0 con 0x7f01bc09aa30 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01bc09c960 con 0x7f01bc09aa30 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2849800388 0 0) 0x7f01a0003530 con 0x7f01bc09e230 2024-09-16T05:42:15.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01bc09d5e0 con 0x7f01bc09e230 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01b8001880 con 0x7f01bc09aa30 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): dump: 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01ac001ba0 con 0x7f01bc09e230 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient(hunting): dump: 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.265 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3901854043 0 0) 0x7f01bc09d5e0 con 0x7f01bc09e230 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 >> v1:172.21.15.63:6789/0 conn(0x7f01bc0a0da0 legacy=0x7f01bc1d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 >> v1:172.21.15.6:6789/0 conn(0x7f01bc09aa30 legacy=0x7f01bc09bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 monclient: found mon.c 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01bc13ab50 con 0x7f01bc09e230 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 5 monclient: authenticate success, global_id 6620 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: _renew_subs 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01bc13a590 con 0x7f01bc09e230 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: _renew_subs 2024-09-16T05:42:15.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f01bc09d360 con 0x7f01bc09e230 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01ac001850 con 0x7f01bc09e230 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f01ac004da0 con 0x7f01bc09e230 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 10 monclient: dump: 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.268 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.269 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.269 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:15.269 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c0938640 4 set_mon_vals no callback set 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f01ac00fb40 con 0x7f01bc09e230 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a4005090 con 0x7f01bc09e230 2024-09-16T05:42:15.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f01ac044d40 con 0x7f01bc09e230 2024-09-16T05:42:15.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.243+0000 7f01c213b640 10 monclient: get_auth_request con 0x7f01a0036c00 auth_method 0 2024-09-16T05:42:15.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.247+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f01ac0153d0 con 0x7f01bc09e230 2024-09-16T05:42:15.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.247+0000 7f01b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:15.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.247+0000 7f01b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:15.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f01c439f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:42:15.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f01c439f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:42:15.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f01c439f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:15.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f01a4005b90 con 0x7f01bc09e230 2024-09-16T05:42:15.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f01b2ffd640 1 -- 172.21.15.6:0/3239911284 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f01ac01d880 con 0x7f01bc09e230 2024-09-16T05:42:15.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f01b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:42:15.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f01b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:15.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=1079343430,config=3541143443,health=1258394751,kv=2000773687,logm=4174875436}) 2024-09-16T05:42:15.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=11,kv=9,logm=60} crc {auth=1443604595,config=3541143443,health=3602034646,kv=2441816623,logm=2304506129}) 2024-09-16T05:42:15.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=1079343430,config=3541143443,health=1258394751,kv=2000773687,logm=4174875436}) 2024-09-16T05:42:15.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.479+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=1808725061,config=3541143443,health=1258394751,kv=1946505249,logm=2355020720}) 2024-09-16T05:42:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 >> v1:172.21.15.63:6812/3643207211 conn(0x7f01a0036c00 legacy=0x7f01a00390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f01c439f640 10 monclient: shutdown 2024-09-16T05:42:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f01c439f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:15.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 >> v1:172.21.15.6:6790/0 conn(0x7f01bc09e230 legacy=0x7f01bc09c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325826709}) 2024-09-16T05:42:15.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1305427559}) 2024-09-16T05:42:15.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325826709}) 2024-09-16T05:42:15.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768985396}) 2024-09-16T05:42:15.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 shutdown_connections 2024-09-16T05:42:15.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 >> 172.21.15.6:0/3239911284 conn(0x7f01bc0a1650 msgr2=0x7f01bc0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:15.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 shutdown_connections 2024-09-16T05:42:15.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:15.483+0000 7f01c439f640 1 -- 172.21.15.6:0/3239911284 wait complete. 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995778750}) 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=537955916}) 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995778750}) 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2544587465}) 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129364175}) 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4032338134}) 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129364175}) 2024-09-16T05:42:15.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122733795}) 2024-09-16T05:42:15.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336870503}) 2024-09-16T05:42:15.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628700932}) 2024-09-16T05:42:15.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336870503}) 2024-09-16T05:42:15.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.487+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1037770026}) 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2752782533}) 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2142605960}) 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2752782533}) 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3105200973}) 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108742314}) 2024-09-16T05:42:15.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=841267198}) 2024-09-16T05:42:15.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=108742314}) 2024-09-16T05:42:15.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.491+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3148557171}) 2024-09-16T05:42:15.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816785707}) 2024-09-16T05:42:15.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2568868551}) 2024-09-16T05:42:15.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816785707}) 2024-09-16T05:42:15.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4245677401}) 2024-09-16T05:42:15.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034961397}) 2024-09-16T05:42:15.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2560653638}) 2024-09-16T05:42:15.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034961397}) 2024-09-16T05:42:15.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760161429}) 2024-09-16T05:42:15.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790263767}) 2024-09-16T05:42:15.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.495+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1085862355}) 2024-09-16T05:42:15.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790263767}) 2024-09-16T05:42:15.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782157698}) 2024-09-16T05:42:15.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791677202}) 2024-09-16T05:42:15.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436462165}) 2024-09-16T05:42:15.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791677202}) 2024-09-16T05:42:15.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=567184480}) 2024-09-16T05:42:15.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398231410}) 2024-09-16T05:42:15.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4291399373}) 2024-09-16T05:42:15.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398231410}) 2024-09-16T05:42:15.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.499+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068752569}) 2024-09-16T05:42:15.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382675367}) 2024-09-16T05:42:15.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=630538398}) 2024-09-16T05:42:15.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382675367}) 2024-09-16T05:42:15.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479733596}) 2024-09-16T05:42:15.508 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:42:15.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79058225}) 2024-09-16T05:42:15.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1972090455}) 2024-09-16T05:42:15.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79058225}) 2024-09-16T05:42:15.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.503+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3142454133}) 2024-09-16T05:42:15.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.507+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.507+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1676032214}) 2024-09-16T05:42:15.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.507+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315907429}) 2024-09-16T05:42:15.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.507+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.507+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1676032214}) 2024-09-16T05:42:15.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.507+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=812253157}) 2024-09-16T05:42:15.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.511+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.511+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615602095}) 2024-09-16T05:42:15.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.511+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4046414679}) 2024-09-16T05:42:15.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.511+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.511+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615602095}) 2024-09-16T05:42:15.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.511+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860314223}) 2024-09-16T05:42:15.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.515+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.515+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708185855}) 2024-09-16T05:42:15.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.515+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1293140279}) 2024-09-16T05:42:15.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.515+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.515+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708185855}) 2024-09-16T05:42:15.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.515+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=723836750}) 2024-09-16T05:42:15.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.519+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.519+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101545970}) 2024-09-16T05:42:15.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.519+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1652784172}) 2024-09-16T05:42:15.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.519+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.519+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101545970}) 2024-09-16T05:42:15.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.519+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3454962439}) 2024-09-16T05:42:15.527 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.527 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993589622}) 2024-09-16T05:42:15.527 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974764154}) 2024-09-16T05:42:15.527 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.528 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993589622}) 2024-09-16T05:42:15.528 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2857498847}) 2024-09-16T05:42:15.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=774562266}) 2024-09-16T05:42:15.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2693675788}) 2024-09-16T05:42:15.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=774562266}) 2024-09-16T05:42:15.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.523+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462246500}) 2024-09-16T05:42:15.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983536389}) 2024-09-16T05:42:15.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156213295}) 2024-09-16T05:42:15.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983536389}) 2024-09-16T05:42:15.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2822188811}) 2024-09-16T05:42:15.532 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.532 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161697873}) 2024-09-16T05:42:15.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3990447908}) 2024-09-16T05:42:15.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161697873}) 2024-09-16T05:42:15.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.527+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3151505612}) 2024-09-16T05:42:15.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433972786}) 2024-09-16T05:42:15.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3902394381}) 2024-09-16T05:42:15.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433972786}) 2024-09-16T05:42:15.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1911332476}) 2024-09-16T05:42:15.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198607888}) 2024-09-16T05:42:15.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1793062606}) 2024-09-16T05:42:15.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198607888}) 2024-09-16T05:42:15.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3222308460}) 2024-09-16T05:42:15.538 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.538 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963547149}) 2024-09-16T05:42:15.538 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3276561054}) 2024-09-16T05:42:15.538 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.538 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963547149}) 2024-09-16T05:42:15.538 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.531+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575392375}) 2024-09-16T05:42:15.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299825207}) 2024-09-16T05:42:15.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1263983573}) 2024-09-16T05:42:15.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299825207}) 2024-09-16T05:42:15.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152827184}) 2024-09-16T05:42:15.541 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.541 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867783494}) 2024-09-16T05:42:15.541 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525339288}) 2024-09-16T05:42:15.541 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.541 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867783494}) 2024-09-16T05:42:15.541 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.535+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1896310361}) 2024-09-16T05:42:15.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188358371}) 2024-09-16T05:42:15.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2811447550}) 2024-09-16T05:42:15.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188358371}) 2024-09-16T05:42:15.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136266041}) 2024-09-16T05:42:15.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037003915}) 2024-09-16T05:42:15.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237554692}) 2024-09-16T05:42:15.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.545 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037003915}) 2024-09-16T05:42:15.545 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.539+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739342132}) 2024-09-16T05:42:15.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083532142}) 2024-09-16T05:42:15.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034789682}) 2024-09-16T05:42:15.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.547 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083532142}) 2024-09-16T05:42:15.547 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3166831856}) 2024-09-16T05:42:15.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936989794}) 2024-09-16T05:42:15.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=615271822}) 2024-09-16T05:42:15.549 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.549 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936989794}) 2024-09-16T05:42:15.549 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634232829}) 2024-09-16T05:42:15.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077206691}) 2024-09-16T05:42:15.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1191912362}) 2024-09-16T05:42:15.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077206691}) 2024-09-16T05:42:15.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.543+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4244219428}) 2024-09-16T05:42:15.551 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159557005}) 2024-09-16T05:42:15.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=586494591}) 2024-09-16T05:42:15.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159557005}) 2024-09-16T05:42:15.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825764054}) 2024-09-16T05:42:15.553 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.553 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252817991}) 2024-09-16T05:42:15.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4238410783}) 2024-09-16T05:42:15.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252817991}) 2024-09-16T05:42:15.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.547+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=881467684}) 2024-09-16T05:42:15.555 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.555 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3907637458}) 2024-09-16T05:42:15.555 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740236222}) 2024-09-16T05:42:15.555 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.555 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3907637458}) 2024-09-16T05:42:15.555 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1606194410}) 2024-09-16T05:42:15.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573903110}) 2024-09-16T05:42:15.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1843781546}) 2024-09-16T05:42:15.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573903110}) 2024-09-16T05:42:15.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.551+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867831199}) 2024-09-16T05:42:15.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855365282}) 2024-09-16T05:42:15.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=417682151}) 2024-09-16T05:42:15.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855365282}) 2024-09-16T05:42:15.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1639507723}) 2024-09-16T05:42:15.559 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.559 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089772378}) 2024-09-16T05:42:15.559 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1003301833}) 2024-09-16T05:42:15.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1089772378}) 2024-09-16T05:42:15.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.555+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382317203}) 2024-09-16T05:42:15.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.559+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.559+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=28} crc {logm=303628167,mdsmap=1832746017,mgr=222963410,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2109858417,mon_config_key=2517826709,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1327458600}) 2024-09-16T05:42:15.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.559+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=670425728,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=807484973,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=518058562}) 2024-09-16T05:42:15.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.559+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.559+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=28} crc {logm=303628167,mdsmap=1832746017,mgr=222963410,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2109858417,mon_config_key=2517826709,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1327458600}) 2024-09-16T05:42:15.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.559+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=28} crc {logm=2054793947,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=1807995258,mon_config_key=782542404,monmap=3625042558,osd_metadata=544613883,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=431604209}) 2024-09-16T05:42:15.566 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.566 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3946912025}) 2024-09-16T05:42:15.566 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=457096484}) 2024-09-16T05:42:15.566 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.566 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3946912025}) 2024-09-16T05:42:15.566 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1263518017}) 2024-09-16T05:42:15.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=510813218}) 2024-09-16T05:42:15.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2324848174}) 2024-09-16T05:42:15.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=510813218}) 2024-09-16T05:42:15.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.563+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=545431937}) 2024-09-16T05:42:15.573 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.567+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.567+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=650424641}) 2024-09-16T05:42:15.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.567+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1298032711}) 2024-09-16T05:42:15.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.567+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.567+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=650424641}) 2024-09-16T05:42:15.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.567+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=821389272}) 2024-09-16T05:42:15.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.579+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.579+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=2295422928}) 2024-09-16T05:42:15.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.579+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=25} crc {osdmap=1649395625}) 2024-09-16T05:42:15.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.579+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:42:15.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.579+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=2295422928}) 2024-09-16T05:42:15.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:42:15.579+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=1751857146}) 2024-09-16T05:42:15.769 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:42:15.770 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:42:15.770 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:42:15.770 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph auth del client.bazar 2024-09-16T05:42:15.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient: get_monmap_and_config 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient: build_initial_monmap 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient: monmap: 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:42:15.897030+0000 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:42:15.897030+0000 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:15.897 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:15.898 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:42:15.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 1 Processor -- start 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 1 -- start start 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient: init 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient(hunting): picked mon.noname-b con 0x7f4d6009ac90 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient(hunting): picked mon.noname-a con 0x7f4d6009b070 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient(hunting): picked mon.noname-c con 0x7f4d6013a9a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d600722a0 con 0x7f4d6009b070 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d600a0510 con 0x7f4d6013a9a0 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d6009e660 con 0x7f4d6009ac90 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:15.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient(hunting): authenticate will time out at 1392.674722s 2024-09-16T05:42:15.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5f59e640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f4d6013a9a0 0x7f4d6013cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51316/0 (socket says 172.21.15.63:51316) 2024-09-16T05:42:15.900 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5f59e640 1 -- 172.21.15.63:0/2586148699 learned_addr learned my addr 172.21.15.63:0/2586148699 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686430820 0 0) 0x7f4d600a0510 con 0x7f4d6013a9a0 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient(hunting): my global_id is 114469 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d4c003530 con 0x7f4d6013a9a0 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3539052285 0 0) 0x7f4d4c003530 con 0x7f4d6013a9a0 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 >> v1:172.21.15.6:6790/0 conn(0x7f4d6009ac90 legacy=0x7f4d6009f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 >> v1:172.21.15.6:6789/0 conn(0x7f4d6009b070 legacy=0x7f4d6009b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: _finish_hunting 0 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 monclient: found mon.noname-c 2024-09-16T05:42:15.905 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d601414e0 con 0x7f4d6013a9a0 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: _finish_auth 0 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: _check_auth_tickets 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d580019a0 con 0x7f4d6013a9a0 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: dump: 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.906 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 5 monclient: authenticate success, global_id 114469 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d58001c10 con 0x7f4d6013a9a0 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 1 -- 172.21.15.63:0/2586148699 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d58002020 con 0x7f4d6013a9a0 2024-09-16T05:42:15.907 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d650ce640 4 set_mon_vals no callback set 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.892+0000 7f4d5dd9b640 10 monclient: dump: 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.908 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: shutdown 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- 172.21.15.63:0/2586148699 >> v1:172.21.15.63:6789/0 conn(0x7f4d6013a9a0 legacy=0x7f4d6013cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- 172.21.15.63:0/2586148699 shutdown_connections 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- 172.21.15.63:0/2586148699 >> 172.21.15.63:0/2586148699 conn(0x7f4d600a1bf0 msgr2=0x7f4d600a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- 172.21.15.63:0/2586148699 shutdown_connections 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- 172.21.15.63:0/2586148699 wait complete. 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: build_initial_monmap 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: monmap: 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:15.909 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 Processor -- start 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- start start 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: init 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:42:15.910 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient(hunting): picked mon.noname-b con 0x7f4d6009ac90 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient(hunting): picked mon.noname-a con 0x7f4d6009b070 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient(hunting): picked mon.noname-c con 0x7f4d6013a9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d60136790 con 0x7f4d6009b070 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d6013df40 con 0x7f4d6009ac90 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d601410d0 con 0x7f4d6013a9a0 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient(hunting): authenticate will time out at 1392.677865s 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d5ed9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f4d6009ac90 0x7f4d60135970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51318/0 (socket says 172.21.15.63:51318) 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d5ed9d640 1 -- 172.21.15.63:0/244710377 learned_addr learned my addr 172.21.15.63:0/244710377 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984394036 0 0) 0x7f4d6013df40 con 0x7f4d6009ac90 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.911 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): my global_id is 114472 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d40003530 con 0x7f4d6009ac90 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450054084 0 0) 0x7f4d60136790 con 0x7f4d6009b070 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): my global_id is 124134 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d6013df40 con 0x7f4d6009b070 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 589074369 0 0) 0x7f4d40003530 con 0x7f4d6009ac90 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d60136790 con 0x7f4d6009ac90 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d480019d0 con 0x7f4d6009ac90 2024-09-16T05:42:15.912 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): dump: 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:42:15.913 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3297944508 0 0) 0x7f4d60136790 con 0x7f4d6009ac90 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 >> v1:172.21.15.6:6790/0 conn(0x7f4d6013a9a0 legacy=0x7f4d601d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 >> v1:172.21.15.6:6789/0 conn(0x7f4d6009b070 legacy=0x7f4d60136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient: _finish_hunting 0 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 monclient: found mon.b 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d60072f10 con 0x7f4d6009ac90 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient: _finish_auth 0 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient: _check_auth_tickets 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 5 monclient: authenticate success, global_id 114472 2024-09-16T05:42:15.914 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: _renew_subs 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4d601389f0 con 0x7f4d6009ac90 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: _renew_subs 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4d60138fb0 con 0x7f4d6009ac90 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d480038d0 con 0x7f4d6009ac90 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4d48003ca0 con 0x7f4d6009ac90 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 10 monclient: dump: 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:15.915 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d5d59a640 4 set_mon_vals no callback set 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d650ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4d48003e40 con 0x7f4d6009ac90 2024-09-16T05:42:15.916 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.896+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f4d48044fc0 con 0x7f4d6009ac90 2024-09-16T05:42:15.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.900+0000 7f4d650ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:15.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.900+0000 7f4d650ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:15.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.900+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d600008e0 con 0x7f4d6009ac90 2024-09-16T05:42:15.917 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.900+0000 7f4d5e59c640 10 monclient: get_auth_request con 0x7f4d40036a00 auth_method 0 2024-09-16T05:42:15.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.904+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4d48015590 con 0x7f4d6009ac90 2024-09-16T05:42:15.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.904+0000 7f4d577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:15.918 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:15.904+0000 7f4d577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:16.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.128+0000 7f4d650ce640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T05:42:16.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.128+0000 7f4d650ce640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T05:42:16.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.128+0000 7f4d650ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:16.133 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.128+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) -- 0x7f4d601379d0 con 0x7f4d6009ac90 2024-09-16T05:42:16.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.136+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v16) ==== 82+0+0 (unknown 2677726070 0 0) 0x7f4d48014e90 con 0x7f4d6009ac90 2024-09-16T05:42:16.140 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.136+0000 7f4d577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T05:42:16.141 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.136+0000 7f4d577fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:16.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.288+0000 7f4d650ce640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T05:42:16.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.288+0000 7f4d650ce640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T05:42:16.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.288+0000 7f4d650ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:42:16.295 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.288+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) -- 0x7f4d600714f0 con 0x7f4d6009ac90 2024-09-16T05:42:16.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d577fe640 1 -- 172.21.15.63:0/244710377 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v16) ==== 82+0+0 (unknown 2677726070 0 0) 0x7f4d480191e0 con 0x7f4d6009ac90 2024-09-16T05:42:16.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d577fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T05:42:16.297 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d577fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:42:16.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4d40036a00 legacy=0x7f4d40038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d650ce640 10 monclient: shutdown 2024-09-16T05:42:16.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d650ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:16.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 >> v1:172.21.15.63:6789/0 conn(0x7f4d6009ac90 legacy=0x7f4d60135970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 shutdown_connections 2024-09-16T05:42:16.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 >> 172.21.15.63:0/244710377 conn(0x7f4d600a1bf0 msgr2=0x7f4d6009f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:16.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 shutdown_connections 2024-09-16T05:42:16.299 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:42:16.292+0000 7f4d650ce640 1 -- 172.21.15.63:0/244710377 wait complete. 2024-09-16T05:42:16.312 INFO:tasks.workunit.client.0.smithi063.stderr:+ c=''\''allow command "auth ls", allow command quorum_status'\''' 2024-09-16T05:42:16.312 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 0 2024-09-16T05:42:16.312 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 2024-09-16T05:42:16.313 INFO:tasks.workunit.client.0.smithi063.stdout:ceph auth get-or-create client.foo mon 'allow command "auth ls", allow command quorum_status' > /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T05:42:16.313 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=0 2024-09-16T05:42:16.313 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T05:42:16.313 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T05:42:16.509 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:42:16.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:42:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 10 monclient: get_monmap_and_config 2024-09-16T05:42:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 10 monclient: build_initial_monmap 2024-09-16T05:42:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 10 monclient: monmap: 2024-09-16T05:42:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:16.636961+0000 2024-09-16T05:42:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:16.636961+0000 2024-09-16T05:42:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:16.638 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 1 Processor -- start 2024-09-16T05:42:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 1 -- start start 2024-09-16T05:42:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 10 monclient: init 2024-09-16T05:42:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:42:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.631+0000 7fb6f6580640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f009de60 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f009e240 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f0139460 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f009d2f0 con 0x7fb6f009de60 2024-09-16T05:42:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f013df80 con 0x7fb6f0139460 2024-09-16T05:42:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f013f160 con 0x7fb6f009e240 2024-09-16T05:42:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): authenticate will time out at 1398.444158s 2024-09-16T05:42:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6effff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb6f009de60 0x7fb6f00a0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36118/0 (socket says 172.21.15.6:36118) 2024-09-16T05:42:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6effff640 1 -- 172.21.15.6:0/320127599 learned_addr learned my addr 172.21.15.6:0/320127599 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2898253324 0 0) 0x7fb6f013f160 con 0x7fb6f009e240 2024-09-16T05:42:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient(hunting): my global_id is 6638 2024-09-16T05:42:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6d8003530 con 0x7fb6f009e240 2024-09-16T05:42:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1490996432 0 0) 0x7fb6f009d2f0 con 0x7fb6f009de60 2024-09-16T05:42:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient(hunting): my global_id is 124143 2024-09-16T05:42:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6f013f160 con 0x7fb6f009de60 2024-09-16T05:42:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 102511816 0 0) 0x7fb6d8003530 con 0x7fb6f009e240 2024-09-16T05:42:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 >> v1:172.21.15.63:6789/0 conn(0x7fb6f0139460 legacy=0x7fb6f013b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 >> v1:172.21.15.6:6789/0 conn(0x7fb6f009de60 legacy=0x7fb6f00a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:42:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 monclient: found mon.noname-b 2024-09-16T05:42:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:42:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f0140340 con 0x7fb6f009e240 2024-09-16T05:42:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: _finish_auth 0 2024-09-16T05:42:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:42:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb6e40019a0 con 0x7fb6f009e240 2024-09-16T05:42:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 5 monclient: authenticate success, global_id 6638 2024-09-16T05:42:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: dump: 2024-09-16T05:42:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:16.661 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:16.661 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:16.661 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:16.661 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:16.661 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:16.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:16.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6e4001c50 con 0x7fb6f009e240 2024-09-16T05:42:16.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:16.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 1 -- 172.21.15.6:0/320127599 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb6e4002060 con 0x7fb6f009e240 2024-09-16T05:42:16.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:16.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 4 set_mon_vals no callback set 2024-09-16T05:42:16.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:16.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:16.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6eeffd640 10 monclient: dump: 2024-09-16T05:42:16.663 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:16.663 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:16.663 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.664 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.664 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:16.664 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:16.664 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:16.664 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:16.665 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:16.665 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:16.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient: shutdown 2024-09-16T05:42:16.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:16.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- 172.21.15.6:0/320127599 >> v1:172.21.15.6:6790/0 conn(0x7fb6f009e240 legacy=0x7fb6f00a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- 172.21.15.6:0/320127599 shutdown_connections 2024-09-16T05:42:16.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- 172.21.15.6:0/320127599 >> 172.21.15.6:0/320127599 conn(0x7fb6f00a1650 msgr2=0x7fb6f00a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:16.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- 172.21.15.6:0/320127599 shutdown_connections 2024-09-16T05:42:16.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- 172.21.15.6:0/320127599 wait complete. 2024-09-16T05:42:16.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient: build_initial_monmap 2024-09-16T05:42:16.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient: monmap: 2024-09-16T05:42:16.667 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:16.667 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:16.667 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:16.667 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:16.667 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:16.667 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:16.667 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:16.667 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:16.668 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:16.668 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:16.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 Processor -- start 2024-09-16T05:42:16.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- start start 2024-09-16T05:42:16.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient: init 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f009de60 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f009e240 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f0139460 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f00a0f00 con 0x7fb6f009e240 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f0071500 con 0x7fb6f0139460 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f011c6f0 con 0x7fb6f009de60 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6f6580640 10 monclient(hunting): authenticate will time out at 1398.447279s 2024-09-16T05:42:16.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ef7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb6f009e240 0x7fb6f0070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36126/0 (socket says 172.21.15.6:36126) 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ef7fe640 1 -- 172.21.15.6:0/64132836 learned_addr learned my addr 172.21.15.6:0/64132836 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3641206310 0 0) 0x7fb6f011c6f0 con 0x7fb6f009de60 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): my global_id is 6641 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6dc003530 con 0x7fb6f009de60 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459270135 0 0) 0x7fb6f0140340 con 0x7fb6f009e240 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:16.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:16.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:16.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): my global_id is 124146 2024-09-16T05:42:16.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6f011c6f0 con 0x7fb6f009e240 2024-09-16T05:42:16.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 423991612 0 0) 0x7fb6dc003530 con 0x7fb6f009de60 2024-09-16T05:42:16.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6f0140340 con 0x7fb6f009de60 2024-09-16T05:42:16.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb6e0001840 con 0x7fb6f009de60 2024-09-16T05:42:16.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:16.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:16.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): dump: 2024-09-16T05:42:16.672 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:16.672 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:16.672 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.673 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.673 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:16.673 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:16.673 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:16.674 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:16.674 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:16.674 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:16.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.635+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1629831502 0 0) 0x7fb6f011c6f0 con 0x7fb6f009e240 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6dc003530 con 0x7fb6f009e240 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb6e4002060 con 0x7fb6f009e240 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient(hunting): dump: 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4045043575 0 0) 0x7fb6f0071500 con 0x7fb6f0139460 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient(hunting): my global_id is 114484 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6f011c6f0 con 0x7fb6f0139460 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 269178318 0 0) 0x7fb6f0140340 con 0x7fb6f009de60 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 >> v1:172.21.15.63:6789/0 conn(0x7fb6f0139460 legacy=0x7fb6f01cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 >> v1:172.21.15.6:6789/0 conn(0x7fb6f009e240 legacy=0x7fb6f0070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient: _finish_hunting 0 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 monclient: found mon.c 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f00a46a0 con 0x7fb6f009de60 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient: _finish_auth 0 2024-09-16T05:42:16.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient: _check_auth_tickets 2024-09-16T05:42:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 5 monclient: authenticate success, global_id 6641 2024-09-16T05:42:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 10 monclient: _renew_subs 2024-09-16T05:42:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f00740c0 con 0x7fb6f009de60 2024-09-16T05:42:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 10 monclient: _renew_subs 2024-09-16T05:42:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:16.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6f011c130 con 0x7fb6f009de60 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6e00047f0 con 0x7fb6f009de60 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb6e0004b40 con 0x7fb6f009de60 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ee7fc640 4 set_mon_vals no callback set 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 10 monclient: dump: 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:16.679 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb6e000f8e0 con 0x7fb6f009de60 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d0005090 con 0x7fb6f009de60 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fb6e0044ca0 con 0x7fb6f009de60 2024-09-16T05:42:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.639+0000 7fb6ef7fe640 10 monclient: get_auth_request con 0x7fb6dc036c70 auth_method 0 2024-09-16T05:42:16.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.643+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fb6e0015270 con 0x7fb6f009de60 2024-09-16T05:42:16.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.643+0000 7fb6ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:16.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.643+0000 7fb6ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:16.845 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=0 2024-09-16T05:42:16.845 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T05:42:16.845 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:42:16.845 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 0 2024-09-16T05:42:16.845 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-16T05:42:16.845 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=0 2024-09-16T05:42:16.845 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T05:42:16.846 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T05:42:16.846 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T05:42:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.863+0000 7fb6f6580640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:42:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.863+0000 7fb6f6580640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.863+0000 7fb6f6580640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.863+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb6d0005380 con 0x7fb6f009de60 2024-09-16T05:42:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6ecff9640 1 -- 172.21.15.6:0/64132836 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 4020807011) 0x7fb6e0014b70 con 0x7fb6f009de60 2024-09-16T05:42:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:16.872 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:42:16.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb6dc036c70 legacy=0x7fb6dc039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6f6580640 10 monclient: shutdown 2024-09-16T05:42:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6f6580640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 >> v1:172.21.15.6:6790/0 conn(0x7fb6f009de60 legacy=0x7fb6f00706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 shutdown_connections 2024-09-16T05:42:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 >> 172.21.15.6:0/64132836 conn(0x7fb6f00a1650 msgr2=0x7fb6f013e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 shutdown_connections 2024-09-16T05:42:16.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.867+0000 7fb6f6580640 1 -- 172.21.15.6:0/64132836 wait complete. 2024-09-16T05:42:16.889 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:42:16.889 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:42:16.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:42:16.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:42:17.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient: get_monmap_and_config 2024-09-16T05:42:17.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient: monmap: 2024-09-16T05:42:17.002 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:17.003491+0000 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:17.003491+0000 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:17.003 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 Processor -- start 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- start start 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient: init 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient(hunting): picked mon.noname-a con 0x7fcd5009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient(hunting): picked mon.noname-c con 0x7fcd5009e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient(hunting): picked mon.noname-b con 0x7fcd500a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd500a1370 con 0x7fcd5009aa30 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5009ead0 con 0x7fcd5009e230 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5009edb0 con 0x7fcd500a0da0 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient(hunting): authenticate will time out at 1398.810074s 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd54f46640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcd500a0da0 0x7fcd5013d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46504/0 (socket says 172.21.15.6:46504) 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcd5009aa30 0x7fcd5009ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36130/0 (socket says 172.21.15.6:36130) 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd54f46640 1 -- 172.21.15.6:0/3366676784 learned_addr learned my addr 172.21.15.6:0/3366676784 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1207520094 0 0) 0x7fcd500a1370 con 0x7fcd5009aa30 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): my global_id is 124155 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd3c003530 con 0x7fcd5009aa30 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2254332004 0 0) 0x7fcd5009edb0 con 0x7fcd500a0da0 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): my global_id is 6650 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd500a1370 con 0x7fcd500a0da0 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823891370 0 0) 0x7fcd5009ead0 con 0x7fcd5009e230 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient(hunting): my global_id is 114493 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd5009edb0 con 0x7fcd5009e230 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2652060506 0 0) 0x7fcd3c003530 con 0x7fcd5009aa30 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 >> v1:172.21.15.6:6790/0 conn(0x7fcd500a0da0 legacy=0x7fcd5013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 >> v1:172.21.15.63:6789/0 conn(0x7fcd5009e230 legacy=0x7fcd501399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 monclient: found mon.noname-a 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5009dd50 con 0x7fcd5009aa30 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcd480019a0 con 0x7fcd5009aa30 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 5 monclient: authenticate success, global_id 124155 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: dump: 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.014 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd48001c00 con 0x7fcd5009aa30 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 1 -- 172.21.15.6:0/3366676784 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcd48001f90 con 0x7fcd5009aa30 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd4effd640 10 monclient: dump: 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.016 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 4 set_mon_vals no callback set 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 10 monclient: shutdown 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- 172.21.15.6:0/3366676784 >> v1:172.21.15.6:6789/0 conn(0x7fcd5009aa30 legacy=0x7fcd5009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- 172.21.15.6:0/3366676784 shutdown_connections 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- 172.21.15.6:0/3366676784 >> 172.21.15.6:0/3366676784 conn(0x7fcd500a1650 msgr2=0x7fcd500a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- 172.21.15.6:0/3366676784 shutdown_connections 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:16.999+0000 7fcd569a9640 1 -- 172.21.15.6:0/3366676784 wait complete. 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: monmap: 2024-09-16T05:42:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 1 Processor -- start 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 1 -- start start 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: init 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient(hunting): picked mon.noname-c con 0x7fcd5009aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient(hunting): picked mon.noname-b con 0x7fcd5009e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient(hunting): picked mon.noname-a con 0x7fcd500a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5009c990 con 0x7fcd500a0da0 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5006e640 con 0x7fcd5009e230 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd500740c0 con 0x7fcd5009aa30 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient(hunting): authenticate will time out at 1398.812999s 2024-09-16T05:42:17.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcd5009aa30 0x7fcd5009bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46506/0 (socket says 172.21.15.6:46506) 2024-09-16T05:42:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4ffff640 1 -- 172.21.15.6:0/1742942894 learned_addr learned my addr 172.21.15.6:0/1742942894 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599196346 0 0) 0x7fcd500740c0 con 0x7fcd5009aa30 2024-09-16T05:42:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): my global_id is 6653 2024-09-16T05:42:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd30003530 con 0x7fcd5009aa30 2024-09-16T05:42:17.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912571195 0 0) 0x7fcd5009c990 con 0x7fcd500a0da0 2024-09-16T05:42:17.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): my global_id is 124158 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd500740c0 con 0x7fcd500a0da0 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708015959 0 0) 0x7fcd5006e640 con 0x7fcd5009e230 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): my global_id is 114496 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd5009c990 con 0x7fcd5009e230 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1588192692 0 0) 0x7fcd30003530 con 0x7fcd5009aa30 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd5006e640 con 0x7fcd5009aa30 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcd480018a0 con 0x7fcd5009aa30 2024-09-16T05:42:17.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): dump: 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3224167365 0 0) 0x7fcd500740c0 con 0x7fcd500a0da0 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd30003530 con 0x7fcd500a0da0 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcd40001880 con 0x7fcd500a0da0 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): dump: 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.024 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1605432467 0 0) 0x7fcd5006e640 con 0x7fcd5009aa30 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 >> v1:172.21.15.63:6789/0 conn(0x7fcd5009e230 legacy=0x7fcd5009c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 >> v1:172.21.15.6:6789/0 conn(0x7fcd500a0da0 legacy=0x7fcd501d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 monclient: found mon.c 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5009d810 con 0x7fcd5009aa30 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 5 monclient: authenticate success, global_id 6653 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: _renew_subs 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5013ccb0 con 0x7fcd5009aa30 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: _renew_subs 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcd5013b520 con 0x7fcd5009aa30 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd48001c00 con 0x7fcd5009aa30 2024-09-16T05:42:17.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcd48005060 con 0x7fcd5009aa30 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4e7fc640 4 set_mon_vals no callback set 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: dump: 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcd340065d0 con 0x7fcd34002870 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcd48005200 con 0x7fcd5009aa30 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fcd48045220 con 0x7fcd5009aa30 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1352607180 0 0) 0x7fcd340065d0 con 0x7fcd34002870 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient(hunting): my global_id is 124161 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd5006e640 con 0x7fcd34002870 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd54f46640 10 monclient: get_auth_request con 0x7fcd30036b80 auth_method 0 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1918323297 0 0) 0x7fcd5006e640 con 0x7fcd34002870 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd340065d0 con 0x7fcd34002870 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcd380017e0 con 0x7fcd34002870 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: dump: 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.029 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2921418808 0 0) 0x7fcd340065d0 con 0x7fcd34002870 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd3400a160 con 0x7fcd34002870 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcd38001ae0 con 0x7fcd34002870 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 >> v1:172.21.15.6:6789/0 conn(0x7fcd34002870 legacy=0x7fcd34004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.003+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 >> v1:172.21.15.6:6789/0 conn(0x7fcd34002870 legacy=0x7fcd34004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:17.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:42:17.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcd34009e30 con 0x7fcd340090f0 2024-09-16T05:42:17.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2620847355 0 0) 0x7fcd34009e30 con 0x7fcd340090f0 2024-09-16T05:42:17.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient(hunting): my global_id is 124164 2024-09-16T05:42:17.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd340065d0 con 0x7fcd340090f0 2024-09-16T05:42:17.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3146203392 0 0) 0x7fcd340065d0 con 0x7fcd340090f0 2024-09-16T05:42:17.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd34009e30 con 0x7fcd340090f0 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcd380037c0 con 0x7fcd340090f0 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient: dump: 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.034 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 149566922 0 0) 0x7fcd34009e30 con 0x7fcd340090f0 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd3400c3a0 con 0x7fcd340090f0 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3962 (unknown 0 0 1642318493) 0x7fcd38005a20 con 0x7fcd340090f0 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 >> v1:172.21.15.6:6789/0 conn(0x7fcd340090f0 legacy=0x7fcd34009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd4cff9640 1 -- 172.21.15.6:0/1742942894 >> v1:172.21.15.6:6789/0 conn(0x7fcd340090f0 legacy=0x7fcd34009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:17.035 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:42:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 34, 2024-09-16T05:42:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 2703, 2024-09-16T05:42:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:42:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:42:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:42:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:42:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:42:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:42:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:42:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:42:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:42:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:42:17.050 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:17.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 >> v1:172.21.15.63:6812/3643207211 conn(0x7fcd30036b80 legacy=0x7fcd30039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 10 monclient: shutdown 2024-09-16T05:42:17.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 >> v1:172.21.15.6:6790/0 conn(0x7fcd5009aa30 legacy=0x7fcd5009bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd54f46640 1 -- 172.21.15.6:0/1742942894 reap_dead start 2024-09-16T05:42:17.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd54f46640 1 -- 172.21.15.6:0/1742942894 reap_dead start 2024-09-16T05:42:17.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 shutdown_connections 2024-09-16T05:42:17.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 >> 172.21.15.6:0/1742942894 conn(0x7fcd500a1650 msgr2=0x7fcd50071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 shutdown_connections 2024-09-16T05:42:17.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.011+0000 7fcd569a9640 1 -- 172.21.15.6:0/1742942894 wait complete. 2024-09-16T05:42:17.052 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:42:17.052 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:42:17.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: get_monmap_and_config 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: monmap: 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:17.163044+0000 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:17.163044+0000 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.162 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.163 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:17.163 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:17.163 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 Processor -- start 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- start start 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: init 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient(hunting): picked mon.noname-a con 0x7f8ddc0a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient(hunting): picked mon.noname-c con 0x7f8ddc0a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient(hunting): picked mon.noname-b con 0x7f8ddc0a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ddc0a33a0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ddc0a07a0 con 0x7f8ddc0a2570 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ddc0a2d90 con 0x7f8ddc0a2980 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient(hunting): authenticate will time out at 1398.969443s 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de0d75640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ddc0a2190 0x7f8ddc0a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36170/0 (socket says 172.21.15.6:36170) 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de0d75640 1 -- 172.21.15.6:0/4048882488 learned_addr learned my addr 172.21.15.6:0/4048882488 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de1576640 1 --1- 172.21.15.6:0/4048882488 >> v1:172.21.15.6:6790/0 conn(0x7f8ddc0a2980 0x7f8ddc13f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46514/0 (socket says 172.21.15.6:46514) 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022421412 0 0) 0x7f8ddc0a33a0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient(hunting): my global_id is 124167 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc4003530 con 0x7f8ddc0a2190 2024-09-16T05:42:17.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2670728860 0 0) 0x7f8ddc0a2d90 con 0x7f8ddc0a2980 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient(hunting): my global_id is 6656 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ddc0a33a0 con 0x7f8ddc0a2980 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1183483705 0 0) 0x7f8dc4003530 con 0x7f8ddc0a2190 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 >> v1:172.21.15.6:6790/0 conn(0x7f8ddc0a2980 legacy=0x7f8ddc13f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 >> v1:172.21.15.63:6789/0 conn(0x7f8ddc0a2570 legacy=0x7f8ddc13b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc0a66e0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dd40019a0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 5 monclient: authenticate success, global_id 124167 2024-09-16T05:42:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: dump: 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8dd4001cc0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 1 -- 172.21.15.6:0/4048882488 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dd40020d0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 4 set_mon_vals no callback set 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8ddb7fe640 10 monclient: dump: 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.177 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: shutdown 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- 172.21.15.6:0/4048882488 >> v1:172.21.15.6:6789/0 conn(0x7f8ddc0a2190 legacy=0x7f8ddc0a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- 172.21.15.6:0/4048882488 shutdown_connections 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- 172.21.15.6:0/4048882488 >> 172.21.15.6:0/4048882488 conn(0x7f8ddc0a9850 msgr2=0x7f8ddc0a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- 172.21.15.6:0/4048882488 shutdown_connections 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 1 -- 172.21.15.6:0/4048882488 wait complete. 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.159+0000 7f8de2fd9640 10 monclient: monmap: 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:17.179 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 1 Processor -- start 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 1 -- start start 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient: init 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient(hunting): picked mon.noname-c con 0x7f8ddc0a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient(hunting): picked mon.noname-a con 0x7f8ddc0a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient(hunting): picked mon.noname-b con 0x7f8ddc0a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ddc137800 con 0x7f8ddc0a2570 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ddc137ae0 con 0x7f8ddc0a2980 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ddc134970 con 0x7f8ddc0a2190 2024-09-16T05:42:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient(hunting): authenticate will time out at 1398.972594s 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8ddbfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ddc0a2570 0x7f8ddc134260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36172/0 (socket says 172.21.15.6:36172) 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de0d75640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8ddc0a2190 0x7f8ddc133b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46516/0 (socket says 172.21.15.6:46516) 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8ddbfff640 1 -- 172.21.15.6:0/3445379564 learned_addr learned my addr 172.21.15.6:0/3445379564 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2356825593 0 0) 0x7f8ddc134970 con 0x7f8ddc0a2190 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): my global_id is 6659 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc8003530 con 0x7f8ddc0a2190 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 105070631 0 0) 0x7f8ddc137800 con 0x7f8ddc0a2570 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): my global_id is 124170 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ddc134970 con 0x7f8ddc0a2570 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1650972178 0 0) 0x7f8ddc137ae0 con 0x7f8ddc0a2980 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): my global_id is 114502 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ddc137800 con 0x7f8ddc0a2980 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1505371505 0 0) 0x7f8dc8003530 con 0x7f8ddc0a2190 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ddc137ae0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1248949711 0 0) 0x7f8ddc134970 con 0x7f8ddc0a2570 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dc8003530 con 0x7f8ddc0a2570 2024-09-16T05:42:17.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dd40017c0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): dump: 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.183 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dd0001800 con 0x7f8ddc0a2570 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): dump: 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.184 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2870126637 0 0) 0x7f8ddc137ae0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 >> v1:172.21.15.63:6789/0 conn(0x7f8ddc0a2980 legacy=0x7f8ddc1d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 >> v1:172.21.15.6:6789/0 conn(0x7f8ddc0a2570 legacy=0x7f8ddc134260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 monclient: found mon.c 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc1360d0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 5 monclient: authenticate success, global_id 6659 2024-09-16T05:42:17.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient: _renew_subs 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ddc0a80e0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient: _renew_subs 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ddc136570 con 0x7f8ddc0a2190 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8dd4001cc0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dd4005050 con 0x7f8ddc0a2190 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: dump: 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.186 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.187 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.187 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.187 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.187 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.187 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.187 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8ddaffd640 4 set_mon_vals no callback set 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8dd400fdf0 con 0x7f8ddc0a2190 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dbc0065d0 con 0x7f8dbc002870 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f8dd4045020 con 0x7f8ddc0a2190 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367857720 0 0) 0x7f8dbc0065d0 con 0x7f8dbc002870 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient(hunting): my global_id is 6662 2024-09-16T05:42:17.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ddc137ae0 con 0x7f8dbc002870 2024-09-16T05:42:17.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8de1576640 10 monclient: get_auth_request con 0x7f8dc8036d60 auth_method 0 2024-09-16T05:42:17.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3808107940 0 0) 0x7f8ddc137ae0 con 0x7f8dbc002870 2024-09-16T05:42:17.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dbc0065d0 con 0x7f8dbc002870 2024-09-16T05:42:17.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dd0003620 con 0x7f8dbc002870 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: dump: 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.190 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.191 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2916573670 0 0) 0x7f8dbc0065d0 con 0x7f8dbc002870 2024-09-16T05:42:17.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8dbc00a160 con 0x7f8dbc002870 2024-09-16T05:42:17.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8dd00057b0 con 0x7f8dbc002870 2024-09-16T05:42:17.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:17.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 >> v1:172.21.15.6:6790/0 conn(0x7f8dbc002870 legacy=0x7f8dbc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.163+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 >> v1:172.21.15.6:6790/0 conn(0x7f8dbc002870 legacy=0x7f8dbc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:17.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dbc009e50 con 0x7f8dbc009110 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316027838 0 0) 0x7f8dbc009e50 con 0x7f8dbc009110 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient(hunting): my global_id is 6665 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dbc0065d0 con 0x7f8dbc009110 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3979931218 0 0) 0x7f8dbc0065d0 con 0x7f8dbc009110 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dbc009e50 con 0x7f8dbc009110 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dd0003ce0 con 0x7f8dbc009110 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient: dump: 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1205584879 0 0) 0x7f8dbc009e50 con 0x7f8dbc009110 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8dbc00c3e0 con 0x7f8dbc009110 2024-09-16T05:42:17.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 1385471050) 0x7f8dd0003fc0 con 0x7f8dbc009110 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 >> v1:172.21.15.6:6790/0 conn(0x7f8dbc009110 legacy=0x7f8dbc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8dd97fa640 1 -- 172.21.15.6:0/3445379564 >> v1:172.21.15.6:6790/0 conn(0x7f8dbc009110 legacy=0x7f8dbc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:42:17.195 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 34, 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 466228, 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.197 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:42:17.198 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:17.199 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:42:17.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.202 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:17.203 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.204 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:42:17.205 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.206 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:42:17.207 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:17.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8dc8036d60 legacy=0x7f8dc8039220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 10 monclient: shutdown 2024-09-16T05:42:17.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 >> v1:172.21.15.6:6790/0 conn(0x7f8ddc0a2190 legacy=0x7f8ddc133b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de1576640 1 -- 172.21.15.6:0/3445379564 reap_dead start 2024-09-16T05:42:17.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de1576640 1 -- 172.21.15.6:0/3445379564 reap_dead start 2024-09-16T05:42:17.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 shutdown_connections 2024-09-16T05:42:17.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 >> 172.21.15.6:0/3445379564 conn(0x7f8ddc0a9850 msgr2=0x7f8ddc12fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 shutdown_connections 2024-09-16T05:42:17.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.171+0000 7f8de2fd9640 1 -- 172.21.15.6:0/3445379564 wait complete. 2024-09-16T05:42:17.209 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:42:17.209 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:42:17.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:42:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: get_monmap_and_config 2024-09-16T05:42:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: monmap: 2024-09-16T05:42:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:17.319373+0000 2024-09-16T05:42:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:17.319373+0000 2024-09-16T05:42:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:17.319 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:17.319 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 Processor -- start 2024-09-16T05:42:17.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- start start 2024-09-16T05:42:17.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: init 2024-09-16T05:42:17.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:42:17.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient(hunting): picked mon.noname-a con 0x7f70b409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient(hunting): picked mon.noname-b con 0x7f70b409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient(hunting): picked mon.noname-c con 0x7f70b40a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b40a1370 con 0x7f70b409aa30 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b409ead0 con 0x7f70b40a0da0 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b409edb0 con 0x7f70b409e230 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient(hunting): authenticate will time out at 1399.125529s 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70b8e2c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f70b409e230 0x7f70b41399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46544/0 (socket says 172.21.15.6:46544) 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70b8e2c640 1 -- 172.21.15.6:0/949606700 learned_addr learned my addr 172.21.15.6:0/949606700 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70b962d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70b409aa30 0x7f70b409ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36174/0 (socket says 172.21.15.6:36174) 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3105923796 0 0) 0x7f70b40a1370 con 0x7f70b409aa30 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient(hunting): my global_id is 124173 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70a0003530 con 0x7f70b409aa30 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1123823549 0 0) 0x7f70b409edb0 con 0x7f70b409e230 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient(hunting): my global_id is 6668 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b40a1370 con 0x7f70b409e230 2024-09-16T05:42:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 95541923 0 0) 0x7f70a0003530 con 0x7f70b409aa30 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 >> v1:172.21.15.6:6790/0 conn(0x7f70b409e230 legacy=0x7f70b41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 >> v1:172.21.15.63:6789/0 conn(0x7f70b40a0da0 legacy=0x7f70b413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 monclient: found mon.noname-a 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b409dd50 con 0x7f70b409aa30 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70a8001620 con 0x7f70b409aa30 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: dump: 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 5 monclient: authenticate success, global_id 124173 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70a8001810 con 0x7f70b409aa30 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 4 set_mon_vals no callback set 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 1 -- 172.21.15.6:0/949606700 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70a8001c20 con 0x7f70b409aa30 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70a7fff640 10 monclient: dump: 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.329 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: shutdown 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- 172.21.15.6:0/949606700 >> v1:172.21.15.6:6789/0 conn(0x7f70b409aa30 legacy=0x7f70b409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- 172.21.15.6:0/949606700 shutdown_connections 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- 172.21.15.6:0/949606700 >> 172.21.15.6:0/949606700 conn(0x7f70b40a1650 msgr2=0x7f70b40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- 172.21.15.6:0/949606700 shutdown_connections 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- 172.21.15.6:0/949606700 wait complete. 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: monmap: 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 Processor -- start 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 1 -- start start 2024-09-16T05:42:17.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.315+0000 7f70bb891640 10 monclient: init 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient(hunting): picked mon.noname-a con 0x7f70b409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient(hunting): picked mon.noname-b con 0x7f70b409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient(hunting): picked mon.noname-c con 0x7f70b40a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b409c990 con 0x7f70b409aa30 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b413cf80 con 0x7f70b409e230 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b40740c0 con 0x7f70b40a0da0 2024-09-16T05:42:17.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient(hunting): authenticate will time out at 1399.128355s 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70b962d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70b409aa30 0x7f70b409bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36184/0 (socket says 172.21.15.6:36184) 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70b962d640 1 -- 172.21.15.6:0/1828834 learned_addr learned my addr 172.21.15.6:0/1828834 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70b9e2e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f70b40a0da0 0x7f70b41d37b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46558/0 (socket says 172.21.15.6:46558) 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2262622155 0 0) 0x7f70b40740c0 con 0x7f70b40a0da0 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): my global_id is 6671 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7098003530 con 0x7f70b40a0da0 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4245539697 0 0) 0x7f70b409c990 con 0x7f70b409aa30 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): my global_id is 124176 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b40740c0 con 0x7f70b409aa30 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3238007804 0 0) 0x7f7098003530 con 0x7f70b40a0da0 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b409c990 con 0x7f70b40a0da0 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70ac001bc0 con 0x7f70b40a0da0 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): dump: 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.334 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 384995193 0 0) 0x7f70b40740c0 con 0x7f70b409aa30 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7098003530 con 0x7f70b409aa30 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70a8001c20 con 0x7f70b409aa30 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): dump: 2024-09-16T05:42:17.335 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103314141 0 0) 0x7f70b413cf80 con 0x7f70b409e230 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): my global_id is 114508 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b40740c0 con 0x7f70b409e230 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1430493127 0 0) 0x7f70b409c990 con 0x7f70b40a0da0 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 >> v1:172.21.15.63:6789/0 conn(0x7f70b409e230 legacy=0x7f70b409c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 >> v1:172.21.15.6:6789/0 conn(0x7f70b409aa30 legacy=0x7f70b409bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 monclient: found mon.c 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b413dac0 con 0x7f70b40a0da0 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 5 monclient: authenticate success, global_id 6671 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient: _renew_subs 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70b406e980 con 0x7f70b40a0da0 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient: _renew_subs 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f70b413cca0 con 0x7f70b40a0da0 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70ac0039e0 con 0x7f70b40a0da0 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70ac003db0 con 0x7f70b40a0da0 2024-09-16T05:42:17.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: dump: 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a77fe640 4 set_mon_vals no callback set 2024-09-16T05:42:17.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f70ac003f90 con 0x7f70b40a0da0 2024-09-16T05:42:17.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f70ac045120 con 0x7f70b40a0da0 2024-09-16T05:42:17.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f70b40008e0 con 0x7f70b409b520 2024-09-16T05:42:17.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70b962d640 10 monclient: get_auth_request con 0x7f7098036c70 auth_method 0 2024-09-16T05:42:17.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586183404 0 0) 0x7f70b40008e0 con 0x7f70b409b520 2024-09-16T05:42:17.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient(hunting): my global_id is 114511 2024-09-16T05:42:17.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b409c990 con 0x7f70b409b520 2024-09-16T05:42:17.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3001603052 0 0) 0x7f70b409c990 con 0x7f70b409b520 2024-09-16T05:42:17.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b40008e0 con 0x7f70b409b520 2024-09-16T05:42:17.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70b00012e0 con 0x7f70b409b520 2024-09-16T05:42:17.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:17.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 10 monclient: dump: 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1356222677 0 0) 0x7f70b40008e0 con 0x7f70b409b520 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.319+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70b40635b0 con 0x7f70b409b520 2024-09-16T05:42:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.323+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f70b0001140 con 0x7f70b409b520 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.323+0000 7f70a5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.323+0000 7f70a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.323+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 >> v1:172.21.15.63:6789/0 conn(0x7f70b409b520 legacy=0x7f70b4062ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.323+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 >> v1:172.21.15.63:6789/0 conn(0x7f70b409b520 legacy=0x7f70b4062ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70bb891640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f70b41380e0 con 0x7f70b402d7d0 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235833176 0 0) 0x7f70b41380e0 con 0x7f70b402d7d0 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 10 monclient(hunting): my global_id is 114514 2024-09-16T05:42:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70b40008e0 con 0x7f70b402d7d0 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2526210221 0 0) 0x7f70b40008e0 con 0x7f70b402d7d0 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70b41380e0 con 0x7f70b402d7d0 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f70b0004990 con 0x7f70b402d7d0 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 10 monclient: dump: 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 479718564 0 0) 0x7f70b41380e0 con 0x7f70b402d7d0 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.327+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70b402dbb0 con 0x7f70b402d7d0 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 1140827276) 0x7f70b0004990 con 0x7f70b402d7d0 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70a5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70a5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 >> v1:172.21.15.63:6789/0 conn(0x7f70b402d7d0 legacy=0x7f70b41379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70a5ffb640 1 -- 172.21.15.6:0/1828834 >> v1:172.21.15.63:6789/0 conn(0x7f70b402d7d0 legacy=0x7f70b41379d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:17.347 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 34, 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 111894, 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:42:17.348 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.349 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.350 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:42:17.351 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:17.352 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.353 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.354 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.356 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:42:17.357 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.358 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:17.359 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:17.360 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:42:17.360 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:17.361 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=0 2024-09-16T05:42:17.361 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T05:42:17.361 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:42:17.361 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 0 2024-09-16T05:42:17.361 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 2024-09-16T05:42:17.362 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=0 2024-09-16T05:42:17.362 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T05:42:17.362 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T05:42:17.363 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7098036c70 legacy=0x7f7098039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70bb891640 10 monclient: shutdown 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70bb891640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 >> v1:172.21.15.6:6790/0 conn(0x7f70b40a0da0 legacy=0x7f70b41d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70b9e2e640 1 -- 172.21.15.6:0/1828834 reap_dead start 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70b9e2e640 1 -- 172.21.15.6:0/1828834 reap_dead start 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 shutdown_connections 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 >> 172.21.15.6:0/1828834 conn(0x7f70b40a1650 msgr2=0x7f70b406ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 shutdown_connections 2024-09-16T05:42:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.331+0000 7f70bb891640 1 -- 172.21.15.6:0/1828834 wait complete. 2024-09-16T05:42:17.364 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:42:17.364 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:42:17.364 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:42:17.364 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:42:17.364 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-16T05:42:17.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-16T05:42:17.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient: get_monmap_and_config 2024-09-16T05:42:17.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient: monmap: 2024-09-16T05:42:17.472 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.472 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.472 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:17.472764+0000 2024-09-16T05:42:17.472 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:17.472764+0000 2024-09-16T05:42:17.473 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.473 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.473 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.473 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:17.473 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:17.473 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 1 Processor -- start 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 1 -- start start 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient: init 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient(hunting): picked mon.noname-c con 0x7fb82c09aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient(hunting): picked mon.noname-b con 0x7fb82c09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient(hunting): picked mon.noname-a con 0x7fb82c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c0a1370 con 0x7fb82c0a0da0 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c09ead0 con 0x7fb82c09aa30 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c09edb0 con 0x7fb82c09e230 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb82c09e230 0x7fb82c1399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46570/0 (socket says 172.21.15.6:46570) 2024-09-16T05:42:17.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb82c0a0da0 0x7fb82c13d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36188/0 (socket says 172.21.15.6:36188) 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82affd640 1 -- 172.21.15.6:0/1688064386 learned_addr learned my addr 172.21.15.6:0/1688064386 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb831f1a640 10 monclient(hunting): authenticate will time out at 1399.279595s 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2860692950 0 0) 0x7fb82c09edb0 con 0x7fb82c09e230 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): my global_id is 6680 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb80c003530 con 0x7fb82c09e230 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3411138501 0 0) 0x7fb82c0a1370 con 0x7fb82c0a0da0 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): my global_id is 124185 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb82c09edb0 con 0x7fb82c0a0da0 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760775873 0 0) 0x7fb82c09ead0 con 0x7fb82c09aa30 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 10 monclient(hunting): my global_id is 114523 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.467+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb82c0a1370 con 0x7fb82c09aa30 2024-09-16T05:42:17.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3465328274 0 0) 0x7fb82c09edb0 con 0x7fb82c0a0da0 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 >> v1:172.21.15.6:6790/0 conn(0x7fb82c09e230 legacy=0x7fb82c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 >> v1:172.21.15.63:6789/0 conn(0x7fb82c09aa30 legacy=0x7fb82c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 1 monclient: found mon.noname-a 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82c09dd50 con 0x7fb82c0a0da0 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb824001530 con 0x7fb82c0a0da0 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 5 monclient: authenticate success, global_id 124185 2024-09-16T05:42:17.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: dump: 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb824001820 con 0x7fb82c0a0da0 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 1 -- 172.21.15.6:0/1688064386 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb824001c30 con 0x7fb82c0a0da0 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 4 set_mon_vals no callback set 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82a7fc640 10 monclient: dump: 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.488 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: shutdown 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- 172.21.15.6:0/1688064386 >> v1:172.21.15.6:6789/0 conn(0x7fb82c0a0da0 legacy=0x7fb82c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- 172.21.15.6:0/1688064386 shutdown_connections 2024-09-16T05:42:17.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- 172.21.15.6:0/1688064386 >> 172.21.15.6:0/1688064386 conn(0x7fb82c0a1650 msgr2=0x7fb82c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- 172.21.15.6:0/1688064386 shutdown_connections 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- 172.21.15.6:0/1688064386 wait complete. 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: monmap: 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:17.490 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 Processor -- start 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- start start 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: init 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient(hunting): picked mon.noname-c con 0x7fb82c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient(hunting): picked mon.noname-a con 0x7fb82c09e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient(hunting): picked mon.noname-b con 0x7fb82c0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c133eb0 con 0x7fb82c09e230 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c06e8e0 con 0x7fb82c0a0da0 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82c135080 con 0x7fb82c09aa30 2024-09-16T05:42:17.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient(hunting): authenticate will time out at 1399.282243s 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb82c09e230 0x7fb82c13d070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36202/0 (socket says 172.21.15.6:36202) 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb82affd640 1 -- 172.21.15.6:0/2008780054 learned_addr learned my addr 172.21.15.6:0/2008780054 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856360349 0 0) 0x7fb82c133eb0 con 0x7fb82c09e230 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): my global_id is 124188 2024-09-16T05:42:17.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb810003560 con 0x7fb82c09e230 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1209860482 0 0) 0x7fb82c135080 con 0x7fb82c09aa30 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): my global_id is 6683 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb82c133eb0 con 0x7fb82c09aa30 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 603093277 0 0) 0x7fb82c06e8e0 con 0x7fb82c0a0da0 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): my global_id is 114526 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb82c135080 con 0x7fb82c0a0da0 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1646915910 0 0) 0x7fb810003560 con 0x7fb82c09e230 2024-09-16T05:42:17.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb82c06e8e0 con 0x7fb82c09e230 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb814001dc0 con 0x7fb82c09e230 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): dump: 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.494 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:17.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 367987370 0 0) 0x7fb82c133eb0 con 0x7fb82c09aa30 2024-09-16T05:42:17.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb810003560 con 0x7fb82c09aa30 2024-09-16T05:42:17.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb81c001820 con 0x7fb82c09aa30 2024-09-16T05:42:17.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): dump: 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.496 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1518345399 0 0) 0x7fb82c135080 con 0x7fb82c0a0da0 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb82c133eb0 con 0x7fb82c0a0da0 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb8240014f0 con 0x7fb82c0a0da0 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient(hunting): dump: 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.497 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2594490121 0 0) 0x7fb82c06e8e0 con 0x7fb82c09e230 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 >> v1:172.21.15.6:6790/0 conn(0x7fb82c09aa30 legacy=0x7fb82c1c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 >> v1:172.21.15.63:6789/0 conn(0x7fb82c0a0da0 legacy=0x7fb82c1d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 monclient: found mon.a 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82c135ae0 con 0x7fb82c09e230 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 5 monclient: authenticate success, global_id 124188 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: _renew_subs 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb82c133c30 con 0x7fb82c09e230 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: _renew_subs 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb82c13d950 con 0x7fb82c09e230 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb814001f60 con 0x7fb82c09e230 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb814004fc0 con 0x7fb82c09e230 2024-09-16T05:42:17.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb829ffb640 4 set_mon_vals no callback set 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb823fff640 10 monclient: dump: 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.500 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.471+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb8040065d0 con 0x7fb804002870 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb81400fbc0 con 0x7fb82c09e230 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fb814045070 con 0x7fb82c09e230 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2966998867 0 0) 0x7fb8040065d0 con 0x7fb804002870 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient(hunting): my global_id is 114529 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb82c06e8e0 con 0x7fb804002870 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb82bfff640 10 monclient: get_auth_request con 0x7fb810036c40 auth_method 0 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4252278247 0 0) 0x7fb82c06e8e0 con 0x7fb804002870 2024-09-16T05:42:17.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8040065d0 con 0x7fb804002870 2024-09-16T05:42:17.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb81c003990 con 0x7fb804002870 2024-09-16T05:42:17.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient: dump: 2024-09-16T05:42:17.503 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2748248365 0 0) 0x7fb8040065d0 con 0x7fb804002870 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb80400a160 con 0x7fb804002870 2024-09-16T05:42:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb81c004e70 con 0x7fb804002870 2024-09-16T05:42:17.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:17.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 >> v1:172.21.15.63:6789/0 conn(0x7fb804002870 legacy=0x7fb804004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.475+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 >> v1:172.21.15.63:6789/0 conn(0x7fb804002870 legacy=0x7fb804004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T05:42:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb804009e60 con 0x7fb804009120 2024-09-16T05:42:17.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046950702 0 0) 0x7fb804009e60 con 0x7fb804009120 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient(hunting): my global_id is 114532 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8040065d0 con 0x7fb804009120 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4173162190 0 0) 0x7fb8040065d0 con 0x7fb804009120 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb804009e60 con 0x7fb804009120 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb81c0055b0 con 0x7fb804009120 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient: dump: 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 841030314 0 0) 0x7fb804009e60 con 0x7fb804009120 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fb80400c3e0 con 0x7fb804009120 2024-09-16T05:42:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7fb81c0053d0 con 0x7fb804009120 2024-09-16T05:42:17.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T05:42:17.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:17.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 >> v1:172.21.15.63:6789/0 conn(0x7fb804009120 legacy=0x7fb8040095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb823fff640 1 -- 172.21.15.6:0/2008780054 >> v1:172.21.15.63:6789/0 conn(0x7fb804009120 legacy=0x7fb8040095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:17.509 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "ret": 0, 2024-09-16T05:42:17.509 INFO:teuthology.orchestra.run.smithi006.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T05:42:17.509 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:17.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb810036c40 legacy=0x7fb810039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 10 monclient: shutdown 2024-09-16T05:42:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 >> v1:172.21.15.6:6789/0 conn(0x7fb82c09e230 legacy=0x7fb82c13d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb82bfff640 1 -- 172.21.15.6:0/2008780054 reap_dead start 2024-09-16T05:42:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb82bfff640 1 -- 172.21.15.6:0/2008780054 reap_dead start 2024-09-16T05:42:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 shutdown_connections 2024-09-16T05:42:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 >> 172.21.15.6:0/2008780054 conn(0x7fb82c0a1650 msgr2=0x7fb82c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 shutdown_connections 2024-09-16T05:42:17.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.483+0000 7fb831f1a640 1 -- 172.21.15.6:0/2008780054 wait complete. 2024-09-16T05:42:17.511 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:42:17.512 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:42:17.512 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:42:17.526 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:42:17.526 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:42:17.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:42:17.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient: get_monmap_and_config 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient: monmap: 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:17.644826+0000 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:17.644826+0000 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.644 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.645 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:17.645 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:17.645 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 1 Processor -- start 2024-09-16T05:42:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 1 -- start start 2024-09-16T05:42:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient: init 2024-09-16T05:42:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:42:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient(hunting): picked mon.noname-b con 0x7fb57c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient(hunting): picked mon.noname-a con 0x7fb57c09b520 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient(hunting): picked mon.noname-c con 0x7fb57c141690 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c0a1350 con 0x7fb57c09b520 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c09cdf0 con 0x7fb57c141690 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c09dad0 con 0x7fb57c09aa30 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb582637640 10 monclient(hunting): authenticate will time out at 1399.451384s 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb580bd4640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fb57c141690 legacy=0x7fb57c143a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb57b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb57c09b520 0x7fb57c13ff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36210/0 (socket says 172.21.15.6:36210) 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb57bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb57c09aa30 0x7fb57c09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46582/0 (socket says 172.21.15.6:46582) 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb57b7fe640 1 -- 172.21.15.6:0/1805933721 learned_addr learned my addr 172.21.15.6:0/1805933721 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762007579 0 0) 0x7fb570001120 con 0x7fb57c09b520 2024-09-16T05:42:17.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb57affd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb57affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb57affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.639+0000 7fb57affd640 10 monclient(hunting): my global_id is 124191 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb564003530 con 0x7fb57c09b520 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 13984868 0 0) 0x7fb564003530 con 0x7fb57c09b520 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 >> v1:172.21.15.6:6790/0 conn(0x7fb57c09aa30 legacy=0x7fb57c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 >> v1:172.21.15.63:6789/0 conn(0x7fb57c141690 legacy=0x7fb57c143a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 monclient: found mon.noname-a 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c0a10d0 con 0x7fb57c09b520 2024-09-16T05:42:17.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb5700017f0 con 0x7fb57c09b520 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 5 monclient: authenticate success, global_id 124191 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: dump: 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.654 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb570001ae0 con 0x7fb57c09b520 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 1 -- 172.21.15.6:0/1805933721 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb5700048b0 con 0x7fb57c09b520 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:17.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 4 set_mon_vals no callback set 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57affd640 10 monclient: dump: 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: shutdown 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- 172.21.15.6:0/1805933721 >> v1:172.21.15.6:6789/0 conn(0x7fb57c09b520 legacy=0x7fb57c13ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- 172.21.15.6:0/1805933721 shutdown_connections 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- 172.21.15.6:0/1805933721 >> 172.21.15.6:0/1805933721 conn(0x7fb57c0a1650 msgr2=0x7fb57c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- 172.21.15.6:0/1805933721 shutdown_connections 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- 172.21.15.6:0/1805933721 wait complete. 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: build_initial_monmap 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: monmap: 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:17.657 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 Processor -- start 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- start start 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: init 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient(hunting): picked mon.noname-b con 0x7fb57c09aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:17.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient(hunting): picked mon.noname-c con 0x7fb57c09b520 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient(hunting): picked mon.noname-a con 0x7fb57c141690 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c125de0 con 0x7fb57c141690 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c0740c0 con 0x7fb57c09aa30 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c141330 con 0x7fb57c09b520 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient(hunting): authenticate will time out at 1399.454626s 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57bfff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fb57c09aa30 legacy=0x7fb57c13aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb580bd4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb57c141690 0x7fb57c1d8180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36222/0 (socket says 172.21.15.6:36222) 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb580bd4640 1 -- 172.21.15.6:0/3356574973 learned_addr learned my addr 172.21.15.6:0/3356574973 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030715851 0 0) 0x7fb574003a00 con 0x7fb57c141690 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:17.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient(hunting): my global_id is 124194 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb568003530 con 0x7fb57c141690 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 868818435 0 0) 0x7fb568003530 con 0x7fb57c141690 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb574003a00 con 0x7fb57c141690 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb574004210 con 0x7fb57c141690 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient(hunting): dump: 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.660 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 280861495 0 0) 0x7fb574003a00 con 0x7fb57c141690 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 >> v1:172.21.15.6:6790/0 conn(0x7fb57c09b520 legacy=0x7fb57c1d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 >> v1:172.21.15.63:6789/0 conn(0x7fb57c09aa30 legacy=0x7fb57c13aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 monclient: found mon.a 2024-09-16T05:42:17.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c1409a0 con 0x7fb57c141690 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient: _finish_auth 0 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 5 monclient: authenticate success, global_id 124194 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: _renew_subs 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb57c1266e0 con 0x7fb57c141690 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: _renew_subs 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb57c13ff70 con 0x7fb57c141690 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb574004570 con 0x7fb57c141690 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb574004940 con 0x7fb57c141690 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb57a7fc640 4 set_mon_vals no callback set 2024-09-16T05:42:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.643+0000 7fb578ff9640 10 monclient: dump: 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.647+0000 7fb582637640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.647+0000 7fb582637640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.647+0000 7fb582637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.647+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb574004ae0 con 0x7fb57c141690 2024-09-16T05:42:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.647+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb55c005090 con 0x7fb57c141690 2024-09-16T05:42:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.647+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fb574044f20 con 0x7fb57c141690 2024-09-16T05:42:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.647+0000 7fb57bfff640 10 monclient: get_auth_request con 0x7fb5680369f0 auth_method 0 2024-09-16T05:42:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.651+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fb57c1266e0 con 0x7fb57c141690 2024-09-16T05:42:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.651+0000 7fb578ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:17.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.651+0000 7fb578ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:17.866 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=0 2024-09-16T05:42:17.866 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T05:42:17.866 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:42:17.866 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T05:42:17.866 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 13 2024-09-16T05:42:17.866 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 2024-09-16T05:42:17.867 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:42:17.867 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T05:42:17.867 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T05:42:17.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.871+0000 7fb582637640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:42:17.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.871+0000 7fb582637640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:17.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.871+0000 7fb582637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:17.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.871+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb55c005380 con 0x7fb57c141690 2024-09-16T05:42:17.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.871+0000 7fb578ff9640 1 -- 172.21.15.6:0/3356574973 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3960722248) 0x7fb57c125de0 con 0x7fb57c141690 2024-09-16T05:42:17.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.871+0000 7fb578ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:17.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.871+0000 7fb578ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:17.877 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:42:17.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.875+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb5680369f0 legacy=0x7fb568038eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.875+0000 7fb582637640 10 monclient: shutdown 2024-09-16T05:42:17.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.875+0000 7fb582637640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:17.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.875+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 >> v1:172.21.15.6:6789/0 conn(0x7fb57c141690 legacy=0x7fb57c1d8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:17.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.875+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 shutdown_connections 2024-09-16T05:42:17.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.875+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 >> 172.21.15.6:0/3356574973 conn(0x7fb57c0a1650 msgr2=0x7fb57c0a3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:17.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.875+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 shutdown_connections 2024-09-16T05:42:17.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:17.875+0000 7fb582637640 1 -- 172.21.15.6:0/3356574973 wait complete. 2024-09-16T05:42:18.223 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:42:18.223 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:42:18.223 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:42:18.223 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 13 2024-09-16T05:42:18.224 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 2024-09-16T05:42:18.224 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:42:18.224 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T05:42:18.224 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T05:42:18.224 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T05:42:18.588 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:42:18.589 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:42:18.589 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:42:18.589 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T05:42:18.589 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 13 2024-09-16T05:42:18.589 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 2024-09-16T05:42:18.589 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:42:18.590 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T05:42:18.590 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T05:42:18.884 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:42:18.884 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:42:18.884 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:42:18.884 INFO:tasks.workunit.client.0.smithi063.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T05:42:18.884 INFO:tasks.workunit.client.0.smithi063.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T05:42:18.884 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 110 2024-09-16T05:42:18.885 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 2024-09-16T05:42:18.885 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=110 2024-09-16T05:42:18.885 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T05:42:18.886 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T05:42:18.886 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T05:42:20.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:42:21.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient: get_monmap_and_config 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient: build_initial_monmap 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient: monmap: 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:21.028596+0000 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:21.028596+0000 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:21.028 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 1 Processor -- start 2024-09-16T05:42:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 1 -- start start 2024-09-16T05:42:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient: init 2024-09-16T05:42:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient(hunting): picked mon.noname-a con 0x7fde8009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient(hunting): picked mon.noname-b con 0x7fde8009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient(hunting): picked mon.noname-c con 0x7fde800a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde800a1370 con 0x7fde8009aa30 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8009ead0 con 0x7fde800a0da0 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8009edb0 con 0x7fde8009e230 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde854b2640 10 monclient(hunting): authenticate will time out at 1402.835178s 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7f7fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fde800a0da0 legacy=0x7fde8013d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fde8009aa30 0x7fde8009ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32946/0 (socket says 172.21.15.6:32946) 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7effd640 1 -- 172.21.15.6:0/1571095552 learned_addr learned my addr 172.21.15.6:0/1571095552 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2306748673 0 0) 0x7fde800a1370 con 0x7fde8009aa30 2024-09-16T05:42:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7dffb640 10 monclient(hunting): my global_id is 124221 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.023+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde6c003530 con 0x7fde8009aa30 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 482189689 0 0) 0x7fde6c003530 con 0x7fde8009aa30 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 >> v1:172.21.15.6:6790/0 conn(0x7fde8009e230 legacy=0x7fde801399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 >> v1:172.21.15.63:6789/0 conn(0x7fde800a0da0 legacy=0x7fde8013d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 1 monclient: found mon.noname-a 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8009dd50 con 0x7fde8009aa30 2024-09-16T05:42:21.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: _finish_auth 0 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde68001620 con 0x7fde8009aa30 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: dump: 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:21.034 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 5 monclient: authenticate success, global_id 124221 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde68001910 con 0x7fde8009aa30 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 1 -- 172.21.15.6:0/1571095552 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde68001ce0 con 0x7fde8009aa30 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7dffb640 10 monclient: dump: 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:21.035 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 4 set_mon_vals no callback set 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: shutdown 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- 172.21.15.6:0/1571095552 >> v1:172.21.15.6:6789/0 conn(0x7fde8009aa30 legacy=0x7fde8009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- 172.21.15.6:0/1571095552 shutdown_connections 2024-09-16T05:42:21.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- 172.21.15.6:0/1571095552 >> 172.21.15.6:0/1571095552 conn(0x7fde800a1650 msgr2=0x7fde800a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- 172.21.15.6:0/1571095552 shutdown_connections 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- 172.21.15.6:0/1571095552 wait complete. 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: build_initial_monmap 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: monmap: 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:21.037 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:21.038 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 Processor -- start 2024-09-16T05:42:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- start start 2024-09-16T05:42:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: init 2024-09-16T05:42:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:42:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient(hunting): picked mon.noname-c con 0x7fde8009aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient(hunting): picked mon.noname-a con 0x7fde8009e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:21.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient(hunting): picked mon.noname-b con 0x7fde800a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde801ca4d0 con 0x7fde8009e230 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8006e640 con 0x7fde800a0da0 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde800740c0 con 0x7fde8009aa30 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7e7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fde8009e230 0x7fde801c9dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32958/0 (socket says 172.21.15.6:32958) 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7e7fc640 1 -- 172.21.15.6:0/3224076502 learned_addr learned my addr 172.21.15.6:0/3224076502 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient(hunting): authenticate will time out at 1402.838601s 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7f7fe640 1 -- 172.21.15.6:0/3224076502 >> v1:172.21.15.63:6789/0 conn(0x7fde800a0da0 legacy=0x7fde801d1910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 613527957 0 0) 0x7fde801ca4d0 con 0x7fde8009e230 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:21.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient(hunting): my global_id is 124224 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde60003530 con 0x7fde8009e230 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3334556836 0 0) 0x7fde60003530 con 0x7fde8009e230 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde801ca4d0 con 0x7fde8009e230 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde700018c0 con 0x7fde8009e230 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient(hunting): dump: 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:21.044 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2905127136 0 0) 0x7fde801ca4d0 con 0x7fde8009e230 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 >> v1:172.21.15.6:6790/0 conn(0x7fde8009aa30 legacy=0x7fde801c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 >> v1:172.21.15.63:6789/0 conn(0x7fde800a0da0 legacy=0x7fde801d1910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient: _finish_hunting 0 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 monclient: found mon.a 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:21.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde801344a0 con 0x7fde8009e230 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient: _finish_auth 0 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient: _check_auth_tickets 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 5 monclient: authenticate success, global_id 124224 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: _renew_subs 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fde80135ae0 con 0x7fde8009e230 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: _renew_subs 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fde80134260 con 0x7fde8009e230 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde70001cf0 con 0x7fde8009e230 2024-09-16T05:42:21.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde70003a70 con 0x7fde8009e230 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde7d7fa640 4 set_mon_vals no callback set 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.027+0000 7fde777fe640 10 monclient: dump: 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:21.047 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.031+0000 7fde854b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.031+0000 7fde854b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.031+0000 7fde854b2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.031+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fde70003c90 con 0x7fde8009e230 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.031+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde64005090 con 0x7fde8009e230 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.031+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fde70044f40 con 0x7fde8009e230 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.031+0000 7fde7effd640 10 monclient: get_auth_request con 0x7fde600369a0 auth_method 0 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.035+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fde700154f0 con 0x7fde8009e230 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.035+0000 7fde777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:21.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.035+0000 7fde777fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:21.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.259+0000 7fde854b2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.259+0000 7fde854b2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.259+0000 7fde854b2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.259+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde64005380 con 0x7fde8009e230 2024-09-16T05:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.259+0000 7fde777fe640 1 -- 172.21.15.6:0/3224076502 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3779427791) 0x7fde700191a0 con 0x7fde8009e230 2024-09-16T05:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.259+0000 7fde777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.259+0000 7fde777fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:21.265 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:42:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.263+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 >> v1:172.21.15.63:6812/3643207211 conn(0x7fde600369a0 legacy=0x7fde60038e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.263+0000 7fde854b2640 10 monclient: shutdown 2024-09-16T05:42:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.263+0000 7fde854b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.263+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 >> v1:172.21.15.6:6789/0 conn(0x7fde8009e230 legacy=0x7fde801c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.263+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 shutdown_connections 2024-09-16T05:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.263+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 >> 172.21.15.6:0/3224076502 conn(0x7fde800a1650 msgr2=0x7fde80071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.263+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 shutdown_connections 2024-09-16T05:42:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:21.263+0000 7fde854b2640 1 -- 172.21.15.6:0/3224076502 wait complete. 2024-09-16T05:42:24.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:42:24.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.411+0000 7ff95cae3640 10 monclient: get_monmap_and_config 2024-09-16T05:42:24.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.411+0000 7ff95cae3640 10 monclient: build_initial_monmap 2024-09-16T05:42:24.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.411+0000 7ff95cae3640 10 monclient: monmap: 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:24.417318+0000 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:24.417318+0000 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:24.417 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.411+0000 7ff95cae3640 1 Processor -- start 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.411+0000 7ff95cae3640 1 -- start start 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.411+0000 7ff95cae3640 10 monclient: init 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.411+0000 7ff95cae3640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.411+0000 7ff95cae3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95cae3640 10 monclient(hunting): picked mon.noname-c con 0x7ff9580a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95cae3640 10 monclient(hunting): picked mon.noname-a con 0x7ff9580a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95cae3640 10 monclient(hunting): picked mon.noname-b con 0x7ff9580a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95cae3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff95806e520 con 0x7ff9580a4290 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95cae3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9580a0780 con 0x7ff9580a37a0 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95cae3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9580a4c70 con 0x7ff9580a1a00 2024-09-16T05:42:24.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95cae3640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:24.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95cae3640 10 monclient(hunting): authenticate will time out at 1406.224219s 2024-09-16T05:42:24.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff95659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9580a3b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:24.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff955d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff9580a4290 0x7ff95813bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32964/0 (socket says 172.21.15.6:32964) 2024-09-16T05:42:24.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff955d9b640 1 -- 172.21.15.6:0/3703814495 learned_addr learned my addr 172.21.15.6:0/3703814495 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:24.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.415+0000 7ff956d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff9580a1a00 0x7ff95813f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39154/0 (socket says 172.21.15.6:39154) 2024-09-16T05:42:24.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:24.619+0000 7ff95659c640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9580a3b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:25.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:25.023+0000 7ff95659c640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9580a3b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:25.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:25.827+0000 7ff95659c640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9580a3b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 10 monclient(hunting): tick 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 1 monclient(hunting): continuing hunt 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.6:6790/0 conn(0x7ff9580a1a00 legacy=0x7ff95813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9580a3b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.6:6789/0 conn(0x7ff9580a4290 legacy=0x7ff95813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 10 monclient(hunting): picked mon.noname-a con 0x7ff9380036d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 10 monclient(hunting): picked mon.noname-c con 0x7ff9580a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 10 monclient(hunting): picked mon.noname-b con 0x7ff9580a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9580a65f0 con 0x7ff9380036d0 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff955d9b640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9580a0780 con 0x7ff9580a37a0 2024-09-16T05:42:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9380103b0 con 0x7ff9580a1a00 2024-09-16T05:42:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.415+0000 7ff947fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:27.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:27.619+0000 7ff955d9b640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:28.023+0000 7ff955d9b640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:28.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:28.827+0000 7ff955d9b640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:30.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 10 monclient(hunting): tick 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 1 monclient(hunting): continuing hunt 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.6:6790/0 conn(0x7ff9580a1a00 legacy=0x7ff93800cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.6:6789/0 conn(0x7ff9380036d0 legacy=0x7ff938005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 10 monclient(hunting): picked mon.noname-b con 0x7ff9380106c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 10 monclient(hunting): picked mon.noname-a con 0x7ff93800ab50 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:30.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 10 monclient(hunting): picked mon.noname-c con 0x7ff9580a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff956d9d640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff938011600 con 0x7ff93800ab50 2024-09-16T05:42:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9580a0780 con 0x7ff9580a37a0 2024-09-16T05:42:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 1 -- 172.21.15.6:0/3703814495 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff93800cfe0 con 0x7ff9380106c0 2024-09-16T05:42:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.419+0000 7ff947fff640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:30.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:30.623+0000 7ff956d9d640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:31.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.027+0000 7ff956d9d640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:31.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1009440793 0 0) 0x7ff93800cfe0 con 0x7ff9380106c0 2024-09-16T05:42:31.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient(hunting): _negotiate 2024-09-16T05:42:31.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient(hunting): my global_id is 6686 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff940003530 con 0x7ff9380106c0 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 327271705 0 0) 0x7ff940003530 con 0x7ff9380106c0 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.63:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9380093e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.6:6789/0 conn(0x7ff93800ab50 legacy=0x7ff9380118f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: _finish_hunting 0 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 monclient: found mon.noname-b 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff938005b30 con 0x7ff9380106c0 2024-09-16T05:42:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: _finish_auth 0 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: _check_auth_tickets 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff94c004160 con 0x7ff9380106c0 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: dump: 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 5 monclient: authenticate success, global_id 6686 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff94c004440 con 0x7ff9380106c0 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 1 -- 172.21.15.6:0/3703814495 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff94c004850 con 0x7ff9380106c0 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 4 set_mon_vals no callback set 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95559a640 10 monclient: dump: 2024-09-16T05:42:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:31.711 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 10 monclient: shutdown 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 1 -- 172.21.15.6:0/3703814495 >> v1:172.21.15.6:6790/0 conn(0x7ff9380106c0 legacy=0x7ff938010aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 1 -- 172.21.15.6:0/3703814495 shutdown_connections 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 1 -- 172.21.15.6:0/3703814495 >> 172.21.15.6:0/3703814495 conn(0x7ff9580a58a0 msgr2=0x7ff9580a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 1 -- 172.21.15.6:0/3703814495 shutdown_connections 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 1 -- 172.21.15.6:0/3703814495 wait complete. 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 10 monclient: build_initial_monmap 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.703+0000 7ff95cae3640 10 monclient: monmap: 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 1 Processor -- start 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 1 -- start start 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient: init 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient(hunting): picked mon.noname-b con 0x7ff93800ab50 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient(hunting): picked mon.noname-c con 0x7ff9380106c0 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient(hunting): picked mon.noname-a con 0x7ff9580a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9581265c0 con 0x7ff9580a37a0 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9581268a0 con 0x7ff93800ab50 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff958124d30 con 0x7ff9380106c0 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient(hunting): authenticate will time out at 1413.517488s 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95659c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7ff93800ab50 legacy=0x7ff9581cbaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff956d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff9580a37a0 0x7ff9581d4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46152/0 (socket says 172.21.15.6:46152) 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff956d9d640 1 -- 172.21.15.6:0/2720077468 learned_addr learned my addr 172.21.15.6:0/2720077468 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110082475 0 0) 0x7ff958124d30 con 0x7ff9380106c0 2024-09-16T05:42:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient(hunting): my global_id is 6689 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff938005d00 con 0x7ff9380106c0 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 153849526 0 0) 0x7ff938005d00 con 0x7ff9380106c0 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff958124d30 con 0x7ff9380106c0 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff9480017b0 con 0x7ff9380106c0 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient(hunting): dump: 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 337137774 0 0) 0x7ff958124d30 con 0x7ff9380106c0 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 >> v1:172.21.15.63:6789/0 conn(0x7ff93800ab50 legacy=0x7ff9581cbaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 >> v1:172.21.15.6:6789/0 conn(0x7ff9580a37a0 legacy=0x7ff9581d4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:42:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 monclient: found mon.c 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9581253f0 con 0x7ff9380106c0 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient: _finish_auth 0 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 5 monclient: authenticate success, global_id 6689 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient: _renew_subs 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff958137600 con 0x7ff9380106c0 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient: _renew_subs 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9581262e0 con 0x7ff9380106c0 2024-09-16T05:42:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9480047f0 con 0x7ff9380106c0 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff948004bc0 con 0x7ff9380106c0 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff954d99640 4 set_mon_vals no callback set 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.707+0000 7ff946ffd640 10 monclient: dump: 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.711+0000 7ff95cae3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.711+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff948004de0 con 0x7ff9380106c0 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.711+0000 7ff95cae3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.711+0000 7ff95cae3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.711+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff93c005090 con 0x7ff9380106c0 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.711+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7ff948044ea0 con 0x7ff9380106c0 2024-09-16T05:42:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.711+0000 7ff95659c640 10 monclient: get_auth_request con 0x7ff938037f60 auth_method 0 2024-09-16T05:42:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.715+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff9480154d0 con 0x7ff9380106c0 2024-09-16T05:42:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.715+0000 7ff946ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:31.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.715+0000 7ff946ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.955+0000 7ff95cae3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:42:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.955+0000 7ff95cae3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.955+0000 7ff95cae3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:31.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.955+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff93c005380 con 0x7ff9380106c0 2024-09-16T05:42:31.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.955+0000 7ff946ffd640 1 -- 172.21.15.6:0/2720077468 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 1882643005) 0x7ff948014dd0 con 0x7ff9380106c0 2024-09-16T05:42:31.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff946ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:42:31.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff946ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:31.962 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":38,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:42:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff938037f60 legacy=0x7ff93803a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff95cae3640 10 monclient: shutdown 2024-09-16T05:42:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff95cae3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 >> v1:172.21.15.6:6790/0 conn(0x7ff9380106c0 legacy=0x7ff9581d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 shutdown_connections 2024-09-16T05:42:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 >> 172.21.15.6:0/2720077468 conn(0x7ff9580a58a0 msgr2=0x7ff958136df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 shutdown_connections 2024-09-16T05:42:31.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:31.959+0000 7ff95cae3640 1 -- 172.21.15.6:0/2720077468 wait complete. 2024-09-16T05:42:31.980 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:42:31.981 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:42:31.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.091+0000 7f5f4eec6640 10 monclient: get_monmap_and_config 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.091+0000 7f5f4eec6640 10 monclient: build_initial_monmap 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient: monmap: 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:32.098951+0000 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:32.098951+0000 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.098 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:32.099 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:32.099 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:32.099 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 Processor -- start 2024-09-16T05:42:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- start start 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient: init 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient(hunting): picked mon.noname-c con 0x7f5f480a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f480a1370 con 0x7f5f4809aa30 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4809ead0 con 0x7f5f480a0da0 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4809edb0 con 0x7f5f4809e230 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient(hunting): authenticate will time out at 1413.905496s 2024-09-16T05:42:32.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f47fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5f4809e230 0x7f5f481399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:48512/0 (socket says 172.21.15.6:48512) 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f47fff640 1 -- 172.21.15.6:0/4167025639 learned_addr learned my addr 172.21.15.6:0/4167025639 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4d463640 1 -- 172.21.15.6:0/4167025639 >> v1:172.21.15.63:6789/0 conn(0x7f5f480a0da0 legacy=0x7f5f4813d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4cc62640 1 --1- 172.21.15.6:0/4167025639 >> v1:172.21.15.6:6789/0 conn(0x7f5f4809aa30 0x7f5f4809ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46154/0 (socket says 172.21.15.6:46154) 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264567733 0 0) 0x7f5f4809edb0 con 0x7f5f4809e230 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient(hunting): my global_id is 6692 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f30003530 con 0x7f5f4809e230 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666057649 0 0) 0x7f5f480a1370 con 0x7f5f4809aa30 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient(hunting): my global_id is 124227 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f4809edb0 con 0x7f5f4809aa30 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 105497120 0 0) 0x7f5f30003530 con 0x7f5f4809e230 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 >> v1:172.21.15.63:6789/0 conn(0x7f5f480a0da0 legacy=0x7f5f4813d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 >> v1:172.21.15.6:6789/0 conn(0x7f5f4809aa30 legacy=0x7f5f4809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: _finish_hunting 0 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 monclient: found mon.noname-b 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4809dd50 con 0x7f5f4809e230 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: _finish_auth 0 2024-09-16T05:42:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: _check_auth_tickets 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5f3c001550 con 0x7f5f4809e230 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: dump: 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.109 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 5 monclient: authenticate success, global_id 6692 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f3c001800 con 0x7f5f4809e230 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 1 -- 172.21.15.6:0/4167025639 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5f3c001bd0 con 0x7f5f4809e230 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f477fe640 10 monclient: dump: 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 4 set_mon_vals no callback set 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient: shutdown 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- 172.21.15.6:0/4167025639 >> v1:172.21.15.6:6790/0 conn(0x7f5f4809e230 legacy=0x7f5f481399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- 172.21.15.6:0/4167025639 shutdown_connections 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- 172.21.15.6:0/4167025639 >> 172.21.15.6:0/4167025639 conn(0x7f5f480a1650 msgr2=0x7f5f480a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- 172.21.15.6:0/4167025639 shutdown_connections 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 1 -- 172.21.15.6:0/4167025639 wait complete. 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient: build_initial_monmap 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.095+0000 7f5f4eec6640 10 monclient: monmap: 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:32.112 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:32.113 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 1 Processor -- start 2024-09-16T05:42:32.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 1 -- start start 2024-09-16T05:42:32.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient: init 2024-09-16T05:42:32.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:32.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:42:32.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient(hunting): picked mon.noname-c con 0x7f5f4809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient(hunting): picked mon.noname-b con 0x7f5f480a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4811c500 con 0x7f5f4809aa30 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f481ca4d0 con 0x7f5f480a0da0 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4806e760 con 0x7f5f4809e230 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient(hunting): authenticate will time out at 1413.908588s 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4cc62640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5f4809aa30 0x7f5f481c96b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46166/0 (socket says 172.21.15.6:46166) 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4cc62640 1 -- 172.21.15.6:0/991866995 learned_addr learned my addr 172.21.15.6:0/991866995 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4d463640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.63:6789/0 conn(0x7f5f480a0da0 legacy=0x7f5f481d1910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1351887042 0 0) 0x7f5f4806e760 con 0x7f5f4809e230 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): my global_id is 6695 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f34003530 con 0x7f5f4809e230 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401889867 0 0) 0x7f5f4811c500 con 0x7f5f4809aa30 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): my global_id is 124230 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f4806e760 con 0x7f5f4809aa30 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3338720928 0 0) 0x7f5f34003530 con 0x7f5f4809e230 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f4811c500 con 0x7f5f4809e230 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5f3c001510 con 0x7f5f4809e230 2024-09-16T05:42:32.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): dump: 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 695686166 0 0) 0x7f5f4806e760 con 0x7f5f4809aa30 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f34003530 con 0x7f5f4809aa30 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5f38001840 con 0x7f5f4809aa30 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): dump: 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.117 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2685972510 0 0) 0x7f5f4811c500 con 0x7f5f4809e230 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.63:6789/0 conn(0x7f5f480a0da0 legacy=0x7f5f481d1910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.6:6789/0 conn(0x7f5f4809aa30 legacy=0x7f5f481c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: _finish_hunting 0 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 monclient: found mon.c 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f480740c0 con 0x7f5f4809e230 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: _finish_auth 0 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: _check_auth_tickets 2024-09-16T05:42:32.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 5 monclient: authenticate success, global_id 6695 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient: _renew_subs 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4811c220 con 0x7f5f4809e230 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient: _renew_subs 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f4813d420 con 0x7f5f4809e230 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f480008e0 con 0x7f5f4809b420 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f3c001800 con 0x7f5f4809e230 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5f3c004aa0 con 0x7f5f4809e230 2024-09-16T05:42:32.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: dump: 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.120 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f46ffd640 4 set_mon_vals no callback set 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5f3c00f840 con 0x7f5f4809e230 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f5f3c044cf0 con 0x7f5f4809e230 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715211586 0 0) 0x7f5f480008e0 con 0x7f5f4809b420 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient(hunting): my global_id is 124233 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f4811c500 con 0x7f5f4809b420 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f4d463640 10 monclient: get_auth_request con 0x7f5f34036b10 auth_method 0 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1872357608 0 0) 0x7f5f4811c500 con 0x7f5f4809b420 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f480008e0 con 0x7f5f4809b420 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5f38004990 con 0x7f5f4809b420 2024-09-16T05:42:32.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: dump: 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1092613887 0 0) 0x7f5f480008e0 con 0x7f5f4809b420 2024-09-16T05:42:32.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f4813c370 con 0x7f5f4809b420 2024-09-16T05:42:32.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5f38005820 con 0x7f5f4809b420 2024-09-16T05:42:32.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:32.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.099+0000 7f5f457fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:32.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.103+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.6:6789/0 conn(0x7f5f4809b420 legacy=0x7f5f481369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.103+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.6:6789/0 conn(0x7f5f4809b420 legacy=0x7f5f481369c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:32.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f481377e0 con 0x7f5f4813c520 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3553248173 0 0) 0x7f5f481377e0 con 0x7f5f4813c520 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient(hunting): my global_id is 124236 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f480008e0 con 0x7f5f4813c520 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2437187407 0 0) 0x7f5f480008e0 con 0x7f5f4813c520 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f481377e0 con 0x7f5f4813c520 2024-09-16T05:42:32.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5f38004f80 con 0x7f5f4813c520 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient: dump: 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.126 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 691517250 0 0) 0x7f5f481377e0 con 0x7f5f4813c520 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f481da780 con 0x7f5f4813c520 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4119 (unknown 0 0 3667948332) 0x7f5f38005380 con 0x7f5f4813c520 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.6:6789/0 conn(0x7f5f4813c520 legacy=0x7f5f481370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f457fa640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.6:6789/0 conn(0x7f5f4813c520 legacy=0x7f5f481370d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:32.128 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:32.128 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:32.129 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:32.129 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:42:32.129 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 38, 2024-09-16T05:42:32.129 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 17799, 2024-09-16T05:42:32.129 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:42:32.129 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:32.130 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:32.131 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:42:32.132 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:42:32.133 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:42:32.133 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:42:32.133 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:42:32.134 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:42:32.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.136 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:32.137 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:32.138 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:32.139 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:42:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:42:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:42:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:42:32.143 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:32.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5f34036b10 legacy=0x7f5f34038fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 10 monclient: shutdown 2024-09-16T05:42:32.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:32.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 >> v1:172.21.15.6:6790/0 conn(0x7f5f4809e230 legacy=0x7f5f481c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4d463640 1 -- 172.21.15.6:0/991866995 reap_dead start 2024-09-16T05:42:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4d463640 1 -- 172.21.15.6:0/991866995 reap_dead start 2024-09-16T05:42:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 shutdown_connections 2024-09-16T05:42:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 >> 172.21.15.6:0/991866995 conn(0x7f5f480a1650 msgr2=0x7f5f4806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:32.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 shutdown_connections 2024-09-16T05:42:32.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.107+0000 7f5f4eec6640 1 -- 172.21.15.6:0/991866995 wait complete. 2024-09-16T05:42:32.147 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:42:32.147 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:42:32.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:42:32.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient: get_monmap_and_config 2024-09-16T05:42:32.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient: build_initial_monmap 2024-09-16T05:42:32.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient: monmap: 2024-09-16T05:42:32.266 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:32.266 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:42:32.267406+0000 2024-09-16T05:42:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:42:32.267406+0000 2024-09-16T05:42:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:42:32.267 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:42:32.267 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 1 Processor -- start 2024-09-16T05:42:32.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 1 -- start start 2024-09-16T05:42:32.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient: init 2024-09-16T05:42:32.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:32.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:42:32.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient(hunting): picked mon.noname-c con 0x7f61780a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:32.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient(hunting): picked mon.noname-a con 0x7f61780a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:32.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient(hunting): picked mon.noname-b con 0x7f61780a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:32.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f617806e520 con 0x7f61780a4290 2024-09-16T05:42:32.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61780a0780 con 0x7f61780a37a0 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61780a4c70 con 0x7f61780a1a00 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient(hunting): authenticate will time out at 1414.074494s 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617e54d640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f61780a37a0 legacy=0x7f61780a3b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617dd4c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61780a4290 0x7f617813bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46194/0 (socket says 172.21.15.6:46194) 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617ed4e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f61780a1a00 0x7f617813f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:48538/0 (socket says 172.21.15.6:48538) 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617dd4c640 1 -- 172.21.15.6:0/171494884 learned_addr learned my addr 172.21.15.6:0/171494884 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794359825 0 0) 0x7f61780a4c70 con 0x7f61780a1a00 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient(hunting): my global_id is 6698 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6164003530 con 0x7f61780a1a00 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1663764239 0 0) 0x7f617806e520 con 0x7f61780a4290 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient(hunting): my global_id is 124239 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61780a4c70 con 0x7f61780a4290 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3761083659 0 0) 0x7f61780a4c70 con 0x7f61780a4290 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 >> v1:172.21.15.6:6790/0 conn(0x7f61780a1a00 legacy=0x7f617813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 >> v1:172.21.15.63:6789/0 conn(0x7f61780a37a0 legacy=0x7f61780a3b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: _finish_hunting 0 2024-09-16T05:42:32.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 monclient: found mon.noname-a 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61780a65f0 con 0x7f61780a4290 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: _finish_auth 0 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: _check_auth_tickets 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6168001600 con 0x7f61780a4290 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 5 monclient: authenticate success, global_id 124239 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: dump: 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61680018e0 con 0x7f61780a4290 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient: get_monmap_and_config success 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 1 -- 172.21.15.6:0/171494884 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6168001cf0 con 0x7f61780a4290 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 4 set_mon_vals no callback set 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f617d54b640 10 monclient: dump: 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.280 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 10 monclient: shutdown 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.263+0000 7f61807b1640 1 -- 172.21.15.6:0/171494884 >> v1:172.21.15.6:6789/0 conn(0x7f61780a4290 legacy=0x7f617813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- 172.21.15.6:0/171494884 shutdown_connections 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- 172.21.15.6:0/171494884 >> 172.21.15.6:0/171494884 conn(0x7f61780a58a0 msgr2=0x7f61780a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- 172.21.15.6:0/171494884 shutdown_connections 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- 172.21.15.6:0/171494884 wait complete. 2024-09-16T05:42:32.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: build_initial_monmap 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: monmap: 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 Processor -- start 2024-09-16T05:42:32.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- start start 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: init 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: _reopen_session rank -1 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient(hunting): picked mon.noname-b con 0x7f61780a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient(hunting): picked mon.noname-a con 0x7f61780a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient(hunting): picked mon.noname-c con 0x7f61780a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6178133710 con 0x7f61780a37a0 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61780a7dd0 con 0x7f61780a1a00 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61780a0100 con 0x7f61780a4290 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient(hunting): _renew_subs 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient(hunting): authenticate will time out at 1414.077596s 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f617ed4e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f61780a4290 0x7f61781d3d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:48550/0 (socket says 172.21.15.6:48550) 2024-09-16T05:42:32.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f617dd4c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61780a37a0 0x7f61781d0610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46198/0 (socket says 172.21.15.6:46198) 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f617ed4e640 1 -- 172.21.15.6:0/1697333700 learned_addr learned my addr 172.21.15.6:0/1697333700 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f617e54d640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.63:6789/0 conn(0x7f61780a1a00 legacy=0x7f6178136f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035973609 0 0) 0x7f61780a0100 con 0x7f61780a4290 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): my global_id is 6701 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f615c003530 con 0x7f61780a4290 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2274948767 0 0) 0x7f6178133710 con 0x7f61780a37a0 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): my global_id is 124242 2024-09-16T05:42:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61780a0100 con 0x7f61780a37a0 2024-09-16T05:42:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2869737257 0 0) 0x7f615c003530 con 0x7f61780a4290 2024-09-16T05:42:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6178133710 con 0x7f61780a4290 2024-09-16T05:42:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6170001dc0 con 0x7f61780a4290 2024-09-16T05:42:32.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): dump: 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.286 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 369282510 0 0) 0x7f61780a0100 con 0x7f61780a37a0 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f615c003530 con 0x7f61780a37a0 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6168001500 con 0x7f61780a37a0 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): dump: 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.287 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1970339083 0 0) 0x7f6178133710 con 0x7f61780a4290 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.63:6789/0 conn(0x7f61780a1a00 legacy=0x7f6178136f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.6:6789/0 conn(0x7f61780a37a0 legacy=0x7f61781d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient: _finish_hunting 0 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 monclient: found mon.c 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6178135a00 con 0x7f61780a4290 2024-09-16T05:42:32.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient: _finish_auth 0 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient: _check_auth_tickets 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 5 monclient: authenticate success, global_id 6701 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: _renew_subs 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6178137620 con 0x7f61780a4290 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: _renew_subs 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f61781cbc80 con 0x7f61780a4290 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6170002020 con 0x7f61780a4290 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:42:32.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6170004fc0 con 0x7f61780a4290 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient: dump: 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.290 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f617cd4a640 4 set_mon_vals no callback set 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61780008e0 con 0x7f6178140b10 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119149781 0 0) 0x7f61780008e0 con 0x7f6178140b10 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 10 monclient(hunting): my global_id is 6704 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6178133710 con 0x7f6178140b10 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.267+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f617000fd60 con 0x7f61780a4290 2024-09-16T05:42:32.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f6170045270 con 0x7f61780a4290 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 564616913 0 0) 0x7f6178133710 con 0x7f6178140b10 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61780008e0 con 0x7f6178140b10 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6174003ef0 con 0x7f6178140b10 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 10 monclient: dump: 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.292 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3681513684 0 0) 0x7f61780008e0 con 0x7f6178140b10 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6178135d50 con 0x7f6178140b10 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f617dd4c640 10 monclient: get_auth_request con 0x7f615c036b20 auth_method 0 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6174006230 con 0x7f6178140b10 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:42:32.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.6:6790/0 conn(0x7f6178140b10 legacy=0x7f617809b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.271+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.6:6790/0 conn(0x7f6178140b10 legacy=0x7f617809b4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:32.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f61807b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:42:32.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f617809c2d0 con 0x7f61781cb290 2024-09-16T05:42:32.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590748686 0 0) 0x7f617809c2d0 con 0x7f61781cb290 2024-09-16T05:42:32.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 10 monclient(hunting): _negotiate 2024-09-16T05:42:32.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 10 monclient(hunting): my global_id is 6707 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61780008e0 con 0x7f61781cb290 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 889507638 0 0) 0x7f61780008e0 con 0x7f61781cb290 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f617809c2d0 con 0x7f61781cb290 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f61740045a0 con 0x7f61781cb290 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 10 monclient: dump: 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:42:32.299 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4673568 0 0) 0x7f617809c2d0 con 0x7f61781cb290 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.275+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61781cb670 con 0x7f61781cb290 2024-09-16T05:42:32.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4118 (unknown 0 0 2979837643) 0x7f6174005930 con 0x7f61781cb290 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f616effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f616effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.6:6790/0 conn(0x7f61781cb290 legacy=0x7f617809bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f616effd640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.6:6790/0 conn(0x7f61781cb290 legacy=0x7f617809bbc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:42:32.301 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 38, 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 481331, 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:32.302 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:32.303 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:42:32.304 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:42:32.305 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:42:32.306 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:42:32.307 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:42:32.307 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:42:32.307 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:42:32.307 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:42:32.307 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.307 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:42:32.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:42:32.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.308 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:42:32.308 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:42:32.308 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:32.308 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:32.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.308 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:32.309 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:42:32.311 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:42:32.312 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:42:32.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:42:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:42:32.315 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:42:32.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.63:6812/3643207211 conn(0x7f615c036b20 legacy=0x7f615c038fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f61807b1640 10 monclient: shutdown 2024-09-16T05:42:32.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f61807b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:42:32.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 >> v1:172.21.15.6:6790/0 conn(0x7f61780a4290 legacy=0x7f61781d3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:42:32.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f617ed4e640 1 -- 172.21.15.6:0/1697333700 reap_dead start 2024-09-16T05:42:32.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f617ed4e640 1 -- 172.21.15.6:0/1697333700 reap_dead start 2024-09-16T05:42:32.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 shutdown_connections 2024-09-16T05:42:32.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 >> 172.21.15.6:0/1697333700 conn(0x7f61780a58a0 msgr2=0x7f61780a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:42:32.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 shutdown_connections 2024-09-16T05:42:32.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:42:32.279+0000 7f61807b1640 1 -- 172.21.15.6:0/1697333700 wait complete. 2024-09-16T05:42:32.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:42:32.320 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:44:02.320 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:44:02.321 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:44:02.321 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:44:02.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:44:02.325 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:44:02.325 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:44:02.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:44:02.426 INFO:tasks.ceph.mon.b.smithi063.stderr:2024-09-16T05:44:02.423+0000 7f6fa5beee00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 346 -> 0 (sync_force? mon rebuild?) 2024-09-16T05:44:02.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient: get_monmap_and_config 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient: build_initial_monmap 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient: monmap: 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:02.459991+0000 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:02.459991+0000 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:02.459 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:02.460 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:02.460 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 1 Processor -- start 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 1 -- start start 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient: init 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient(hunting): picked mon.noname-b con 0x7f11d80a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient(hunting): picked mon.noname-c con 0x7f11d80a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient(hunting): picked mon.noname-a con 0x7f11d80a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d80a33a0 con 0x7f11d80a2980 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d80a07a0 con 0x7f11d80a2570 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d80a2d90 con 0x7f11d80a2190 2024-09-16T05:44:02.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 10 monclient(hunting): authenticate will time out at 1504.267028s 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11de198640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f11d80a2980 0x7f11d813f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43470/0 (socket says 172.21.15.6:43470) 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11de198640 1 -- 172.21.15.6:0/2917687148 learned_addr learned my addr 172.21.15.6:0/2917687148 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1261500238 0 0) 0x7f11d80a33a0 con 0x7f11d80a2980 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient(hunting): my global_id is 124245 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11c4003530 con 0x7f11d80a2980 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1312948190 0 0) 0x7f11d80a2d90 con 0x7f11d80a2190 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient(hunting): my global_id is 6710 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11d80a33a0 con 0x7f11d80a2190 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4136355959 0 0) 0x7f11d80a33a0 con 0x7f11d80a2190 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 >> v1:172.21.15.63:6789/0 conn(0x7f11d80a2570 legacy=0x7f11d813b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 >> v1:172.21.15.6:6789/0 conn(0x7f11d80a2980 legacy=0x7f11d813f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient: _finish_hunting 0 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 monclient: found mon.noname-b 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d80a66e0 con 0x7f11d80a2190 2024-09-16T05:44:02.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient: _finish_auth 0 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient: _check_auth_tickets 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11d4001600 con 0x7f11d80a2190 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 5 monclient: authenticate success, global_id 6710 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dfbfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.458+0000 7f11dc995640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dc995640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dc995640 10 monclient: dump: 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.466 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11d4001870 con 0x7f11d80a2190 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dc995640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 4 set_mon_vals no callback set 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dc995640 1 -- 172.21.15.6:0/2917687148 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11d4001c80 con 0x7f11d80a2190 2024-09-16T05:44:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dc995640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dc995640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dc995640 10 monclient: dump: 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: shutdown 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- 172.21.15.6:0/2917687148 >> v1:172.21.15.6:6790/0 conn(0x7f11d80a2190 legacy=0x7f11d80a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- 172.21.15.6:0/2917687148 shutdown_connections 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- 172.21.15.6:0/2917687148 >> 172.21.15.6:0/2917687148 conn(0x7f11d80a9850 msgr2=0x7f11d80a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- 172.21.15.6:0/2917687148 shutdown_connections 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- 172.21.15.6:0/2917687148 wait complete. 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: build_initial_monmap 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: monmap: 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:02.469 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:02.470 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.470 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:02.470 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:02.470 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:02.470 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 Processor -- start 2024-09-16T05:44:02.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- start start 2024-09-16T05:44:02.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: init 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient(hunting): picked mon.noname-a con 0x7f11d80a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient(hunting): picked mon.noname-b con 0x7f11d80a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient(hunting): picked mon.noname-c con 0x7f11d80a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d812d640 con 0x7f11d80a2190 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d81215a0 con 0x7f11d80a2570 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d8122c20 con 0x7f11d80a2980 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient(hunting): authenticate will time out at 1504.270468s 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dd997640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f11d80a2190 0x7f11d8120780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43474/0 (socket says 172.21.15.6:43474) 2024-09-16T05:44:02.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dd997640 1 -- 172.21.15.6:0/466409945 learned_addr learned my addr 172.21.15.6:0/466409945 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051105809 0 0) 0x7f11d80a66e0 con 0x7f11d80a2190 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): my global_id is 124248 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11bc003530 con 0x7f11d80a2190 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2306403012 0 0) 0x7f11d8122c20 con 0x7f11d80a2980 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): my global_id is 6713 2024-09-16T05:44:02.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11d80a66e0 con 0x7f11d80a2980 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2070136422 0 0) 0x7f11bc003530 con 0x7f11d80a2190 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11d8122c20 con 0x7f11d80a2190 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11d4001cc0 con 0x7f11d80a2190 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): dump: 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 884448800 0 0) 0x7f11d80a66e0 con 0x7f11d80a2980 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11bc003530 con 0x7f11d80a2980 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11d0001c20 con 0x7f11d80a2980 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient(hunting): dump: 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.478 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1655890475 0 0) 0x7f11d8122c20 con 0x7f11d80a2190 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 >> v1:172.21.15.6:6790/0 conn(0x7f11d80a2980 legacy=0x7f11d81d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 >> v1:172.21.15.63:6789/0 conn(0x7f11d80a2570 legacy=0x7f11d8120e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 monclient: found mon.a 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d81240d0 con 0x7f11d80a2190 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient: _finish_auth 0 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 5 monclient: authenticate success, global_id 124248 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: _renew_subs 2024-09-16T05:44:02.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f11d8123df0 con 0x7f11d80a2190 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: _renew_subs 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f11d81400f0 con 0x7f11d80a2190 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11d4003b40 con 0x7f11d80a2190 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11d4003f10 con 0x7f11d80a2190 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11cb7fe640 4 set_mon_vals no callback set 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.462+0000 7f11ca7fc640 10 monclient: dump: 2024-09-16T05:44:02.481 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.466+0000 7f11dfbfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.466+0000 7f11dfbfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.466+0000 7f11dfbfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.466+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c0005090 con 0x7f11d80a2190 2024-09-16T05:44:02.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.466+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f11d40040b0 con 0x7f11d80a2190 2024-09-16T05:44:02.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.466+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f11d4044ea0 con 0x7f11d80a2190 2024-09-16T05:44:02.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.466+0000 7f11dd196640 10 monclient: get_auth_request con 0x7f11bc036ac0 auth_method 0 2024-09-16T05:44:02.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.470+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f11d812d640 con 0x7f11d80a2190 2024-09-16T05:44:02.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.470+0000 7f11ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:02.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.470+0000 7f11ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:02.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.702+0000 7f11dfbfb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:44:02.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.702+0000 7f11dfbfb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:02.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.702+0000 7f11dfbfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.702+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f11c0005380 con 0x7f11d80a2190 2024-09-16T05:44:02.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11ca7fc640 1 -- 172.21.15.6:0/466409945 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 4240254188) 0x7f11d4015530 con 0x7f11d80a2190 2024-09-16T05:44:02.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:02.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:02.706 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:44:02.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 >> v1:172.21.15.63:6812/3643207211 conn(0x7f11bc036ac0 legacy=0x7f11bc038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11dfbfb640 10 monclient: shutdown 2024-09-16T05:44:02.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11dfbfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:02.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 >> v1:172.21.15.6:6789/0 conn(0x7f11d80a2190 legacy=0x7f11d8120780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 shutdown_connections 2024-09-16T05:44:02.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 >> 172.21.15.6:0/466409945 conn(0x7f11d80a9850 msgr2=0x7f11d806ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:02.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 shutdown_connections 2024-09-16T05:44:02.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.706+0000 7f11dfbfb640 1 -- 172.21.15.6:0/466409945 wait complete. 2024-09-16T05:44:02.724 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:44:02.724 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:44:02.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:44:02.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient: get_monmap_and_config 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient: build_initial_monmap 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient: monmap: 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:02.835776+0000 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:02.835776+0000 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:02.836 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 1 Processor -- start 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 1 -- start start 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient: init 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient(hunting): picked mon.noname-a con 0x7fb26409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient(hunting): picked mon.noname-c con 0x7fb26409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient(hunting): picked mon.noname-b con 0x7fb2640a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2640a1370 con 0x7fb26409aa30 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26409ead0 con 0x7fb26409e230 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26409edb0 con 0x7fb2640a0da0 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient(hunting): authenticate will time out at 1504.642303s 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2688ed640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb26409aa30 0x7fb26409ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43482/0 (socket says 172.21.15.6:43482) 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2688ed640 1 -- 172.21.15.6:0/966532193 learned_addr learned my addr 172.21.15.6:0/966532193 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221775464 0 0) 0x7fb2640a1370 con 0x7fb26409aa30 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient(hunting): my global_id is 124251 2024-09-16T05:44:02.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb250003530 con 0x7fb26409aa30 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2752189184 0 0) 0x7fb26409edb0 con 0x7fb2640a0da0 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient(hunting): my global_id is 6716 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2640a1370 con 0x7fb2640a0da0 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 746902119 0 0) 0x7fb250003530 con 0x7fb26409aa30 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 >> v1:172.21.15.6:6790/0 conn(0x7fb2640a0da0 legacy=0x7fb26413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 >> v1:172.21.15.63:6789/0 conn(0x7fb26409e230 legacy=0x7fb2641399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: _finish_hunting 0 2024-09-16T05:44:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 monclient: found mon.noname-a 2024-09-16T05:44:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb26409dd50 con 0x7fb26409aa30 2024-09-16T05:44:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: _finish_auth 0 2024-09-16T05:44:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: _check_auth_tickets 2024-09-16T05:44:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb254001620 con 0x7fb26409aa30 2024-09-16T05:44:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: dump: 2024-09-16T05:44:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.852 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 5 monclient: authenticate success, global_id 124251 2024-09-16T05:44:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb254001880 con 0x7fb26409aa30 2024-09-16T05:44:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 1 -- 172.21.15.6:0/966532193 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb254001bd0 con 0x7fb26409aa30 2024-09-16T05:44:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 4 set_mon_vals no callback set 2024-09-16T05:44:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb2637fe640 10 monclient: dump: 2024-09-16T05:44:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.855 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 10 monclient: shutdown 2024-09-16T05:44:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 1 -- 172.21.15.6:0/966532193 >> v1:172.21.15.6:6789/0 conn(0x7fb26409aa30 legacy=0x7fb26409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 1 -- 172.21.15.6:0/966532193 shutdown_connections 2024-09-16T05:44:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.834+0000 7fb26ab51640 1 -- 172.21.15.6:0/966532193 >> 172.21.15.6:0/966532193 conn(0x7fb2640a1650 msgr2=0x7fb2640a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 -- 172.21.15.6:0/966532193 shutdown_connections 2024-09-16T05:44:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 -- 172.21.15.6:0/966532193 wait complete. 2024-09-16T05:44:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: build_initial_monmap 2024-09-16T05:44:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: monmap: 2024-09-16T05:44:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:02.857 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:02.858 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:02.858 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 Processor -- start 2024-09-16T05:44:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 -- start start 2024-09-16T05:44:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: init 2024-09-16T05:44:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient(hunting): picked mon.noname-a con 0x7fb26409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient(hunting): picked mon.noname-b con 0x7fb26409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient(hunting): picked mon.noname-c con 0x7fb2640a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26411c530 con 0x7fb26409aa30 2024-09-16T05:44:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26413da30 con 0x7fb26409e230 2024-09-16T05:44:02.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26406e6d0 con 0x7fb2640a0da0 2024-09-16T05:44:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient(hunting): authenticate will time out at 1504.645526s 2024-09-16T05:44:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2688ed640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb26409aa30 0x7fb26413b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43494/0 (socket says 172.21.15.6:43494) 2024-09-16T05:44:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2688ed640 1 -- 172.21.15.6:0/3587799598 learned_addr learned my addr 172.21.15.6:0/3587799598 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:02.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366271920 0 0) 0x7fb26411c530 con 0x7fb26409aa30 2024-09-16T05:44:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): my global_id is 124254 2024-09-16T05:44:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb244003530 con 0x7fb26409aa30 2024-09-16T05:44:02.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541657187 0 0) 0x7fb26406e6d0 con 0x7fb2640a0da0 2024-09-16T05:44:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): my global_id is 6719 2024-09-16T05:44:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb26411c530 con 0x7fb2640a0da0 2024-09-16T05:44:02.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 818004624 0 0) 0x7fb244003530 con 0x7fb26409aa30 2024-09-16T05:44:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb26406e6d0 con 0x7fb26409aa30 2024-09-16T05:44:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb254001520 con 0x7fb26409aa30 2024-09-16T05:44:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:02.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): dump: 2024-09-16T05:44:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.866 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 501792336 0 0) 0x7fb26411c530 con 0x7fb2640a0da0 2024-09-16T05:44:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb244003530 con 0x7fb2640a0da0 2024-09-16T05:44:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb25c001c20 con 0x7fb2640a0da0 2024-09-16T05:44:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient(hunting): dump: 2024-09-16T05:44:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.869 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 280802173 0 0) 0x7fb26406e6d0 con 0x7fb26409aa30 2024-09-16T05:44:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 >> v1:172.21.15.6:6790/0 conn(0x7fb2640a0da0 legacy=0x7fb2641d1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 >> v1:172.21.15.63:6789/0 conn(0x7fb26409e230 legacy=0x7fb26413ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient: _finish_hunting 0 2024-09-16T05:44:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 monclient: found mon.a 2024-09-16T05:44:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2640740c0 con 0x7fb26409aa30 2024-09-16T05:44:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient: _finish_auth 0 2024-09-16T05:44:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient: _check_auth_tickets 2024-09-16T05:44:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 5 monclient: authenticate success, global_id 124254 2024-09-16T05:44:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: _renew_subs 2024-09-16T05:44:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb26411b880 con 0x7fb26409aa30 2024-09-16T05:44:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: _renew_subs 2024-09-16T05:44:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb26411b1a0 con 0x7fb26409aa30 2024-09-16T05:44:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb254001880 con 0x7fb26409aa30 2024-09-16T05:44:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb254004c00 con 0x7fb26409aa30 2024-09-16T05:44:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb262ffd640 4 set_mon_vals no callback set 2024-09-16T05:44:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.838+0000 7fb2617fa640 10 monclient: dump: 2024-09-16T05:44:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.875 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb26ab51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb25400f9a0 con 0x7fb26409aa30 2024-09-16T05:44:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fb254044d70 con 0x7fb26409aa30 2024-09-16T05:44:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb2480065d0 con 0x7fb248002870 2024-09-16T05:44:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb263fff640 10 monclient: get_auth_request con 0x7fb244036a80 auth_method 0 2024-09-16T05:44:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736185376 0 0) 0x7fb2480065d0 con 0x7fb248002870 2024-09-16T05:44:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient(hunting): my global_id is 124257 2024-09-16T05:44:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb26406e6d0 con 0x7fb248002870 2024-09-16T05:44:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1235613896 0 0) 0x7fb26406e6d0 con 0x7fb248002870 2024-09-16T05:44:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2480065d0 con 0x7fb248002870 2024-09-16T05:44:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb25c003d90 con 0x7fb248002870 2024-09-16T05:44:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient: dump: 2024-09-16T05:44:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.879 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.879 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.879 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.879 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.879 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.880 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.880 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1330898282 0 0) 0x7fb2480065d0 con 0x7fb248002870 2024-09-16T05:44:02.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb24800a160 con 0x7fb248002870 2024-09-16T05:44:02.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb25c0061d0 con 0x7fb248002870 2024-09-16T05:44:02.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:02.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:02.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 >> v1:172.21.15.6:6789/0 conn(0x7fb248002870 legacy=0x7fb248004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.842+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 >> v1:172.21.15.6:6789/0 conn(0x7fb248002870 legacy=0x7fb248004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:02.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.846+0000 7fb26ab51640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:44:02.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.846+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb248009e30 con 0x7fb2480090f0 2024-09-16T05:44:02.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.846+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549789213 0 0) 0x7fb248009e30 con 0x7fb2480090f0 2024-09-16T05:44:02.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.846+0000 7fb2617fa640 10 monclient(hunting): _negotiate 2024-09-16T05:44:02.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.846+0000 7fb2617fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:02.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.846+0000 7fb2617fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:02.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.846+0000 7fb2617fa640 10 monclient(hunting): my global_id is 124260 2024-09-16T05:44:02.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.846+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2480065d0 con 0x7fb2480090f0 2024-09-16T05:44:02.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3107960779 0 0) 0x7fb2480065d0 con 0x7fb2480090f0 2024-09-16T05:44:02.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb248009e30 con 0x7fb2480090f0 2024-09-16T05:44:02.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb25c004420 con 0x7fb2480090f0 2024-09-16T05:44:02.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:02.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:02.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 10 monclient: dump: 2024-09-16T05:44:02.887 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:02.887 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:02.887 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.887 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:02.887 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:02.888 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:02.888 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:02.888 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:02.888 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:02.889 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:02.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3167854813 0 0) 0x7fb248009e30 con 0x7fb2480090f0 2024-09-16T05:44:02.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb24800c3a0 con 0x7fb2480090f0 2024-09-16T05:44:02.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4131 (unknown 0 0 3627443012) 0x7fb25c005780 con 0x7fb2480090f0 2024-09-16T05:44:02.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:44:02.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 >> v1:172.21.15.6:6789/0 conn(0x7fb2480090f0 legacy=0x7fb248009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2617fa640 1 -- 172.21.15.6:0/3587799598 >> v1:172.21.15.6:6789/0 conn(0x7fb2480090f0 legacy=0x7fb248009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:02.891 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:02.891 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:02.891 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:02.891 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:44:02.892 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 40, 2024-09-16T05:44:02.892 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 108543, 2024-09-16T05:44:02.892 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:44:02.892 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:44:02.892 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:44:02.893 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:44:02.893 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:02.893 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:44:02.893 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:02.893 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:44:02.893 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:44:02.894 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:02.894 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:02.894 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:02.894 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:02.894 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:02.895 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:02.895 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:02.895 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:02.895 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:02.895 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:02.896 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:02.896 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:02.896 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:44:02.896 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:44:02.896 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:02.896 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:02.897 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:02.897 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:02.897 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:02.897 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:02.898 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:02.898 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:02.898 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:02.898 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:02.898 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:02.898 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:02.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.899 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:44:02.899 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:44:02.899 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:44:02.899 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:44:02.899 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:44:02.900 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:44:02.900 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:02.900 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:02.900 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:44:02.900 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:44:02.900 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:44:02.901 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:44:02.901 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:44:02.901 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:44:02.901 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:44:02.901 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:02.901 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:44:02.902 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:02.902 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:02.902 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:02.902 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:02.902 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:02.902 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:02.903 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:02.903 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:02.903 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:02.903 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:02.903 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:02.903 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:02.904 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:44:02.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:02.905 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:02.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:44:02.908 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:02.909 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:02.910 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:02.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:02.911 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:44:02.911 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb244036a80 legacy=0x7fb244038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb26ab51640 10 monclient: shutdown 2024-09-16T05:44:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb26ab51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 >> v1:172.21.15.6:6789/0 conn(0x7fb26409aa30 legacy=0x7fb26413b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2690ee640 1 -- 172.21.15.6:0/3587799598 reap_dead start 2024-09-16T05:44:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb2690ee640 1 -- 172.21.15.6:0/3587799598 reap_dead start 2024-09-16T05:44:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 shutdown_connections 2024-09-16T05:44:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 >> 172.21.15.6:0/3587799598 conn(0x7fb2640a1650 msgr2=0x7fb26406ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:02.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 shutdown_connections 2024-09-16T05:44:02.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:02.850+0000 7fb26ab51640 1 -- 172.21.15.6:0/3587799598 wait complete. 2024-09-16T05:44:02.913 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:44:02.913 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:44:02.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:44:03.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient: get_monmap_and_config 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient: build_initial_monmap 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient: monmap: 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:03.025009+0000 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:03.025009+0000 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:03.025 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:03.025 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 1 Processor -- start 2024-09-16T05:44:03.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 1 -- start start 2024-09-16T05:44:03.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient: init 2024-09-16T05:44:03.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:03.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:44:03.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient(hunting): picked mon.noname-b con 0x7f84140a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient(hunting): picked mon.noname-a con 0x7f84140a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient(hunting): picked mon.noname-c con 0x7f84140a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84140a33a0 con 0x7f84140a2570 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84140a07a0 con 0x7f84140a2980 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84140a2d90 con 0x7f84140a2190 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f841a6ed640 10 monclient(hunting): authenticate will time out at 1504.831573s 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f8413fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f84140a2190 0x7f84140a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49954/0 (socket says 172.21.15.6:49954) 2024-09-16T05:44:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f84137fe640 1 --1- 172.21.15.6:0/1557982590 >> v1:172.21.15.6:6789/0 conn(0x7f84140a2570 0x7f841413b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43524/0 (socket says 172.21.15.6:43524) 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f8413fff640 1 -- 172.21.15.6:0/1557982590 learned_addr learned my addr 172.21.15.6:0/1557982590 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1983033311 0 0) 0x7f84140a33a0 con 0x7f84140a2570 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f8412ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f8412ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f8412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.022+0000 7f8412ffd640 10 monclient(hunting): my global_id is 124263 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83fc003530 con 0x7f84140a2570 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021806175 0 0) 0x7f84140a2d90 con 0x7f84140a2190 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient(hunting): my global_id is 6722 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84140a33a0 con 0x7f84140a2190 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3722633403 0 0) 0x7f83fc003530 con 0x7f84140a2570 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 >> v1:172.21.15.6:6790/0 conn(0x7f84140a2190 legacy=0x7f84140a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 >> v1:172.21.15.63:6789/0 conn(0x7f84140a2980 legacy=0x7f841413f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 monclient: found mon.noname-a 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84140a66e0 con 0x7f84140a2570 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: _finish_auth 0 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f840c0018d0 con 0x7f84140a2570 2024-09-16T05:44:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: dump: 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.035 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 5 monclient: authenticate success, global_id 124263 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f840c001bb0 con 0x7f84140a2570 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 1 -- 172.21.15.6:0/1557982590 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f840c001fc0 con 0x7f84140a2570 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 4 set_mon_vals no callback set 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8412ffd640 10 monclient: dump: 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: shutdown 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- 172.21.15.6:0/1557982590 >> v1:172.21.15.6:6789/0 conn(0x7f84140a2570 legacy=0x7f841413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- 172.21.15.6:0/1557982590 shutdown_connections 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- 172.21.15.6:0/1557982590 >> 172.21.15.6:0/1557982590 conn(0x7f84140a9850 msgr2=0x7f84140a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- 172.21.15.6:0/1557982590 shutdown_connections 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- 172.21.15.6:0/1557982590 wait complete. 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: build_initial_monmap 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: monmap: 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:03.038 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 Processor -- start 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- start start 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: init 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient(hunting): picked mon.noname-c con 0x7f84140a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient(hunting): picked mon.noname-b con 0x7f84140a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient(hunting): picked mon.noname-a con 0x7f84140a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8414124110 con 0x7f84140a2980 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84141213d0 con 0x7f84140a2570 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8414123120 con 0x7f84140a2190 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient(hunting): authenticate will time out at 1504.834534s 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8413fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f84140a2190 0x7f84141205b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49958/0 (socket says 172.21.15.6:49958) 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8413fff640 1 -- 172.21.15.6:0/2867210935 learned_addr learned my addr 172.21.15.6:0/2867210935 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1059750619 0 0) 0x7f8414123120 con 0x7f84140a2190 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): my global_id is 6725 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8400003530 con 0x7f84140a2190 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737756500 0 0) 0x7f8414124110 con 0x7f84140a2980 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): my global_id is 124266 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8414123120 con 0x7f84140a2980 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1817092026 0 0) 0x7f8400003530 con 0x7f84140a2190 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8414124110 con 0x7f84140a2190 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8404001a40 con 0x7f84140a2190 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): dump: 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 879020203 0 0) 0x7f8414123120 con 0x7f84140a2980 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8400003530 con 0x7f84140a2980 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f84080017c0 con 0x7f84140a2980 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): dump: 2024-09-16T05:44:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299787400 0 0) 0x7f84141213d0 con 0x7f84140a2570 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient(hunting): my global_id is 134101 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8414123120 con 0x7f84140a2570 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1861939010 0 0) 0x7f8414124110 con 0x7f84140a2190 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 >> v1:172.21.15.63:6789/0 conn(0x7f84140a2570 legacy=0x7f8414120cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 >> v1:172.21.15.6:6789/0 conn(0x7f84140a2980 legacy=0x7f84141d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 monclient: found mon.c 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84141225a0 con 0x7f84140a2190 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient: _finish_auth 0 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:44:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 5 monclient: authenticate success, global_id 6725 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: _renew_subs 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f84140a8000 con 0x7f84140a2190 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: _renew_subs 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f84141243f0 con 0x7f84140a2190 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8404001be0 con 0x7f84140a2190 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8404004c40 con 0x7f84140a2190 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f84127fc640 4 set_mon_vals no callback set 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:03.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.026+0000 7f8410ff9640 10 monclient: dump: 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f841a6ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83f40065d0 con 0x7f83f4002870 2024-09-16T05:44:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f840400f9e0 con 0x7f84140a2190 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f8404044e10 con 0x7f84140a2190 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288063409 0 0) 0x7f83f40065d0 con 0x7f83f4002870 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient(hunting): my global_id is 6728 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8414124110 con 0x7f83f4002870 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8418c8a640 10 monclient: get_auth_request con 0x7f8400036be0 auth_method 0 2024-09-16T05:44:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1059240886 0 0) 0x7f8414124110 con 0x7f83f4002870 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83f40065d0 con 0x7f83f4002870 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f840c001bb0 con 0x7f83f4002870 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient: dump: 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.048 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.049 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3149901361 0 0) 0x7f83f40065d0 con 0x7f83f4002870 2024-09-16T05:44:03.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83f400a160 con 0x7f83f4002870 2024-09-16T05:44:03.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f840c001880 con 0x7f83f4002870 2024-09-16T05:44:03.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:03.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:03.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 >> v1:172.21.15.6:6790/0 conn(0x7f83f4002870 legacy=0x7f83f4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.030+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 >> v1:172.21.15.6:6790/0 conn(0x7f83f4002870 legacy=0x7f83f4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f841a6ed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:44:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83f4009e50 con 0x7f83f4009110 2024-09-16T05:44:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217015422 0 0) 0x7f83f4009e50 con 0x7f83f4009110 2024-09-16T05:44:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 10 monclient(hunting): my global_id is 6731 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83f40065d0 con 0x7f83f4009110 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2882897677 0 0) 0x7f83f40065d0 con 0x7f83f4009110 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83f4009e50 con 0x7f83f4009110 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f840c005000 con 0x7f83f4009110 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.034+0000 7f8410ff9640 10 monclient: dump: 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 952349701 0 0) 0x7f83f4009e50 con 0x7f83f4009110 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f83f400c3e0 con 0x7f83f4009110 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 1128316506) 0x7f840c005230 con 0x7f83f4009110 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8410ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8410ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 >> v1:172.21.15.6:6790/0 conn(0x7f83f4009110 legacy=0x7f83f40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8410ff9640 1 -- 172.21.15.6:0/2867210935 >> v1:172.21.15.6:6790/0 conn(0x7f83f4009110 legacy=0x7f83f40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 40, 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 572095, 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:44:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:44:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:44:03.065 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:03.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8400036be0 legacy=0x7f84000390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f841a6ed640 10 monclient: shutdown 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f841a6ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 >> v1:172.21.15.6:6790/0 conn(0x7f84140a2190 legacy=0x7f84141205b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8418c8a640 1 -- 172.21.15.6:0/2867210935 reap_dead start 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f8418c8a640 1 -- 172.21.15.6:0/2867210935 reap_dead start 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 shutdown_connections 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 >> 172.21.15.6:0/2867210935 conn(0x7f84140a9850 msgr2=0x7f841412f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 shutdown_connections 2024-09-16T05:44:03.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.038+0000 7f841a6ed640 1 -- 172.21.15.6:0/2867210935 wait complete. 2024-09-16T05:44:03.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:44:03.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:44:03.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient: get_monmap_and_config 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient: build_initial_monmap 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient: monmap: 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:03.184035+0000 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:03.184035+0000 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:03.184 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 1 Processor -- start 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 1 -- start start 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient: init 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient(hunting): picked mon.noname-b con 0x7f53bc09de60 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient(hunting): picked mon.noname-a con 0x7f53bc09e240 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient(hunting): picked mon.noname-c con 0x7f53bc139460 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53bc09d2f0 con 0x7f53bc09e240 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53bc13df80 con 0x7f53bc139460 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53bc13f160 con 0x7f53bc09de60 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 10 monclient(hunting): authenticate will time out at 1504.991205s 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53bbfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f53bc09de60 0x7f53bc0a0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49974/0 (socket says 172.21.15.6:49974) 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53bbfff640 1 -- 172.21.15.6:0/2075452479 learned_addr learned my addr 172.21.15.6:0/2075452479 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211850830 0 0) 0x7f53bc13f160 con 0x7f53bc09de60 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient(hunting): my global_id is 6734 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53a4003530 con 0x7f53bc09de60 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127774565 0 0) 0x7f53bc09d2f0 con 0x7f53bc09e240 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient(hunting): my global_id is 124269 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53bc13f160 con 0x7f53bc09e240 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 622249201 0 0) 0x7f53a4003530 con 0x7f53bc09de60 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 >> v1:172.21.15.63:6789/0 conn(0x7f53bc139460 legacy=0x7f53bc13b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 >> v1:172.21.15.6:6789/0 conn(0x7f53bc09e240 legacy=0x7f53bc0a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient: _finish_hunting 0 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 monclient: found mon.noname-b 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53bc140340 con 0x7f53bc09de60 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient: _finish_auth 0 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient: _check_auth_tickets 2024-09-16T05:44:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53ac001620 con 0x7f53bc09de60 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53baffd640 10 monclient: dump: 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.191 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 5 monclient: authenticate success, global_id 6734 2024-09-16T05:44:03.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.182+0000 7f53c2a1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:03.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53ac001890 con 0x7f53bc09de60 2024-09-16T05:44:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53baffd640 1 -- 172.21.15.6:0/2075452479 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53ac001c60 con 0x7f53bc09de60 2024-09-16T05:44:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53baffd640 10 monclient: dump: 2024-09-16T05:44:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 4 set_mon_vals no callback set 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: shutdown 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- 172.21.15.6:0/2075452479 >> v1:172.21.15.6:6790/0 conn(0x7f53bc09de60 legacy=0x7f53bc0a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- 172.21.15.6:0/2075452479 shutdown_connections 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- 172.21.15.6:0/2075452479 >> 172.21.15.6:0/2075452479 conn(0x7f53bc0a1650 msgr2=0x7f53bc0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- 172.21.15.6:0/2075452479 shutdown_connections 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- 172.21.15.6:0/2075452479 wait complete. 2024-09-16T05:44:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: build_initial_monmap 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: monmap: 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 Processor -- start 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- start start 2024-09-16T05:44:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: init 2024-09-16T05:44:03.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient(hunting): picked mon.noname-c con 0x7f53bc09de60 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient(hunting): picked mon.noname-a con 0x7f53bc09e240 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient(hunting): picked mon.noname-b con 0x7f53bc139460 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53bc120ae0 con 0x7f53bc09e240 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53bc120dc0 con 0x7f53bc139460 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53bc09f550 con 0x7f53bc09de60 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient(hunting): authenticate will time out at 1504.994098s 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53bbfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f53bc09de60 0x7f53bc138250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49986/0 (socket says 172.21.15.6:49986) 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53bb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f53bc09e240 0x7f53bc1d0530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43558/0 (socket says 172.21.15.6:43558) 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53bbfff640 1 -- 172.21.15.6:0/3305454538 learned_addr learned my addr 172.21.15.6:0/3305454538 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831314098 0 0) 0x7f53bc09f550 con 0x7f53bc09de60 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): my global_id is 6737 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53a8003530 con 0x7f53bc09de60 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565523078 0 0) 0x7f53bc120ae0 con 0x7f53bc09e240 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): my global_id is 124272 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53bc09f550 con 0x7f53bc09e240 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1646944737 0 0) 0x7f53a8003530 con 0x7f53bc09de60 2024-09-16T05:44:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53bc120ae0 con 0x7f53bc09de60 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53ac001be0 con 0x7f53bc09de60 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): dump: 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.210 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 671643461 0 0) 0x7f53bc09f550 con 0x7f53bc09e240 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53a8003530 con 0x7f53bc09e240 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53b00018e0 con 0x7f53bc09e240 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient(hunting): dump: 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.212 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1857516074 0 0) 0x7f53bc120ae0 con 0x7f53bc09de60 2024-09-16T05:44:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 >> v1:172.21.15.63:6789/0 conn(0x7f53bc139460 legacy=0x7f53bc1d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 >> v1:172.21.15.6:6789/0 conn(0x7f53bc09e240 legacy=0x7f53bc1d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 monclient: found mon.c 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53bc0740c0 con 0x7f53bc09de60 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient: _finish_auth 0 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 5 monclient: authenticate success, global_id 6737 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: _renew_subs 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53bc120800 con 0x7f53bc09de60 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: _renew_subs 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f53bc138960 con 0x7f53bc09de60 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53ac001890 con 0x7f53bc09de60 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53ac004da0 con 0x7f53bc09de60 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53ba7fc640 4 set_mon_vals no callback set 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 10 monclient: dump: 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f539c0065d0 con 0x7f539c002870 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.186+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f53ac00fc00 con 0x7f53bc09de60 2024-09-16T05:44:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f53ac044ef0 con 0x7f53bc09de60 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53c0fba640 10 monclient: get_auth_request con 0x7f53a8036970 auth_method 0 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1521867074 0 0) 0x7f539c0065d0 con 0x7f539c002870 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient(hunting): my global_id is 134110 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53bc120ae0 con 0x7f539c002870 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1066656017 0 0) 0x7f53bc120ae0 con 0x7f539c002870 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f539c0065d0 con 0x7f539c002870 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53b0003a30 con 0x7f539c002870 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:44:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient: dump: 2024-09-16T05:44:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2372970170 0 0) 0x7f539c0065d0 con 0x7f539c002870 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f539c00a160 con 0x7f539c002870 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f53b00057d0 con 0x7f539c002870 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 >> v1:172.21.15.63:6789/0 conn(0x7f539c002870 legacy=0x7f539c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.190+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 >> v1:172.21.15.63:6789/0 conn(0x7f539c002870 legacy=0x7f539c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f539c009e30 con 0x7f539c0090f0 2024-09-16T05:44:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038926046 0 0) 0x7f539c009e30 con 0x7f539c0090f0 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient(hunting): my global_id is 134113 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f539c0065d0 con 0x7f539c0090f0 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 551492968 0 0) 0x7f539c0065d0 con 0x7f539c0090f0 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f539c009e30 con 0x7f539c0090f0 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f53b0004130 con 0x7f539c0090f0 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient: dump: 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1328080648 0 0) 0x7f539c009e30 con 0x7f539c0090f0 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f539c00c3a0 con 0x7f539c0090f0 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3959 (unknown 0 0 4175384408) 0x7f53b00043e0 con 0x7f539c0090f0 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:44:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:03.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 >> v1:172.21.15.63:6789/0 conn(0x7f539c0090f0 legacy=0x7f539c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53b8ff9640 1 -- 172.21.15.6:0/3305454538 >> v1:172.21.15.63:6789/0 conn(0x7f539c0090f0 legacy=0x7f539c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:03.221 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:03.221 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:03.221 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:03.221 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 40, 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 787, 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:44:03.222 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:03.224 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:03.225 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:03.226 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.227 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:44:03.228 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:44:03.229 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:03.230 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:03.231 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:03.232 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:03.233 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:03.233 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:03.233 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:03.233 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:03.233 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:03.233 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:44:03.233 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:03.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 >> v1:172.21.15.63:6812/3643207211 conn(0x7f53a8036970 legacy=0x7f53a8038e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 10 monclient: shutdown 2024-09-16T05:44:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 >> v1:172.21.15.6:6790/0 conn(0x7f53bc09de60 legacy=0x7f53bc138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c0fba640 1 -- 172.21.15.6:0/3305454538 reap_dead start 2024-09-16T05:44:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c0fba640 1 -- 172.21.15.6:0/3305454538 reap_dead start 2024-09-16T05:44:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 shutdown_connections 2024-09-16T05:44:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 >> 172.21.15.6:0/3305454538 conn(0x7f53bc0a1650 msgr2=0x7f53bc09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 shutdown_connections 2024-09-16T05:44:03.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.198+0000 7f53c2a1d640 1 -- 172.21.15.6:0/3305454538 wait complete. 2024-09-16T05:44:03.235 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:44:03.235 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:44:03.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:44:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.342+0000 7efe6435d640 10 monclient: get_monmap_and_config 2024-09-16T05:44:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.342+0000 7efe6435d640 10 monclient: build_initial_monmap 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.342+0000 7efe6435d640 10 monclient: monmap: 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:03.345936+0000 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:03.345936+0000 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:03.345 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 Processor -- start 2024-09-16T05:44:03.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- start start 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient: init 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient(hunting): picked mon.noname-c con 0x7efe5c0a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient(hunting): picked mon.noname-b con 0x7efe5c0a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient(hunting): picked mon.noname-a con 0x7efe5c0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe5c06e520 con 0x7efe5c0a1a00 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe5c0a0780 con 0x7efe5c0a37a0 2024-09-16T05:44:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe5c0a4c70 con 0x7efe5c0a4290 2024-09-16T05:44:03.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:03.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient(hunting): authenticate will time out at 1505.153060s 2024-09-16T05:44:03.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe628fa640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efe5c0a1a00 0x7efe5c13f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43572/0 (socket says 172.21.15.6:43572) 2024-09-16T05:44:03.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe628fa640 1 -- 172.21.15.6:0/829798523 learned_addr learned my addr 172.21.15.6:0/829798523 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:03.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4004767834 0 0) 0x7efe5c06e520 con 0x7efe5c0a1a00 2024-09-16T05:44:03.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): my global_id is 124275 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe48003530 con 0x7efe5c0a1a00 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458813239 0 0) 0x7efe5c0a4c70 con 0x7efe5c0a4290 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): my global_id is 6740 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe5c06e520 con 0x7efe5c0a4290 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823198930 0 0) 0x7efe5c0a0780 con 0x7efe5c0a37a0 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient(hunting): my global_id is 134116 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe5c0a4c70 con 0x7efe5c0a37a0 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3893260663 0 0) 0x7efe5c06e520 con 0x7efe5c0a4290 2024-09-16T05:44:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 >> v1:172.21.15.63:6789/0 conn(0x7efe5c0a37a0 legacy=0x7efe5c0a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 >> v1:172.21.15.6:6789/0 conn(0x7efe5c0a1a00 legacy=0x7efe5c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: _finish_hunting 0 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 monclient: found mon.noname-b 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5c0a65f0 con 0x7efe5c0a4290 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: _finish_auth 0 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: _check_auth_tickets 2024-09-16T05:44:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe4c001580 con 0x7efe5c0a4290 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 5 monclient: authenticate success, global_id 6740 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: dump: 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.360 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.361 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.361 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.361 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.361 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.361 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.361 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.361 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:03.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe4c001860 con 0x7efe5c0a4290 2024-09-16T05:44:03.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:03.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 1 -- 172.21.15.6:0/829798523 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe4c001c30 con 0x7efe5c0a4290 2024-09-16T05:44:03.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:03.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 4 set_mon_vals no callback set 2024-09-16T05:44:03.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe610f7640 10 monclient: dump: 2024-09-16T05:44:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.363 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient: shutdown 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- 172.21.15.6:0/829798523 >> v1:172.21.15.6:6790/0 conn(0x7efe5c0a4290 legacy=0x7efe5c13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- 172.21.15.6:0/829798523 shutdown_connections 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- 172.21.15.6:0/829798523 >> 172.21.15.6:0/829798523 conn(0x7efe5c0a58a0 msgr2=0x7efe5c0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:03.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- 172.21.15.6:0/829798523 shutdown_connections 2024-09-16T05:44:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- 172.21.15.6:0/829798523 wait complete. 2024-09-16T05:44:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient: build_initial_monmap 2024-09-16T05:44:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient: monmap: 2024-09-16T05:44:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:03.366 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.366 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:03.366 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:03.366 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:03.366 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 Processor -- start 2024-09-16T05:44:03.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 1 -- start start 2024-09-16T05:44:03.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.346+0000 7efe6435d640 10 monclient: init 2024-09-16T05:44:03.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:03.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:03.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient(hunting): picked mon.noname-a con 0x7efe5c0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:03.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient(hunting): picked mon.noname-b con 0x7efe5c0a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:03.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient(hunting): picked mon.noname-c con 0x7efe5c0a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:03.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe5c0a0100 con 0x7efe5c0a1a00 2024-09-16T05:44:03.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe5c1cbdf0 con 0x7efe5c0a37a0 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe5c136130 con 0x7efe5c0a4290 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe620f9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efe5c0a1a00 0x7efe5c13f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43584/0 (socket says 172.21.15.6:43584) 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe620f9640 1 -- 172.21.15.6:0/1754852874 learned_addr learned my addr 172.21.15.6:0/1754852874 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient(hunting): authenticate will time out at 1505.156357s 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1669525294 0 0) 0x7efe5c0a0100 con 0x7efe5c0a1a00 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): my global_id is 124278 2024-09-16T05:44:03.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe40003530 con 0x7efe5c0a1a00 2024-09-16T05:44:03.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618988792 0 0) 0x7efe5c136130 con 0x7efe5c0a4290 2024-09-16T05:44:03.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:03.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:03.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:03.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): my global_id is 6743 2024-09-16T05:44:03.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe5c0a0100 con 0x7efe5c0a4290 2024-09-16T05:44:03.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4196632999 0 0) 0x7efe40003530 con 0x7efe5c0a1a00 2024-09-16T05:44:03.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe5c136130 con 0x7efe5c0a1a00 2024-09-16T05:44:03.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe580019a0 con 0x7efe5c0a1a00 2024-09-16T05:44:03.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:03.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:03.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): dump: 2024-09-16T05:44:03.371 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 185200095 0 0) 0x7efe5c0a0100 con 0x7efe5c0a4290 2024-09-16T05:44:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe40003530 con 0x7efe5c0a4290 2024-09-16T05:44:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe54001c20 con 0x7efe5c0a4290 2024-09-16T05:44:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient(hunting): dump: 2024-09-16T05:44:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.373 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.374 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.374 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1409131534 0 0) 0x7efe5c136130 con 0x7efe5c0a1a00 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 >> v1:172.21.15.6:6790/0 conn(0x7efe5c0a4290 legacy=0x7efe5c1d3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 >> v1:172.21.15.63:6789/0 conn(0x7efe5c0a37a0 legacy=0x7efe5c1d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 monclient: found mon.a 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5c1355d0 con 0x7efe5c0a1a00 2024-09-16T05:44:03.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient: _finish_auth 0 2024-09-16T05:44:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:44:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 5 monclient: authenticate success, global_id 124278 2024-09-16T05:44:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient: _renew_subs 2024-09-16T05:44:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe5c0a7d60 con 0x7efe5c0a1a00 2024-09-16T05:44:03.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient: _renew_subs 2024-09-16T05:44:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efe5c13f7d0 con 0x7efe5c0a1a00 2024-09-16T05:44:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe58001c80 con 0x7efe5c0a1a00 2024-09-16T05:44:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe58003a70 con 0x7efe5c0a1a00 2024-09-16T05:44:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:03.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe608f6640 4 set_mon_vals no callback set 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 10 monclient: dump: 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:03.378 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe44005090 con 0x7efe5c0a1a00 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7efe58003c10 con 0x7efe5c0a1a00 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7efe58044e10 con 0x7efe5c0a1a00 2024-09-16T05:44:03.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.350+0000 7efe618f8640 10 monclient: get_auth_request con 0x7efe40036a10 auth_method 0 2024-09-16T05:44:03.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.354+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7efe580154c0 con 0x7efe5c0a1a00 2024-09-16T05:44:03.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.354+0000 7efe52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:03.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.354+0000 7efe52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.578+0000 7efe6435d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:44:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.578+0000 7efe6435d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:44:03.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.578+0000 7efe6435d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:03.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.578+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7efe44005b90 con 0x7efe5c0a1a00 2024-09-16T05:44:03.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.578+0000 7efe52ffd640 1 -- 172.21.15.6:0/1754852874 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7efe58014dc0 con 0x7efe5c0a1a00 2024-09-16T05:44:03.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.578+0000 7efe52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:44:03.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.578+0000 7efe52ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:03.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.578+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.578+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1078604759,config=3541143443,health=1513525884,kv=2361287224,logm=427802627}) 2024-09-16T05:44:03.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.578+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=10,kv=10,logm=55} crc {auth=760767138,config=3541143443,health=1954957875,kv=3130718821,logm=3931782164}) 2024-09-16T05:44:03.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.578+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.578+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1078604759,config=3541143443,health=1513525884,kv=2361287224,logm=427802627}) 2024-09-16T05:44:03.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.578+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=3968369155,config=3541143443,health=1513525884,kv=2361287224,logm=3493442598}) 2024-09-16T05:44:03.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.582+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 >> v1:172.21.15.63:6812/3643207211 conn(0x7efe40036a10 legacy=0x7efe40038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.582+0000 7efe6435d640 10 monclient: shutdown 2024-09-16T05:44:03.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.582+0000 7efe6435d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:03.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.582+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 >> v1:172.21.15.6:6789/0 conn(0x7efe5c0a1a00 legacy=0x7efe5c13f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:03.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.582+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 shutdown_connections 2024-09-16T05:44:03.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.582+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 >> 172.21.15.6:0/1754852874 conn(0x7efe5c0a58a0 msgr2=0x7efe5c130570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:03.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.582+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 shutdown_connections 2024-09-16T05:44:03.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:03.582+0000 7efe6435d640 1 -- 172.21.15.6:0/1754852874 wait complete. 2024-09-16T05:44:03.584 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.582+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.582+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762518558}) 2024-09-16T05:44:03.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.582+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3808757163}) 2024-09-16T05:44:03.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.582+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.582+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762518558}) 2024-09-16T05:44:03.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.582+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235642221}) 2024-09-16T05:44:03.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979875981}) 2024-09-16T05:44:03.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=935680668}) 2024-09-16T05:44:03.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979875981}) 2024-09-16T05:44:03.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697403377}) 2024-09-16T05:44:03.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534317817}) 2024-09-16T05:44:03.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622652039}) 2024-09-16T05:44:03.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534317817}) 2024-09-16T05:44:03.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=601258108}) 2024-09-16T05:44:03.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.586+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2023373613}) 2024-09-16T05:44:03.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101515591}) 2024-09-16T05:44:03.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2023373613}) 2024-09-16T05:44:03.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941607612}) 2024-09-16T05:44:03.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125458271}) 2024-09-16T05:44:03.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2867269861}) 2024-09-16T05:44:03.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125458271}) 2024-09-16T05:44:03.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522587456}) 2024-09-16T05:44:03.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.594 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2931259152}) 2024-09-16T05:44:03.594 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.590+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133106178}) 2024-09-16T05:44:03.594 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.594 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2931259152}) 2024-09-16T05:44:03.594 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3907380177}) 2024-09-16T05:44:03.596 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.596 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305109632}) 2024-09-16T05:44:03.596 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=296585993}) 2024-09-16T05:44:03.596 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.596 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305109632}) 2024-09-16T05:44:03.596 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.594+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3857405878}) 2024-09-16T05:44:03.599 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.598+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.599 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.598+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151992532}) 2024-09-16T05:44:03.600 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.598+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3533387710}) 2024-09-16T05:44:03.600 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.598+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.600 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.598+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151992532}) 2024-09-16T05:44:03.600 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.598+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2958913852}) 2024-09-16T05:44:03.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:44:03.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.602+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.602+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2377072673}) 2024-09-16T05:44:03.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.602+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2269694845}) 2024-09-16T05:44:03.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.602+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.602+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2377072673}) 2024-09-16T05:44:03.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.602+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732865812}) 2024-09-16T05:44:03.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704584204}) 2024-09-16T05:44:03.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2327502092}) 2024-09-16T05:44:03.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704584204}) 2024-09-16T05:44:03.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4076899307}) 2024-09-16T05:44:03.608 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.608 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576070437}) 2024-09-16T05:44:03.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2017587488}) 2024-09-16T05:44:03.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576070437}) 2024-09-16T05:44:03.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=257873463}) 2024-09-16T05:44:03.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237039891}) 2024-09-16T05:44:03.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.606+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278499472}) 2024-09-16T05:44:03.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237039891}) 2024-09-16T05:44:03.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495376097}) 2024-09-16T05:44:03.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99660746}) 2024-09-16T05:44:03.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866773357}) 2024-09-16T05:44:03.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99660746}) 2024-09-16T05:44:03.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495631511}) 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637983523}) 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058152803}) 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637983523}) 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.610+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=59104516}) 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085068673}) 2024-09-16T05:44:03.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2414303945}) 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085068673}) 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3346389431}) 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722490894}) 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=515515349}) 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722490894}) 2024-09-16T05:44:03.618 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.614+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444802910}) 2024-09-16T05:44:03.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283240171}) 2024-09-16T05:44:03.620 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=721340534}) 2024-09-16T05:44:03.620 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.620 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283240171}) 2024-09-16T05:44:03.620 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2932639639}) 2024-09-16T05:44:03.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051819295}) 2024-09-16T05:44:03.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3944838863}) 2024-09-16T05:44:03.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051819295}) 2024-09-16T05:44:03.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.618+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1137922347}) 2024-09-16T05:44:03.622 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.622 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717576491}) 2024-09-16T05:44:03.622 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816314603}) 2024-09-16T05:44:03.622 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.622 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717576491}) 2024-09-16T05:44:03.622 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392102776}) 2024-09-16T05:44:03.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136041916}) 2024-09-16T05:44:03.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2979819195}) 2024-09-16T05:44:03.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136041916}) 2024-09-16T05:44:03.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1303179522}) 2024-09-16T05:44:03.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283283870}) 2024-09-16T05:44:03.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1056046916}) 2024-09-16T05:44:03.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.622+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283283870}) 2024-09-16T05:44:03.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3794710911}) 2024-09-16T05:44:03.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204153000}) 2024-09-16T05:44:03.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187142439}) 2024-09-16T05:44:03.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204153000}) 2024-09-16T05:44:03.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=78860485}) 2024-09-16T05:44:03.629 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.629 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459970066}) 2024-09-16T05:44:03.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2377291895}) 2024-09-16T05:44:03.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459970066}) 2024-09-16T05:44:03.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.626+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2694085450}) 2024-09-16T05:44:03.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3347775951}) 2024-09-16T05:44:03.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=486198415}) 2024-09-16T05:44:03.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3347775951}) 2024-09-16T05:44:03.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=384668069}) 2024-09-16T05:44:03.633 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544331142}) 2024-09-16T05:44:03.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4119965132}) 2024-09-16T05:44:03.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544331142}) 2024-09-16T05:44:03.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.630+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273397622}) 2024-09-16T05:44:03.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043430754}) 2024-09-16T05:44:03.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1326103462}) 2024-09-16T05:44:03.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043430754}) 2024-09-16T05:44:03.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=974754818}) 2024-09-16T05:44:03.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495123767}) 2024-09-16T05:44:03.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720985544}) 2024-09-16T05:44:03.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495123767}) 2024-09-16T05:44:03.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.634+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2090628681}) 2024-09-16T05:44:03.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133106553}) 2024-09-16T05:44:03.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1861126765}) 2024-09-16T05:44:03.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133106553}) 2024-09-16T05:44:03.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3441725132}) 2024-09-16T05:44:03.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180526174}) 2024-09-16T05:44:03.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2219663861}) 2024-09-16T05:44:03.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180526174}) 2024-09-16T05:44:03.641 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=455671555}) 2024-09-16T05:44:03.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689015444}) 2024-09-16T05:44:03.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773742712}) 2024-09-16T05:44:03.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689015444}) 2024-09-16T05:44:03.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.638+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3457524124}) 2024-09-16T05:44:03.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2262125844}) 2024-09-16T05:44:03.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=397201732}) 2024-09-16T05:44:03.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2262125844}) 2024-09-16T05:44:03.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282191394}) 2024-09-16T05:44:03.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3210954370}) 2024-09-16T05:44:03.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991801345}) 2024-09-16T05:44:03.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3210954370}) 2024-09-16T05:44:03.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.642+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=429748119}) 2024-09-16T05:44:03.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028920825}) 2024-09-16T05:44:03.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=78558091}) 2024-09-16T05:44:03.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028920825}) 2024-09-16T05:44:03.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3792667252}) 2024-09-16T05:44:03.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367226114}) 2024-09-16T05:44:03.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=984725825}) 2024-09-16T05:44:03.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367226114}) 2024-09-16T05:44:03.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.646+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4294378214}) 2024-09-16T05:44:03.650 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.650 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695112778}) 2024-09-16T05:44:03.650 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066644716}) 2024-09-16T05:44:03.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695112778}) 2024-09-16T05:44:03.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186609688}) 2024-09-16T05:44:03.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1006175735}) 2024-09-16T05:44:03.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3900995127}) 2024-09-16T05:44:03.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1006175735}) 2024-09-16T05:44:03.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.650+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071610284}) 2024-09-16T05:44:03.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609558407}) 2024-09-16T05:44:03.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1550226827}) 2024-09-16T05:44:03.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609558407}) 2024-09-16T05:44:03.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2515644103}) 2024-09-16T05:44:03.656 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.656 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101243922}) 2024-09-16T05:44:03.656 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2570428362}) 2024-09-16T05:44:03.656 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.656 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101243922}) 2024-09-16T05:44:03.656 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.654+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1275405530}) 2024-09-16T05:44:03.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940274088}) 2024-09-16T05:44:03.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271321670}) 2024-09-16T05:44:03.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940274088}) 2024-09-16T05:44:03.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=337838995}) 2024-09-16T05:44:03.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652478169}) 2024-09-16T05:44:03.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=995524459}) 2024-09-16T05:44:03.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652478169}) 2024-09-16T05:44:03.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.658+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=431717949}) 2024-09-16T05:44:03.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=57,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,osd_metadata=2} crc {logm=2275610708,mdsmap=3917278932,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=1452521012,mon_config_key=782542404,monmap=3625042558,osd_metadata=1495888149}) 2024-09-16T05:44:03.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=57,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,osd_metadata=4} crc {logm=978157084,mdsmap=1832746017,mgr=2255173110,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3209814469,mon_config_key=2055942112,monmap=3625042558,osd_metadata=2028894172}) 2024-09-16T05:44:03.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.663 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=57,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,osd_metadata=2} crc {logm=2275610708,mdsmap=3917278932,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=1452521012,mon_config_key=782542404,monmap=3625042558,osd_metadata=1495888149}) 2024-09-16T05:44:03.663 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=12,monmap=3,osd_metadata=3} crc {logm=2341127904,mdsmap=1832746017,mgr=848307873,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=1452521012,mon_config_key=4073353691,monmap=3625042558,osd_metadata=1277120950}) 2024-09-16T05:44:03.665 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.665 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=93} crc {osd_metadata=3695383094,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=3552653607}) 2024-09-16T05:44:03.665 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=3,osd_snap=1,osdmap=96} crc {osd_metadata=147319237,osd_snap=3823785925,osdmap=1897621536}) 2024-09-16T05:44:03.665 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.665 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=93} crc {osd_metadata=3695383094,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=3552653607}) 2024-09-16T05:44:03.665 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.662+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=93} crc {osd_metadata=3695383094,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=4279415812}) 2024-09-16T05:44:03.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.666+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.666+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3553232799}) 2024-09-16T05:44:03.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.666+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3333176171}) 2024-09-16T05:44:03.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.666+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.666+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3553232799}) 2024-09-16T05:44:03.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.666+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=726138189}) 2024-09-16T05:44:03.672 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.670+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.672 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.670+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1271563265}) 2024-09-16T05:44:03.672 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.670+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1512312851}) 2024-09-16T05:44:03.672 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.670+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.672 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.670+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1271563265}) 2024-09-16T05:44:03.672 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.670+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2186203877}) 2024-09-16T05:44:03.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.682+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.682+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=75} crc {osdmap=1511555778}) 2024-09-16T05:44:03.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.682+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=3947037191}) 2024-09-16T05:44:03.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.682+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:44:03.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.682+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=75} crc {osdmap=1511555778}) 2024-09-16T05:44:03.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:44:03.682+0000 7f0da3f99640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=77} crc {osdmap=1000126746}) 2024-09-16T05:44:04.602 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:44:04.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:44:04.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient: get_monmap_and_config 2024-09-16T05:44:04.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient: build_initial_monmap 2024-09-16T05:44:04.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient: monmap: 2024-09-16T05:44:04.732 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:04.732 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:04.732 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:04.733264+0000 2024-09-16T05:44:04.732 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:04.733264+0000 2024-09-16T05:44:04.733 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:04.733 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:04.733 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:04.733 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:04.733 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:04.733 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:04.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 1 Processor -- start 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 1 -- start start 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient: init 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient(hunting): picked mon.noname-c con 0x7fc40c072460 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient(hunting): picked mon.noname-b con 0x7fc40c0a0ed0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient(hunting): picked mon.noname-a con 0x7fc40c0a29f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40c0a0210 con 0x7fc40c0a29f0 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40c072f50 con 0x7fc40c072460 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40c09f7c0 con 0x7fc40c0a0ed0 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.730+0000 7fc413d20640 10 monclient(hunting): authenticate will time out at 1506.539988s 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc4122bd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc40c0a29f0 0x7fc40c13ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43592/0 (socket says 172.21.15.6:43592) 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc4122bd640 1 -- 172.21.15.6:0/1138021219 learned_addr learned my addr 172.21.15.6:0/1138021219 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893418351 0 0) 0x7fc40c0a0210 con 0x7fc40c0a29f0 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient(hunting): _negotiate 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient(hunting): my global_id is 124281 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3f4003530 con 0x7fc40c0a29f0 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2450529878 0 0) 0x7fc40c09f7c0 con 0x7fc40c0a0ed0 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient(hunting): _negotiate 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient(hunting): my global_id is 6746 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc40c0a0210 con 0x7fc40c0a0ed0 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1890995524 0 0) 0x7fc3f4003530 con 0x7fc40c0a29f0 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 >> v1:172.21.15.6:6790/0 conn(0x7fc40c0a0ed0 legacy=0x7fc40c139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 >> v1:172.21.15.63:6789/0 conn(0x7fc40c072460 legacy=0x7fc40c072840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: _finish_hunting 0 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 monclient: found mon.noname-a 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40c141600 con 0x7fc40c0a29f0 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: _finish_auth 0 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: _check_auth_tickets 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc404001550 con 0x7fc40c0a29f0 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: dump: 2024-09-16T05:44:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:04.744 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 5 monclient: authenticate success, global_id 124281 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc404001740 con 0x7fc40c0a29f0 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 4 set_mon_vals no callback set 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 1 -- 172.21.15.6:0/1138021219 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc404001b50 con 0x7fc40c0a29f0 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc410aba640 10 monclient: dump: 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:04.747 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:04.747 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:04.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient: shutdown 2024-09-16T05:44:04.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:04.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- 172.21.15.6:0/1138021219 >> v1:172.21.15.6:6789/0 conn(0x7fc40c0a29f0 legacy=0x7fc40c13ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- 172.21.15.6:0/1138021219 shutdown_connections 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- 172.21.15.6:0/1138021219 >> 172.21.15.6:0/1138021219 conn(0x7fc40c0a3880 msgr2=0x7fc40c0a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- 172.21.15.6:0/1138021219 shutdown_connections 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- 172.21.15.6:0/1138021219 wait complete. 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient: build_initial_monmap 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient: monmap: 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 Processor -- start 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- start start 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient: init 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient(hunting): picked mon.noname-c con 0x7fc40c072460 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient(hunting): picked mon.noname-b con 0x7fc40c0a0ed0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient(hunting): picked mon.noname-a con 0x7fc40c0a29f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40c121020 con 0x7fc40c0a29f0 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40c122750 con 0x7fc40c0a0ed0 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40c071c40 con 0x7fc40c072460 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc413d20640 10 monclient(hunting): authenticate will time out at 1506.543140s 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc411abc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc40c072460 0x7fc40c1c9770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50010/0 (socket says 172.21.15.6:50010) 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc411abc640 1 -- 172.21.15.6:0/2649326015 learned_addr learned my addr 172.21.15.6:0/2649326015 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420510870 0 0) 0x7fc40c121020 con 0x7fc40c0a29f0 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): my global_id is 124284 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3f8003530 con 0x7fc40c0a29f0 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1119863940 0 0) 0x7fc40c071c40 con 0x7fc40c072460 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): my global_id is 6749 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc40c121020 con 0x7fc40c072460 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4168012356 0 0) 0x7fc3f8003530 con 0x7fc40c0a29f0 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc40c071c40 con 0x7fc40c0a29f0 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc404001a90 con 0x7fc40c0a29f0 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): dump: 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1904732760 0 0) 0x7fc40c121020 con 0x7fc40c072460 2024-09-16T05:44:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3f8003530 con 0x7fc40c072460 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3fc001980 con 0x7fc40c072460 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): dump: 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042797885 0 0) 0x7fc40c122750 con 0x7fc40c0a0ed0 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 10 monclient(hunting): my global_id is 134125 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.734+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc40c121020 con 0x7fc40c0a0ed0 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3627633198 0 0) 0x7fc40c071c40 con 0x7fc40c0a29f0 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 >> v1:172.21.15.6:6790/0 conn(0x7fc40c072460 legacy=0x7fc40c1c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 >> v1:172.21.15.63:6789/0 conn(0x7fc40c0a0ed0 legacy=0x7fc40c1ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 monclient: found mon.a 2024-09-16T05:44:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40c13e3c0 con 0x7fc40c0a29f0 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 5 monclient: authenticate success, global_id 124284 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 10 monclient: _renew_subs 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc40c0a2350 con 0x7fc40c0a29f0 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 10 monclient: _renew_subs 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc40c122470 con 0x7fc40c0a29f0 2024-09-16T05:44:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc404001740 con 0x7fc40c0a29f0 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc404004aa0 con 0x7fc40c0a29f0 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40bfff640 4 set_mon_vals no callback set 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 10 monclient: dump: 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3ec005090 con 0x7fc40c0a29f0 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc40400f840 con 0x7fc40c0a29f0 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fc404044d10 con 0x7fc40c0a29f0 2024-09-16T05:44:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.738+0000 7fc411abc640 10 monclient: get_auth_request con 0x7fc3f8036c90 auth_method 0 2024-09-16T05:44:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.742+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc404015400 con 0x7fc40c0a29f0 2024-09-16T05:44:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.742+0000 7fc40a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.742+0000 7fc40a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:04.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.966+0000 7fc413d20640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:44:04.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.966+0000 7fc413d20640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:04.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.966+0000 7fc413d20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:04.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3ec005380 con 0x7fc40c0a29f0 2024-09-16T05:44:04.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc40a7fc640 1 -- 172.21.15.6:0/2649326015 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 4224436410) 0x7fc404014d00 con 0x7fc40c0a29f0 2024-09-16T05:44:04.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc40a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:04.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc40a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:04.971 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:44:04.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc3f8036c90 legacy=0x7fc3f8039150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:04.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 10 monclient: shutdown 2024-09-16T05:44:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 >> v1:172.21.15.6:6789/0 conn(0x7fc40c0a29f0 legacy=0x7fc40c1d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 shutdown_connections 2024-09-16T05:44:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 >> 172.21.15.6:0/2649326015 conn(0x7fc40c0a3880 msgr2=0x7fc40c0701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 shutdown_connections 2024-09-16T05:44:04.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:04.970+0000 7fc413d20640 1 -- 172.21.15.6:0/2649326015 wait complete. 2024-09-16T05:44:04.988 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:44:04.989 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:44:04.989 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:44:04.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:44:05.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient: get_monmap_and_config 2024-09-16T05:44:05.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient: monmap: 2024-09-16T05:44:05.110 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.110 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.110 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:05.111497+0000 2024-09-16T05:44:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:05.111497+0000 2024-09-16T05:44:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:05.111 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 1 Processor -- start 2024-09-16T05:44:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 1 -- start start 2024-09-16T05:44:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient: init 2024-09-16T05:44:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:44:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient(hunting): picked mon.noname-c con 0x7f4b0c0a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient(hunting): picked mon.noname-a con 0x7f4b0c0a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient(hunting): picked mon.noname-b con 0x7f4b0c0a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0c06e520 con 0x7f4b0c0a4290 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0c0a0780 con 0x7f4b0c0a37a0 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0c0a4c70 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient(hunting): authenticate will time out at 1506.918241s 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0ad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4b0c0a1a00 0x7f4b0c13f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50026/0 (socket says 172.21.15.6:50026) 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0ad9d640 1 -- 172.21.15.6:0/2559646289 learned_addr learned my addr 172.21.15.6:0/2559646289 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089020026 0 0) 0x7f4b0c0a4c70 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient(hunting): my global_id is 6752 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4af4003530 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2523277393 0 0) 0x7f4b0c06e520 con 0x7f4b0c0a4290 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient(hunting): my global_id is 124287 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b0c0a4c70 con 0x7f4b0c0a4290 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4199032492 0 0) 0x7f4af4003530 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 >> v1:172.21.15.63:6789/0 conn(0x7f4b0c0a37a0 legacy=0x7f4b0c0a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 >> v1:172.21.15.6:6789/0 conn(0x7f4b0c0a4290 legacy=0x7f4b0c13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 monclient: found mon.noname-b 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c0a65f0 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b040019a0 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: dump: 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 5 monclient: authenticate success, global_id 6752 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b04001cc0 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 1 -- 172.21.15.6:0/2559646289 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b040020d0 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 4 set_mon_vals no callback set 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b0959a640 10 monclient: dump: 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.121 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 10 monclient: shutdown 2024-09-16T05:44:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 1 -- 172.21.15.6:0/2559646289 >> v1:172.21.15.6:6790/0 conn(0x7f4b0c0a1a00 legacy=0x7f4b0c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 1 -- 172.21.15.6:0/2559646289 shutdown_connections 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.110+0000 7f4b109ed640 1 -- 172.21.15.6:0/2559646289 >> 172.21.15.6:0/2559646289 conn(0x7f4b0c0a58a0 msgr2=0x7f4b0c0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- 172.21.15.6:0/2559646289 shutdown_connections 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- 172.21.15.6:0/2559646289 wait complete. 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: monmap: 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 Processor -- start 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- start start 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: init 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient(hunting): picked mon.noname-a con 0x7f4b0c0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient(hunting): picked mon.noname-b con 0x7f4b0c0a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient(hunting): picked mon.noname-c con 0x7f4b0c0a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0c135280 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0c132d30 con 0x7f4b0c0a37a0 2024-09-16T05:44:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0c1357d0 con 0x7f4b0c0a4290 2024-09-16T05:44:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient(hunting): authenticate will time out at 1506.921589s 2024-09-16T05:44:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b0ad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4b0c0a4290 0x7f4b0c1d3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50030/0 (socket says 172.21.15.6:50030) 2024-09-16T05:44:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b0ad9d640 1 -- 172.21.15.6:0/1322697210 learned_addr learned my addr 172.21.15.6:0/1322697210 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b0a59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b0c0a1a00 0x7f4b0c131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43618/0 (socket says 172.21.15.6:43618) 2024-09-16T05:44:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2907233441 0 0) 0x7f4b0c1357d0 con 0x7f4b0c0a4290 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): my global_id is 6755 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aec003530 con 0x7f4b0c0a4290 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2144427963 0 0) 0x7f4b0c135280 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): my global_id is 124290 2024-09-16T05:44:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b0c1357d0 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3904029150 0 0) 0x7f4aec003530 con 0x7f4b0c0a4290 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b0c135280 con 0x7f4b0c0a4290 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2618071555 0 0) 0x7f4b0c1357d0 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4aec003530 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b04001960 con 0x7f4b0c0a4290 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): dump: 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b000017c0 con 0x7f4b0c0a1a00 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): dump: 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452064947 0 0) 0x7f4b0c132d30 con 0x7f4b0c0a37a0 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): my global_id is 134131 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b0c1357d0 con 0x7f4b0c0a37a0 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3031989323 0 0) 0x7f4b0c135280 con 0x7f4b0c0a4290 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 >> v1:172.21.15.63:6789/0 conn(0x7f4b0c0a37a0 legacy=0x7f4b0c132620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 >> v1:172.21.15.6:6789/0 conn(0x7f4b0c0a1a00 legacy=0x7f4b0c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 monclient: found mon.c 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c135ab0 con 0x7f4b0c0a4290 2024-09-16T05:44:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 5 monclient: authenticate success, global_id 6755 2024-09-16T05:44:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: _renew_subs 2024-09-16T05:44:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0c137230 con 0x7f4b0c0a4290 2024-09-16T05:44:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: _renew_subs 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b0c133c90 con 0x7f4b0c0a4290 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b04005000 con 0x7f4b0c0a4290 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b040053d0 con 0x7f4b0c0a4290 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b08d99640 4 set_mon_vals no callback set 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient: dump: 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4af00065d0 con 0x7f4af0002870 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887665863 0 0) 0x7f4af00065d0 con 0x7f4af0002870 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 10 monclient(hunting): my global_id is 124293 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b0c135280 con 0x7f4af0002870 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.114+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4b040055f0 con 0x7f4b0c0a4290 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f4b040453f0 con 0x7f4b0c0a4290 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2235413275 0 0) 0x7f4b0c135280 con 0x7f4af0002870 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4af00065d0 con 0x7f4af0002870 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b000039d0 con 0x7f4af0002870 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 10 monclient: dump: 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2265193981 0 0) 0x7f4af00065d0 con 0x7f4af0002870 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4af000a160 con 0x7f4af0002870 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4b09d9b640 10 monclient: get_auth_request con 0x7f4aec036b20 auth_method 0 2024-09-16T05:44:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4b000057b0 con 0x7f4af0002870 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 >> v1:172.21.15.6:6789/0 conn(0x7f4af0002870 legacy=0x7f4af0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.118+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 >> v1:172.21.15.6:6789/0 conn(0x7f4af0002870 legacy=0x7f4af0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4b109ed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4af0009e50 con 0x7f4af0009110 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141815681 0 0) 0x7f4af0009e50 con 0x7f4af0009110 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient(hunting): my global_id is 124296 2024-09-16T05:44:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4af00065d0 con 0x7f4af0009110 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4105637345 0 0) 0x7f4af00065d0 con 0x7f4af0009110 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4af0009e50 con 0x7f4af0009110 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4b00004050 con 0x7f4af0009110 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient: dump: 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2616459714 0 0) 0x7f4af0009e50 con 0x7f4af0009110 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4af000c3e0 con 0x7f4af0009110 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4131 (unknown 0 0 3626317329) 0x7f4b00003e70 con 0x7f4af0009110 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 >> v1:172.21.15.6:6789/0 conn(0x7f4af0009110 legacy=0x7f4af00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.122+0000 7f4afaffd640 1 -- 172.21.15.6:0/1322697210 >> v1:172.21.15.6:6789/0 conn(0x7f4af0009110 legacy=0x7f4af00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:05.142 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:05.142 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 40, 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 110815, 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.143 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.144 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.145 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:05.146 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.147 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:05.148 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.149 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:44:05.151 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:44:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:05.155 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:44:05.156 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4aec036b20 legacy=0x7f4aec038fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b109ed640 10 monclient: shutdown 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b109ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 >> v1:172.21.15.6:6790/0 conn(0x7f4b0c0a4290 legacy=0x7f4b0c1d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b0ad9d640 1 -- 172.21.15.6:0/1322697210 reap_dead start 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b0ad9d640 1 -- 172.21.15.6:0/1322697210 reap_dead start 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 shutdown_connections 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 >> 172.21.15.6:0/1322697210 conn(0x7f4b0c0a58a0 msgr2=0x7f4b0c0a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 shutdown_connections 2024-09-16T05:44:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.126+0000 7f4b109ed640 1 -- 172.21.15.6:0/1322697210 wait complete. 2024-09-16T05:44:05.160 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:44:05.160 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:44:05.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:44:05.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.274+0000 7f66f0fb8640 10 monclient: get_monmap_and_config 2024-09-16T05:44:05.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.274+0000 7f66f0fb8640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.274+0000 7f66f0fb8640 10 monclient: monmap: 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:05.278789+0000 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:05.278789+0000 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:05.278 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 Processor -- start 2024-09-16T05:44:05.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- start start 2024-09-16T05:44:05.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient: init 2024-09-16T05:44:05.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:44:05.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient(hunting): picked mon.noname-a con 0x7f66ec0a0df0 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient(hunting): picked mon.noname-b con 0x7f66ec135330 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient(hunting): picked mon.noname-c con 0x7f66ec138e40 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec09d2f0 con 0x7f66ec0a0df0 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec13d960 con 0x7f66ec138e40 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec13eb40 con 0x7f66ec135330 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient(hunting): authenticate will time out at 1507.085420s 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e9d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f66ec135330 0x7f66ec137720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50040/0 (socket says 172.21.15.6:50040) 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e9d9b640 1 -- 172.21.15.6:0/1298776167 learned_addr learned my addr 172.21.15.6:0/1298776167 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127109630 0 0) 0x7f66ec09d2f0 con 0x7f66ec0a0df0 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient(hunting): my global_id is 124299 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66d4004640 con 0x7f66ec0a0df0 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2169290548 0 0) 0x7f66ec13eb40 con 0x7f66ec135330 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient(hunting): my global_id is 6758 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ec09d2f0 con 0x7f66ec135330 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2096202397 0 0) 0x7f66d4004640 con 0x7f66ec0a0df0 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 >> v1:172.21.15.6:6790/0 conn(0x7f66ec135330 legacy=0x7f66ec137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 >> v1:172.21.15.63:6789/0 conn(0x7f66ec138e40 legacy=0x7f66ec13b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 monclient: found mon.noname-a 2024-09-16T05:44:05.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:44:05.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec13fd20 con 0x7f66ec0a0df0 2024-09-16T05:44:05.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f66dc001620 con 0x7f66ec0a0df0 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: dump: 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 5 monclient: authenticate success, global_id 124299 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f66dc001810 con 0x7f66ec0a0df0 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 1 -- 172.21.15.6:0/1298776167 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f66dc001c20 con 0x7f66ec0a0df0 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 4 set_mon_vals no callback set 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66e959a640 10 monclient: dump: 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.286 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient: shutdown 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- 172.21.15.6:0/1298776167 >> v1:172.21.15.6:6789/0 conn(0x7f66ec0a0df0 legacy=0x7f66ec09de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- 172.21.15.6:0/1298776167 shutdown_connections 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- 172.21.15.6:0/1298776167 >> 172.21.15.6:0/1298776167 conn(0x7f66ec0a1650 msgr2=0x7f66ec0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- 172.21.15.6:0/1298776167 shutdown_connections 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- 172.21.15.6:0/1298776167 wait complete. 2024-09-16T05:44:05.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 10 monclient: monmap: 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 Processor -- start 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.278+0000 7f66f0fb8640 1 -- start start 2024-09-16T05:44:05.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient: init 2024-09-16T05:44:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:44:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient(hunting): picked mon.noname-c con 0x7f66ec0a0df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient(hunting): picked mon.noname-a con 0x7f66ec135330 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient(hunting): picked mon.noname-b con 0x7f66ec138e40 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec06fc60 con 0x7f66ec135330 2024-09-16T05:44:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec0a3080 con 0x7f66ec138e40 2024-09-16T05:44:05.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec0a2d00 con 0x7f66ec0a0df0 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient(hunting): authenticate will time out at 1507.088557s 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66ea59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f66ec0a0df0 0x7f66ec072480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50056/0 (socket says 172.21.15.6:50056) 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66ea59c640 1 -- 172.21.15.6:0/2845023672 learned_addr learned my addr 172.21.15.6:0/2845023672 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3083855305 0 0) 0x7f66ec0a2d00 con 0x7f66ec0a0df0 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): my global_id is 6761 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66cc003560 con 0x7f66ec0a0df0 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030841018 0 0) 0x7f66ec06fc60 con 0x7f66ec135330 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): my global_id is 124302 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ec0a2d00 con 0x7f66ec135330 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1753633147 0 0) 0x7f66cc003560 con 0x7f66ec0a0df0 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66ec06fc60 con 0x7f66ec0a0df0 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f66dc0036c0 con 0x7f66ec0a0df0 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:05.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): dump: 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.297 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:05.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3373795055 0 0) 0x7f66ec0a2d00 con 0x7f66ec135330 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66cc003560 con 0x7f66ec135330 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f66e0001180 con 0x7f66ec135330 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): dump: 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.299 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3339351355 0 0) 0x7f66ec0a3080 con 0x7f66ec138e40 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): my global_id is 134137 2024-09-16T05:44:05.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ec0a2d00 con 0x7f66ec138e40 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1169084278 0 0) 0x7f66ec06fc60 con 0x7f66ec0a0df0 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 >> v1:172.21.15.63:6789/0 conn(0x7f66ec138e40 legacy=0x7f66ec1d3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 >> v1:172.21.15.6:6789/0 conn(0x7f66ec135330 legacy=0x7f66ec1d04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 monclient: found mon.c 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec06f980 con 0x7f66ec0a0df0 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 5 monclient: authenticate success, global_id 6761 2024-09-16T05:44:05.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient: _renew_subs 2024-09-16T05:44:05.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec06e530 con 0x7f66ec0a0df0 2024-09-16T05:44:05.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient: _renew_subs 2024-09-16T05:44:05.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f66ec072b90 con 0x7f66ec0a0df0 2024-09-16T05:44:05.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f66dc003bd0 con 0x7f66ec0a0df0 2024-09-16T05:44:05.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f66dc003fa0 con 0x7f66ec0a0df0 2024-09-16T05:44:05.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66e8d99640 4 set_mon_vals no callback set 2024-09-16T05:44:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: dump: 2024-09-16T05:44:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f66d00065d0 con 0x7f66d0002870 2024-09-16T05:44:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f66dc0041c0 con 0x7f66ec0a0df0 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f66dc044eb0 con 0x7f66ec0a0df0 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315519547 0 0) 0x7f66d00065d0 con 0x7f66d0002870 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient(hunting): my global_id is 6764 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ec06fc60 con 0x7f66d0002870 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66ead9d640 10 monclient: get_auth_request con 0x7f66cc036cf0 auth_method 0 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4235682201 0 0) 0x7f66ec06fc60 con 0x7f66d0002870 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66d00065d0 con 0x7f66d0002870 2024-09-16T05:44:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f66e0004b80 con 0x7f66d0002870 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: dump: 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.308 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 745833442 0 0) 0x7f66d00065d0 con 0x7f66d0002870 2024-09-16T05:44:05.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66d000a160 con 0x7f66d0002870 2024-09-16T05:44:05.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f66e0004d60 con 0x7f66d0002870 2024-09-16T05:44:05.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 >> v1:172.21.15.6:6790/0 conn(0x7f66d0002870 legacy=0x7f66d0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.282+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 >> v1:172.21.15.6:6790/0 conn(0x7f66d0002870 legacy=0x7f66d0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f66d0009e50 con 0x7f66d0009110 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 195278907 0 0) 0x7f66d0009e50 con 0x7f66d0009110 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient(hunting): my global_id is 6767 2024-09-16T05:44:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66d00065d0 con 0x7f66d0009110 2024-09-16T05:44:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 685245499 0 0) 0x7f66d00065d0 con 0x7f66d0009110 2024-09-16T05:44:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66d0009e50 con 0x7f66d0009110 2024-09-16T05:44:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f66e0005320 con 0x7f66d0009110 2024-09-16T05:44:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient: dump: 2024-09-16T05:44:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2766784633 0 0) 0x7f66d0009e50 con 0x7f66d0009110 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f66d000c3e0 con 0x7f66d0009110 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 4230226245) 0x7f66e0005310 con 0x7f66d0009110 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 >> v1:172.21.15.6:6790/0 conn(0x7f66d0009110 legacy=0x7f66d00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66daffd640 1 -- 172.21.15.6:0/2845023672 >> v1:172.21.15.6:6790/0 conn(0x7f66d0009110 legacy=0x7f66d00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 40, 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 574347, 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:44:05.314 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.316 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:44:05.317 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:44:05.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:44:05.320 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.321 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:05.322 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.323 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.324 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.325 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.326 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:44:05.326 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 >> v1:172.21.15.63:6812/3643207211 conn(0x7f66cc036cf0 legacy=0x7f66cc0391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 10 monclient: shutdown 2024-09-16T05:44:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 >> v1:172.21.15.6:6790/0 conn(0x7f66ec0a0df0 legacy=0x7f66ec072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66ead9d640 1 -- 172.21.15.6:0/2845023672 reap_dead start 2024-09-16T05:44:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66ead9d640 1 -- 172.21.15.6:0/2845023672 reap_dead start 2024-09-16T05:44:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 shutdown_connections 2024-09-16T05:44:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 >> 172.21.15.6:0/2845023672 conn(0x7f66ec0a1650 msgr2=0x7f66ec09b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 shutdown_connections 2024-09-16T05:44:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.290+0000 7f66f0fb8640 1 -- 172.21.15.6:0/2845023672 wait complete. 2024-09-16T05:44:05.328 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:44:05.328 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:44:05.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:44:05.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient: get_monmap_and_config 2024-09-16T05:44:05.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient: monmap: 2024-09-16T05:44:05.442 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.442 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.442 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:05.443527+0000 2024-09-16T05:44:05.443 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:05.443527+0000 2024-09-16T05:44:05.443 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.443 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.443 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.443 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:05.443 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:05.443 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 Processor -- start 2024-09-16T05:44:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- start start 2024-09-16T05:44:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient: init 2024-09-16T05:44:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:44:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient(hunting): picked mon.noname-a con 0x7fa61809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient(hunting): picked mon.noname-b con 0x7fa61809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient(hunting): picked mon.noname-c con 0x7fa6180a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6180a1370 con 0x7fa61809aa30 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61809ead0 con 0x7fa6180a0da0 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61809edb0 con 0x7fa61809e230 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient(hunting): authenticate will time out at 1507.250141s 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa616ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa61809e230 0x7fa6181399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50080/0 (socket says 172.21.15.6:50080) 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa616ffd640 1 -- 172.21.15.6:0/853190521 learned_addr learned my addr 172.21.15.6:0/853190521 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6177fe640 1 --1- 172.21.15.6:0/853190521 >> v1:172.21.15.6:6789/0 conn(0x7fa61809aa30 0x7fa61809ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43678/0 (socket says 172.21.15.6:43678) 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1376483565 0 0) 0x7fa61809edb0 con 0x7fa61809e230 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient(hunting): my global_id is 6770 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa600003530 con 0x7fa61809e230 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3159089715 0 0) 0x7fa6180a1370 con 0x7fa61809aa30 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient(hunting): my global_id is 124305 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61809edb0 con 0x7fa61809aa30 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3196598922 0 0) 0x7fa600003530 con 0x7fa61809e230 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 >> v1:172.21.15.63:6789/0 conn(0x7fa6180a0da0 legacy=0x7fa61813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 >> v1:172.21.15.6:6789/0 conn(0x7fa61809aa30 legacy=0x7fa61809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 monclient: found mon.noname-b 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61809dd50 con 0x7fa61809e230 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa60c001620 con 0x7fa61809e230 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: dump: 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.455 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 5 monclient: authenticate success, global_id 6770 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa60c001890 con 0x7fa61809e230 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 1 -- 172.21.15.6:0/853190521 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa60c001c60 con 0x7fa61809e230 2024-09-16T05:44:05.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa6167fc640 10 monclient: dump: 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 4 set_mon_vals no callback set 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 10 monclient: shutdown 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- 172.21.15.6:0/853190521 >> v1:172.21.15.6:6790/0 conn(0x7fa61809e230 legacy=0x7fa6181399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- 172.21.15.6:0/853190521 shutdown_connections 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- 172.21.15.6:0/853190521 >> 172.21.15.6:0/853190521 conn(0x7fa6180a1650 msgr2=0x7fa6180a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- 172.21.15.6:0/853190521 shutdown_connections 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.442+0000 7fa61e18e640 1 -- 172.21.15.6:0/853190521 wait complete. 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: monmap: 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:05.458 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 1 Processor -- start 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 1 -- start start 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: init 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:05.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient(hunting): picked mon.noname-a con 0x7fa61809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient(hunting): picked mon.noname-b con 0x7fa61809e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient(hunting): picked mon.noname-c con 0x7fa6180a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61811cf50 con 0x7fa61809aa30 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61811cb30 con 0x7fa61809e230 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61806ecf0 con 0x7fa6180a0da0 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient(hunting): authenticate will time out at 1507.253169s 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa6177fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa61809aa30 0x7fa61813b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43682/0 (socket says 172.21.15.6:43682) 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa617fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa6180a0da0 0x7fa6181d1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50090/0 (socket says 172.21.15.6:50090) 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa6177fe640 1 -- 172.21.15.6:0/3631575410 learned_addr learned my addr 172.21.15.6:0/3631575410 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1416811069 0 0) 0x7fa61809edb0 con 0x7fa61809aa30 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): my global_id is 124308 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa604003530 con 0x7fa61809aa30 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840522934 0 0) 0x7fa61806ecf0 con 0x7fa6180a0da0 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): my global_id is 6773 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61809edb0 con 0x7fa6180a0da0 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 748045102 0 0) 0x7fa604003530 con 0x7fa61809aa30 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa61806ecf0 con 0x7fa61809aa30 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa6080018e0 con 0x7fa61809aa30 2024-09-16T05:44:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): dump: 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.462 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4168707881 0 0) 0x7fa61809edb0 con 0x7fa6180a0da0 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa604003530 con 0x7fa6180a0da0 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa610001b40 con 0x7fa6180a0da0 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): dump: 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.463 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176691842 0 0) 0x7fa61811cb30 con 0x7fa61809e230 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): my global_id is 134143 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61809edb0 con 0x7fa61809e230 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3925539146 0 0) 0x7fa61806ecf0 con 0x7fa61809aa30 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 >> v1:172.21.15.6:6790/0 conn(0x7fa6180a0da0 legacy=0x7fa6181d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 >> v1:172.21.15.63:6789/0 conn(0x7fa61809e230 legacy=0x7fa61813bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 monclient: found mon.a 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6180740c0 con 0x7fa61809aa30 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 5 monclient: authenticate success, global_id 124308 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: _renew_subs 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:05.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa61811c850 con 0x7fa61809aa30 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: _renew_subs 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa61813c650 con 0x7fa61809aa30 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6080047f0 con 0x7fa61809aa30 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa608004b40 con 0x7fa61809aa30 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa615ffb640 4 set_mon_vals no callback set 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient: dump: 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.466 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5f80065d0 con 0x7fa5f8002870 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa60800f960 con 0x7fa61809aa30 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fa608044ed0 con 0x7fa61809aa30 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa617fff640 10 monclient: get_auth_request con 0x7fa604036c40 auth_method 0 2024-09-16T05:44:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662132197 0 0) 0x7fa5f80065d0 con 0x7fa5f8002870 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 10 monclient(hunting): my global_id is 134146 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.446+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61806ecf0 con 0x7fa5f8002870 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4281364700 0 0) 0x7fa61806ecf0 con 0x7fa5f8002870 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5f80065d0 con 0x7fa5f8002870 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa60c0016a0 con 0x7fa5f8002870 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 10 monclient: dump: 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.468 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2297231636 0 0) 0x7fa5f80065d0 con 0x7fa5f8002870 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5f800a160 con 0x7fa5f8002870 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa60c001a10 con 0x7fa5f8002870 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:05.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 >> v1:172.21.15.63:6789/0 conn(0x7fa5f8002870 legacy=0x7fa5f8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.450+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 >> v1:172.21.15.63:6789/0 conn(0x7fa5f8002870 legacy=0x7fa5f8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:05.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.454+0000 7fa61e18e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:44:05.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.454+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5f8009e50 con 0x7fa5f8009110 2024-09-16T05:44:05.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322881010 0 0) 0x7fa5f8009e50 con 0x7fa5f8009110 2024-09-16T05:44:05.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient(hunting): my global_id is 134149 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5f80065d0 con 0x7fa5f8009110 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 432251759 0 0) 0x7fa5f80065d0 con 0x7fa5f8009110 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5f8009e50 con 0x7fa5f8009110 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa60c004b40 con 0x7fa5f8009110 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient: dump: 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.472 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4024928924 0 0) 0x7fa5f8009e50 con 0x7fa5f8009110 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa5f800c3e0 con 0x7fa5f8009110 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3960 (unknown 0 0 3085984211) 0x7fa60c004ba0 con 0x7fa5f8009110 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 >> v1:172.21.15.63:6789/0 conn(0x7fa5f8009110 legacy=0x7fa5f80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa5fffff640 1 -- 172.21.15.6:0/3631575410 >> v1:172.21.15.63:6789/0 conn(0x7fa5f8009110 legacy=0x7fa5f80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 40, 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 3047, 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:44:05.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.475 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.476 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:44:05.477 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:05.478 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.479 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:44:05.480 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.481 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:05.482 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.483 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:05.484 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:05.485 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:05.485 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:05.485 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:05.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:05.485 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:44:05.485 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa604036c40 legacy=0x7fa604039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa61e18e640 10 monclient: shutdown 2024-09-16T05:44:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa61e18e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 >> v1:172.21.15.6:6789/0 conn(0x7fa61809aa30 legacy=0x7fa61813b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa617fff640 1 -- 172.21.15.6:0/3631575410 reap_dead start 2024-09-16T05:44:05.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa617fff640 1 -- 172.21.15.6:0/3631575410 reap_dead start 2024-09-16T05:44:05.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 shutdown_connections 2024-09-16T05:44:05.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 >> 172.21.15.6:0/3631575410 conn(0x7fa6180a1650 msgr2=0x7fa6181359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 shutdown_connections 2024-09-16T05:44:05.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.458+0000 7fa61e18e640 1 -- 172.21.15.6:0/3631575410 wait complete. 2024-09-16T05:44:05.487 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:44:05.487 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T05:44:05.488 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:44:05.488 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T05:44:05.488 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-16T05:44:05.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-16T05:44:05.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient: get_monmap_and_config 2024-09-16T05:44:05.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient: monmap: 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:05.604970+0000 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:05.604970+0000 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:05.604 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 1 Processor -- start 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 1 -- start start 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient: init 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient(hunting): picked mon.noname-c con 0x7fab380a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient(hunting): picked mon.noname-a con 0x7fab380a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient(hunting): picked mon.noname-b con 0x7fab380a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab3806e520 con 0x7fab380a4290 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab380a0780 con 0x7fab380a37a0 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab380a4c70 con 0x7fab380a1a00 2024-09-16T05:44:05.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.602+0000 7fab3fc5a640 10 monclient(hunting): authenticate will time out at 1507.411898s 2024-09-16T05:44:05.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3d1f5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fab380a4290 0x7fab3813bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43696/0 (socket says 172.21.15.6:43696) 2024-09-16T05:44:05.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3d1f5640 1 -- 172.21.15.6:0/3619964193 learned_addr learned my addr 172.21.15.6:0/3619964193 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 928892740 0 0) 0x7fab380a4c70 con 0x7fab380a1a00 2024-09-16T05:44:05.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient(hunting): my global_id is 6776 2024-09-16T05:44:05.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab24003530 con 0x7fab380a1a00 2024-09-16T05:44:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2946510396 0 0) 0x7fab3806e520 con 0x7fab380a4290 2024-09-16T05:44:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient(hunting): my global_id is 124311 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab380a4c70 con 0x7fab380a4290 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3254677592 0 0) 0x7fab24003530 con 0x7fab380a1a00 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 >> v1:172.21.15.63:6789/0 conn(0x7fab380a37a0 legacy=0x7fab380a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 >> v1:172.21.15.6:6789/0 conn(0x7fab380a4290 legacy=0x7fab3813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 monclient: found mon.noname-b 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab380a65f0 con 0x7fab380a1a00 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab300019a0 con 0x7fab380a1a00 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: dump: 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.612 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 5 monclient: authenticate success, global_id 6776 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab30001c00 con 0x7fab380a1a00 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 1 -- 172.21.15.6:0/3619964193 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab30001fd0 con 0x7fab380a1a00 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 4 set_mon_vals no callback set 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3c9f4640 10 monclient: dump: 2024-09-16T05:44:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 10 monclient: shutdown 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 1 -- 172.21.15.6:0/3619964193 >> v1:172.21.15.6:6790/0 conn(0x7fab380a1a00 legacy=0x7fab3813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 1 -- 172.21.15.6:0/3619964193 shutdown_connections 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.606+0000 7fab3fc5a640 1 -- 172.21.15.6:0/3619964193 >> 172.21.15.6:0/3619964193 conn(0x7fab380a58a0 msgr2=0x7fab380a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 1 -- 172.21.15.6:0/3619964193 shutdown_connections 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 1 -- 172.21.15.6:0/3619964193 wait complete. 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient: monmap: 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:05.615 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:05.616 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 1 Processor -- start 2024-09-16T05:44:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 1 -- start start 2024-09-16T05:44:05.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient: init 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient(hunting): picked mon.noname-a con 0x7fab380a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient(hunting): picked mon.noname-b con 0x7fab380a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient(hunting): picked mon.noname-c con 0x7fab380a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab3813ec30 con 0x7fab380a1a00 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab3813c080 con 0x7fab380a37a0 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab3813dc80 con 0x7fab380a4290 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3fc5a640 10 monclient(hunting): authenticate will time out at 1507.417713s 2024-09-16T05:44:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3e1f7640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fab380a4290 0x7fab381d3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50108/0 (socket says 172.21.15.6:50108) 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab3e1f7640 1 -- 172.21.15.6:0/2880647178 learned_addr learned my addr 172.21.15.6:0/2880647178 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347781440 0 0) 0x7fab3813ec30 con 0x7fab380a1a00 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): my global_id is 124314 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab1c003530 con 0x7fab380a1a00 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3633708083 0 0) 0x7fab1c003530 con 0x7fab380a1a00 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab3813ec30 con 0x7fab380a1a00 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab340017f0 con 0x7fab380a1a00 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:44:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): dump: 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427303795 0 0) 0x7fab3813dc80 con 0x7fab380a4290 2024-09-16T05:44:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient(hunting): my global_id is 6779 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab1c003530 con 0x7fab380a4290 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 516082508 0 0) 0x7fab3813ec30 con 0x7fab380a1a00 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 >> v1:172.21.15.6:6790/0 conn(0x7fab380a4290 legacy=0x7fab381d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 >> v1:172.21.15.63:6789/0 conn(0x7fab380a37a0 legacy=0x7fab3813b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 monclient: found mon.a 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3813da60 con 0x7fab380a1a00 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.610+0000 7fab2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 5 monclient: authenticate success, global_id 124314 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 10 monclient: _renew_subs 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fab380a82f0 con 0x7fab380a1a00 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab34003620 con 0x7fab380a1a00 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab34003970 con 0x7fab380a1a00 2024-09-16T05:44:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: dump: 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 10 monclient: _renew_subs 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fab3813d7e0 con 0x7fab380a1a00 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2ffff640 4 set_mon_vals no callback set 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fab380008e0 con 0x7fab38140b10 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fab34003b10 con 0x7fab380a1a00 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fab34043c30 con 0x7fab380a1a00 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253192069 0 0) 0x7fab380008e0 con 0x7fab38140b10 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient(hunting): my global_id is 124317 2024-09-16T05:44:05.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab3813ec30 con 0x7fab38140b10 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab3e1f7640 10 monclient: get_auth_request con 0x7fab1c036b60 auth_method 0 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1019797315 0 0) 0x7fab3813ec30 con 0x7fab38140b10 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab380008e0 con 0x7fab38140b10 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab28001750 con 0x7fab38140b10 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: dump: 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.627 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 331855971 0 0) 0x7fab380008e0 con 0x7fab38140b10 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab3813f610 con 0x7fab38140b10 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fab280019f0 con 0x7fab38140b10 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 >> v1:172.21.15.6:6789/0 conn(0x7fab38140b10 legacy=0x7fab38137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.614+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 >> v1:172.21.15.6:6789/0 conn(0x7fab38140b10 legacy=0x7fab38137a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T05:44:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fab381388b0 con 0x7fab3813f7c0 2024-09-16T05:44:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4115851157 0 0) 0x7fab381388b0 con 0x7fab3813f7c0 2024-09-16T05:44:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient(hunting): my global_id is 124320 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab380008e0 con 0x7fab3813f7c0 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 564464753 0 0) 0x7fab380008e0 con 0x7fab3813f7c0 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab381388b0 con 0x7fab3813f7c0 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab28004990 con 0x7fab3813f7c0 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient: dump: 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.632 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2913116294 0 0) 0x7fab381388b0 con 0x7fab3813f7c0 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fab381dcbc0 con 0x7fab3813f7c0 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7fab28004990 con 0x7fab3813f7c0 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 >> v1:172.21.15.6:6789/0 conn(0x7fab3813f7c0 legacy=0x7fab381381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab2e7fc640 1 -- 172.21.15.6:0/2880647178 >> v1:172.21.15.6:6789/0 conn(0x7fab3813f7c0 legacy=0x7fab381381a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:05.633 INFO:teuthology.orchestra.run.smithi006.stdout: "ret": 0, 2024-09-16T05:44:05.634 INFO:teuthology.orchestra.run.smithi006.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T05:44:05.634 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 >> v1:172.21.15.63:6812/3643207211 conn(0x7fab1c036b60 legacy=0x7fab1c039020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 10 monclient: shutdown 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 >> v1:172.21.15.6:6789/0 conn(0x7fab380a1a00 legacy=0x7fab3813b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3e1f7640 1 -- 172.21.15.6:0/2880647178 reap_dead start 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3e1f7640 1 -- 172.21.15.6:0/2880647178 reap_dead start 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 shutdown_connections 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 >> 172.21.15.6:0/2880647178 conn(0x7fab380a58a0 msgr2=0x7fab3809d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 shutdown_connections 2024-09-16T05:44:05.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.622+0000 7fab3fc5a640 1 -- 172.21.15.6:0/2880647178 wait complete. 2024-09-16T05:44:05.640 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:44:05.641 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T05:44:05.641 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:44:05.653 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T05:44:05.654 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:44:05.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:44:05.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 10 monclient: get_monmap_and_config 2024-09-16T05:44:05.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 10 monclient: monmap: 2024-09-16T05:44:05.820 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.820 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.820 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:05.821398+0000 2024-09-16T05:44:05.820 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:05.821398+0000 2024-09-16T05:44:05.821 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.821 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.821 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.821 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:05.821 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:05.821 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 1 Processor -- start 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 1 -- start start 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 10 monclient: init 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 10 monclient(hunting): picked mon.noname-c con 0x7f5ecc072300 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.818+0000 7f5ed43d9640 10 monclient(hunting): picked mon.noname-b con 0x7f5ecc072df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient(hunting): picked mon.noname-a con 0x7f5ecc13d320 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ecc09b330 con 0x7f5ecc13d320 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ecc09b050 con 0x7f5ecc072300 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed2976640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f5ecc13d320 legacy=0x7f5ecc13f710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ecc09b610 con 0x7f5ecc072df0 2024-09-16T05:44:05.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient(hunting): authenticate will time out at 1507.628195s 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1974640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5ecc072df0 0x7f5ecc13bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50120/0 (socket says 172.21.15.6:50120) 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1974640 1 -- 172.21.15.6:0/414142994 learned_addr learned my addr 172.21.15.6:0/414142994 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174835514 0 0) 0x7f5ecc09b610 con 0x7f5ecc072df0 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient(hunting): my global_id is 6782 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5eb8003530 con 0x7f5ecc072df0 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3028490737 0 0) 0x7f5eb8003530 con 0x7f5ecc072df0 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 >> v1:172.21.15.63:6789/0 conn(0x7f5ecc072300 legacy=0x7f5ecc0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 >> v1:172.21.15.6:6789/0 conn(0x7f5ecc13d320 legacy=0x7f5ecc13f710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 monclient: found mon.noname-b 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ecc09c070 con 0x7f5ecc072df0 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec00019e0 con 0x7f5ecc072df0 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 5 monclient: authenticate success, global_id 6782 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: dump: 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ec0001cf0 con 0x7f5ecc072df0 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 1 -- 172.21.15.6:0/414142994 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec0004c30 con 0x7f5ecc072df0 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 4 set_mon_vals no callback set 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1173640 10 monclient: dump: 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.831 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient: shutdown 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- 172.21.15.6:0/414142994 >> v1:172.21.15.6:6790/0 conn(0x7f5ecc072df0 legacy=0x7f5ecc13bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- 172.21.15.6:0/414142994 shutdown_connections 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- 172.21.15.6:0/414142994 >> 172.21.15.6:0/414142994 conn(0x7f5ecc0a1650 msgr2=0x7f5ecc0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- 172.21.15.6:0/414142994 shutdown_connections 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- 172.21.15.6:0/414142994 wait complete. 2024-09-16T05:44:05.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient: build_initial_monmap 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient: monmap: 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 Processor -- start 2024-09-16T05:44:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- start start 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient: init 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient(hunting): picked mon.noname-b con 0x7f5ecc072300 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient(hunting): picked mon.noname-c con 0x7f5ecc072df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient(hunting): picked mon.noname-a con 0x7f5ecc13d320 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ecc13cbf0 con 0x7f5ecc13d320 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ecc1323c0 con 0x7f5ecc072300 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ecc13d080 con 0x7f5ecc072df0 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed43d9640 10 monclient(hunting): authenticate will time out at 1507.631184s 2024-09-16T05:44:05.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed2976640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f5ecc13d320 legacy=0x7f5ecc1d37e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1974640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5ecc072df0 0x7f5ecc131cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:50126/0 (socket says 172.21.15.6:50126) 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ed1974640 1 -- 172.21.15.6:0/1370436241 learned_addr learned my addr 172.21.15.6:0/1370436241 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263686424 0 0) 0x7f5ecc13d080 con 0x7f5ecc072df0 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): my global_id is 6785 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5eb0003530 con 0x7f5ecc072df0 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 788185782 0 0) 0x7f5eb0003530 con 0x7f5ecc072df0 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ecc13d080 con 0x7f5ecc072df0 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec0001800 con 0x7f5ecc072df0 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:05.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): dump: 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:05.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708337770 0 0) 0x7f5ecc1323c0 con 0x7f5ecc072300 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 10 monclient(hunting): my global_id is 134155 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.822+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5eb0003530 con 0x7f5ecc072300 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3783172435 0 0) 0x7f5ecc13d080 con 0x7f5ecc072df0 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 >> v1:172.21.15.63:6789/0 conn(0x7f5ecc072300 legacy=0x7f5ecc1315a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 >> v1:172.21.15.6:6789/0 conn(0x7f5ecc13d320 legacy=0x7f5ecc1d37e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 10 monclient: _finish_hunting 0 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 monclient: found mon.c 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ecc0740c0 con 0x7f5ecc072df0 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 10 monclient: _finish_auth 0 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 10 monclient: _check_auth_tickets 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 5 monclient: authenticate success, global_id 6785 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 10 monclient: _renew_subs 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ecc132ce0 con 0x7f5ecc072df0 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 10 monclient: _renew_subs 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ecc133180 con 0x7f5ecc072df0 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ec0001b40 con 0x7f5ecc072df0 2024-09-16T05:44:05.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec0005390 con 0x7f5ecc072df0 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed0972640 4 set_mon_vals no callback set 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 10 monclient: dump: 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:05.839 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb4005090 con 0x7f5ecc072df0 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5ec0010130 con 0x7f5ecc072df0 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f5ec0045390 con 0x7f5ecc072df0 2024-09-16T05:44:05.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.826+0000 7f5ed2175640 10 monclient: get_auth_request con 0x7f5eb0036aa0 auth_method 0 2024-09-16T05:44:05.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.830+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f5ec0015a20 con 0x7f5ecc072df0 2024-09-16T05:44:05.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.830+0000 7f5ebeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:05.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:05.830+0000 7f5ebeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.054+0000 7f5ed43d9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:44:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.054+0000 7f5ed43d9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.054+0000 7f5ed43d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.054+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5eb4005380 con 0x7f5ecc072df0 2024-09-16T05:44:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.054+0000 7f5ebeffd640 1 -- 172.21.15.6:0/1370436241 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 4168138385) 0x7f5ec001de70 con 0x7f5ecc072df0 2024-09-16T05:44:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.054+0000 7f5ebeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.054+0000 7f5ebeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:06.057 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:44:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.058+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5eb0036aa0 legacy=0x7f5eb0038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.058+0000 7f5ed43d9640 10 monclient: shutdown 2024-09-16T05:44:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.058+0000 7f5ed43d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.058+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 >> v1:172.21.15.6:6790/0 conn(0x7f5ecc072df0 legacy=0x7f5ecc131cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.058+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 shutdown_connections 2024-09-16T05:44:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.058+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 >> 172.21.15.6:0/1370436241 conn(0x7f5ecc0a1650 msgr2=0x7f5ecc0a2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.058+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 shutdown_connections 2024-09-16T05:44:06.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:06.058+0000 7f5ed43d9640 1 -- 172.21.15.6:0/1370436241 wait complete. 2024-09-16T05:44:09.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:44:09.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient: get_monmap_and_config 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient: build_initial_monmap 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient: monmap: 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:09.212369+0000 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:09.212369+0000 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:09.212 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:09.213 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 1 Processor -- start 2024-09-16T05:44:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 1 -- start start 2024-09-16T05:44:09.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient: init 2024-09-16T05:44:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:44:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient(hunting): picked mon.noname-c con 0x7f990009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient(hunting): picked mon.noname-a con 0x7f990009e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient(hunting): picked mon.noname-b con 0x7f99000a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:09.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99000a1370 con 0x7f990009e230 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990009ead0 con 0x7f990009aa30 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990009edb0 con 0x7f99000a0da0 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f99086e5640 10 monclient(hunting): authenticate will time out at 1511.019173s 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f9905c80640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f990009e230 legacy=0x7f99001399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f9906c82640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f99000a0da0 0x7f990013d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:44050/0 (socket says 172.21.15.6:44050) 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f9906c82640 1 -- 172.21.15.6:0/3990714149 learned_addr learned my addr 172.21.15.6:0/3990714149 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 65037471 0 0) 0x7f990009edb0 con 0x7f99000a0da0 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f990547f640 10 monclient(hunting): _negotiate 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f990547f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f990547f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:09.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f990547f640 10 monclient(hunting): my global_id is 6788 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.210+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98ec003530 con 0x7f99000a0da0 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 667590723 0 0) 0x7f990009ead0 con 0x7f990009aa30 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient(hunting): _negotiate 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient(hunting): my global_id is 134158 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f990009edb0 con 0x7f990009aa30 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1149025240 0 0) 0x7f98ec003530 con 0x7f99000a0da0 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 >> v1:172.21.15.63:6789/0 conn(0x7f990009aa30 legacy=0x7f990009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 >> v1:172.21.15.6:6789/0 conn(0x7f990009e230 legacy=0x7f99001399d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: _finish_hunting 0 2024-09-16T05:44:09.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 monclient: found mon.noname-b 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990009dd50 con 0x7f99000a0da0 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: _finish_auth 0 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: _check_auth_tickets 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 5 monclient: authenticate success, global_id 6788 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98f80019a0 con 0x7f99000a0da0 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: dump: 2024-09-16T05:44:09.221 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98f8001c50 con 0x7f99000a0da0 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:09.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 1 -- 172.21.15.6:0/3990714149 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98f8002020 con 0x7f99000a0da0 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f990547f640 10 monclient: dump: 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:09.223 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 4 set_mon_vals no callback set 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: shutdown 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- 172.21.15.6:0/3990714149 >> v1:172.21.15.6:6790/0 conn(0x7f99000a0da0 legacy=0x7f990013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- 172.21.15.6:0/3990714149 shutdown_connections 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- 172.21.15.6:0/3990714149 >> 172.21.15.6:0/3990714149 conn(0x7f99000a1650 msgr2=0x7f99000a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- 172.21.15.6:0/3990714149 shutdown_connections 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- 172.21.15.6:0/3990714149 wait complete. 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: build_initial_monmap 2024-09-16T05:44:09.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: monmap: 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:09.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 Processor -- start 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- start start 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: init 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient(hunting): picked mon.noname-a con 0x7f990009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient(hunting): picked mon.noname-c con 0x7f990009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient(hunting): picked mon.noname-b con 0x7f99000a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f9906481640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f990009aa30 legacy=0x7f9900131ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:09.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990006ea90 con 0x7f990009aa30 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9900135b50 con 0x7f99000a0da0 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990006e540 con 0x7f990009e230 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient(hunting): authenticate will time out at 1511.022588s 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f9905c80640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f990009e230 0x7f99001321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:44056/0 (socket says 172.21.15.6:44056) 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f9905c80640 1 -- 172.21.15.6:0/1787162085 learned_addr learned my addr 172.21.15.6:0/1787162085 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268577871 0 0) 0x7f990006e540 con 0x7f990009e230 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:09.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient(hunting): my global_id is 6791 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98e4003530 con 0x7f990009e230 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 573470175 0 0) 0x7f98e4003530 con 0x7f990009e230 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f990006e540 con 0x7f990009e230 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98fc004160 con 0x7f990009e230 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient(hunting): dump: 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 697119910 0 0) 0x7f990006e540 con 0x7f990009e230 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 >> v1:172.21.15.63:6789/0 conn(0x7f99000a0da0 legacy=0x7f99001328f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 >> v1:172.21.15.6:6789/0 conn(0x7f990009aa30 legacy=0x7f9900131ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 monclient: found mon.c 2024-09-16T05:44:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99000740c0 con 0x7f990009e230 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 5 monclient: authenticate success, global_id 6791 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: _renew_subs 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9900135120 con 0x7f990009e230 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: _renew_subs 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f99001d2710 con 0x7f990009e230 2024-09-16T05:44:09.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98fc0043c0 con 0x7f990009e230 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f98fc004710 con 0x7f990009e230 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f98f2ffd640 10 monclient: dump: 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.214+0000 7f9904c7e640 4 set_mon_vals no callback set 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.218+0000 7f99086e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.218+0000 7f99086e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.218+0000 7f99086e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.218+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99000008e0 con 0x7f990009e230 2024-09-16T05:44:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.218+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f98fc0048b0 con 0x7f990009e230 2024-09-16T05:44:09.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.218+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f98fc044cf0 con 0x7f990009e230 2024-09-16T05:44:09.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.218+0000 7f9906481640 10 monclient: get_auth_request con 0x7f98e4036ab0 auth_method 0 2024-09-16T05:44:09.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.222+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f98fc0152c0 con 0x7f990009e230 2024-09-16T05:44:09.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.222+0000 7f98f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:09.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.222+0000 7f98f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.446+0000 7f99086e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:44:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.446+0000 7f99086e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.446+0000 7f99086e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:09.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.446+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f990009b4a0 con 0x7f990009e230 2024-09-16T05:44:09.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f98f2ffd640 1 -- 172.21.15.6:0/1787162085 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 4112912406) 0x7f98fc014bc0 con 0x7f990009e230 2024-09-16T05:44:09.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f98f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:09.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f98f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:09.450 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:44:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 >> v1:172.21.15.63:6812/3643207211 conn(0x7f98e4036ab0 legacy=0x7f98e4038f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f99086e5640 10 monclient: shutdown 2024-09-16T05:44:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f99086e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 >> v1:172.21.15.6:6790/0 conn(0x7f990009e230 legacy=0x7f99001321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 shutdown_connections 2024-09-16T05:44:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 >> 172.21.15.6:0/1787162085 conn(0x7f99000a1650 msgr2=0x7f99000a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 shutdown_connections 2024-09-16T05:44:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:09.450+0000 7f99086e5640 1 -- 172.21.15.6:0/1787162085 wait complete. 2024-09-16T05:44:12.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:44:12.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient: get_monmap_and_config 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient: build_initial_monmap 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient: monmap: 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:12.602992+0000 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:12.602992+0000 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:12.602 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:12.603 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:12.603 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:12.603 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 1 Processor -- start 2024-09-16T05:44:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 1 -- start start 2024-09-16T05:44:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient: init 2024-09-16T05:44:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:44:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient(hunting): picked mon.noname-c con 0x7fe98c09aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient(hunting): picked mon.noname-b con 0x7fe98c09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c0a1370 con 0x7fe98c0a0da0 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c09ead0 con 0x7fe98c09aa30 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c09edb0 con 0x7fe98c09e230 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe992722640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe98c13d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe994185640 10 monclient(hunting): authenticate will time out at 1514.409774s 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe991720640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe98c09e230 0x7fe98c1399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:44064/0 (socket says 172.21.15.6:44064) 2024-09-16T05:44:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.602+0000 7fe991720640 1 -- 172.21.15.6:0/266905061 learned_addr learned my addr 172.21.15.6:0/266905061 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:12.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:12.806+0000 7fe992722640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe98c13d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:13.210+0000 7fe992722640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe98c13d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:14.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:14.014+0000 7fe992722640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe98c13d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:15.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 10 monclient(hunting): tick 2024-09-16T05:44:15.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:44:15.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T05:44:15.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:44:15.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6790/0 conn(0x7fe98c09e230 legacy=0x7fe98c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:15.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.63:6789/0 conn(0x7fe98c09aa30 legacy=0x7fe98c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe98c13d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fe9700036d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fe97000abc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c09dd50 con 0x7fe98c0a0da0 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c0a1370 con 0x7fe97000abc0 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9700108e0 con 0x7fe9700036d0 2024-09-16T05:44:15.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe97f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:15.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.602+0000 7fe991720640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe9700093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:15.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:15.806+0000 7fe991720640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe9700093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:16.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:16.210+0000 7fe991720640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe9700093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:17.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:17.014+0000 7fe991720640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe9700093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:18.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 10 monclient(hunting): tick 2024-09-16T05:44:18.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6790/0 conn(0x7fe9700036d0 legacy=0x7fe970005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.63:6789/0 conn(0x7fe97000abc0 legacy=0x7fe97000d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe98c0a0da0 legacy=0x7fe9700093f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fe97000f9b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fe970009b20 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fe970009f80 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe970011b90 con 0x7fe97000f9b0 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c09dd50 con 0x7fe970009f80 2024-09-16T05:44:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 1 -- 172.21.15.6:0/266905061 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe970003ab0 con 0x7fe970009b20 2024-09-16T05:44:18.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe97f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:18.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.602+0000 7fe991f21640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe97000f9b0 legacy=0x7fe97000fd90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:18.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:18.806+0000 7fe991f21640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe97000f9b0 legacy=0x7fe97000fd90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:19.210+0000 7fe991f21640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe97000f9b0 legacy=0x7fe97000fd90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:20.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.014+0000 7fe991f21640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe97000f9b0 legacy=0x7fe97000fd90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:20.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2269462153 0 0) 0x7fe970003ab0 con 0x7fe970009b20 2024-09-16T05:44:20.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 10 monclient(hunting): _negotiate 2024-09-16T05:44:20.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:20.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:20.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 10 monclient(hunting): my global_id is 6794 2024-09-16T05:44:20.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe978003530 con 0x7fe970009b20 2024-09-16T05:44:20.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4163748162 0 0) 0x7fe978003530 con 0x7fe970009b20 2024-09-16T05:44:20.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.63:6789/0 conn(0x7fe970009f80 legacy=0x7fe97000a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:20.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6789/0 conn(0x7fe97000f9b0 legacy=0x7fe97000fd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 10 monclient: _finish_hunting 0 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 1 monclient: found mon.noname-b 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe970004ce0 con 0x7fe970009b20 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.678+0000 7fe990f1f640 10 monclient: _finish_auth 0 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 10 monclient: _check_auth_tickets 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe9880015f0 con 0x7fe970009b20 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 5 monclient: authenticate success, global_id 6794 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 10 monclient: dump: 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:20.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe988001850 con 0x7fe970009b20 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 1 -- 172.21.15.6:0/266905061 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe988001c60 con 0x7fe970009b20 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 4 set_mon_vals no callback set 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe990f1f640 10 monclient: dump: 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:20.685 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:20.686 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:20.686 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:20.686 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:20.686 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 10 monclient: shutdown 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 1 -- 172.21.15.6:0/266905061 >> v1:172.21.15.6:6790/0 conn(0x7fe970009b20 legacy=0x7fe9700093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 1 -- 172.21.15.6:0/266905061 shutdown_connections 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 1 -- 172.21.15.6:0/266905061 >> 172.21.15.6:0/266905061 conn(0x7fe98c0a1650 msgr2=0x7fe98c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 1 -- 172.21.15.6:0/266905061 shutdown_connections 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 1 -- 172.21.15.6:0/266905061 wait complete. 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 10 monclient: build_initial_monmap 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 10 monclient: monmap: 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:20.688 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 1 Processor -- start 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 1 -- start start 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.682+0000 7fe994185640 10 monclient: init 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:20.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 10 monclient(hunting): picked mon.noname-c con 0x7fe970009b20 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 10 monclient(hunting): picked mon.noname-b con 0x7fe970009f80 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 10 monclient(hunting): picked mon.noname-a con 0x7fe97000f9b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c0740c0 con 0x7fe97000f9b0 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c09c3a0 con 0x7fe970009f80 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe992722640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fe97000f9b0 legacy=0x7fe98c1d4140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98c124c40 con 0x7fe970009b20 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe994185640 10 monclient(hunting): authenticate will time out at 1522.492856s 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe991f21640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe970009b20 0x7fe98c1cba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60810/0 (socket says 172.21.15.6:60810) 2024-09-16T05:44:20.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.686+0000 7fe991f21640 1 -- 172.21.15.6:0/72832955 learned_addr learned my addr 172.21.15.6:0/72832955 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:20.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3718251758 0 0) 0x7fe98c124c40 con 0x7fe970009b20 2024-09-16T05:44:20.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:20.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient(hunting): my global_id is 6797 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe97000fd90 con 0x7fe970009b20 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 813914893 0 0) 0x7fe97000fd90 con 0x7fe970009b20 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe98c124c40 con 0x7fe970009b20 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe980001880 con 0x7fe970009b20 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient(hunting): dump: 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:20.693 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4015305722 0 0) 0x7fe98c124c40 con 0x7fe970009b20 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 >> v1:172.21.15.63:6789/0 conn(0x7fe970009f80 legacy=0x7fe98c1d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 >> v1:172.21.15.6:6789/0 conn(0x7fe97000f9b0 legacy=0x7fe98c1d4140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:44:20.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 monclient: found mon.c 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98c126c50 con 0x7fe970009b20 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 5 monclient: authenticate success, global_id 6797 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 10 monclient: _renew_subs 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe98c126f30 con 0x7fe970009b20 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 10 monclient: _renew_subs 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe98c123fb0 con 0x7fe970009b20 2024-09-16T05:44:20.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe980003600 con 0x7fe970009b20 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97ffff640 4 set_mon_vals no callback set 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe980003950 con 0x7fe970009b20 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 10 monclient: dump: 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:20.697 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:20.697 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:20.697 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:20.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe980003af0 con 0x7fe970009b20 2024-09-16T05:44:20.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fe980044db0 con 0x7fe970009b20 2024-09-16T05:44:20.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:20.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:20.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:20.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe974005090 con 0x7fe970009b20 2024-09-16T05:44:20.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.690+0000 7fe991720640 10 monclient: get_auth_request con 0x7fe970038750 auth_method 0 2024-09-16T05:44:20.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.698+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe980015320 con 0x7fe970009b20 2024-09-16T05:44:20.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.698+0000 7fe97e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:20.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.698+0000 7fe97e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:20.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.942+0000 7fe994185640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:44:20.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.942+0000 7fe994185640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:20.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.942+0000 7fe994185640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:20.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe974005380 con 0x7fe970009b20 2024-09-16T05:44:20.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe97e7fc640 1 -- 172.21.15.6:0/72832955 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 3877859164) 0x7fe980018fd0 con 0x7fe970009b20 2024-09-16T05:44:20.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe97e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:44:20.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe97e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:20.947 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":42,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:44:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe970038750 legacy=0x7fe97003ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 10 monclient: shutdown 2024-09-16T05:44:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 >> v1:172.21.15.6:6790/0 conn(0x7fe970009b20 legacy=0x7fe98c1cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 shutdown_connections 2024-09-16T05:44:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 >> 172.21.15.6:0/72832955 conn(0x7fe98c0a1650 msgr2=0x7fe98c070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 shutdown_connections 2024-09-16T05:44:20.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:20.946+0000 7fe994185640 1 -- 172.21.15.6:0/72832955 wait complete. 2024-09-16T05:44:20.967 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:44:20.967 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:44:20.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:44:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient: get_monmap_and_config 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient: build_initial_monmap 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient: monmap: 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:21.088250+0000 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:21.088250+0000 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:21.096 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 1 Processor -- start 2024-09-16T05:44:21.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 1 -- start start 2024-09-16T05:44:21.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient: init 2024-09-16T05:44:21.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:21.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:44:21.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient(hunting): picked mon.noname-b con 0x7f989009b2f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:21.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient(hunting): picked mon.noname-a con 0x7f9890136b70 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:21.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient(hunting): picked mon.noname-c con 0x7f989009b6d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:21.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98900a19c0 con 0x7f9890136b70 2024-09-16T05:44:21.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9890099ae0 con 0x7f989009b6d0 2024-09-16T05:44:21.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9890099dc0 con 0x7f989009b2f0 2024-09-16T05:44:21.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient(hunting): authenticate will time out at 1522.894885s 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988e7fc640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f9890136b70 legacy=0x7f98900a12b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f989009b2f0 0x7f98900a0ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60826/0 (socket says 172.21.15.6:60826) 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988effd640 1 -- 172.21.15.6:0/4283753636 learned_addr learned my addr 172.21.15.6:0/4283753636 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 276278774 0 0) 0x7f9890099dc0 con 0x7f989009b2f0 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient(hunting): my global_id is 6800 2024-09-16T05:44:21.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9878003530 con 0x7f989009b2f0 2024-09-16T05:44:21.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2507686691 0 0) 0x7f9878003530 con 0x7f989009b2f0 2024-09-16T05:44:21.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 >> v1:172.21.15.63:6789/0 conn(0x7f989009b6d0 legacy=0x7f989013f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 >> v1:172.21.15.6:6789/0 conn(0x7f9890136b70 legacy=0x7f98900a12b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:44:21.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:44:21.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 monclient: found mon.noname-b 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989009bae0 con 0x7f989009b2f0 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient: _finish_auth 0 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9880001620 con 0x7f989009b2f0 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 5 monclient: authenticate success, global_id 6800 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient: dump: 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.101 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9880001880 con 0x7f989009b2f0 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f988dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.086+0000 7f98957b9640 4 set_mon_vals no callback set 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f988dffb640 1 -- 172.21.15.6:0/4283753636 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9880001cd0 con 0x7f989009b2f0 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f988dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f988dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f988dffb640 10 monclient: dump: 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.102 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: shutdown 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:21.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- 172.21.15.6:0/4283753636 >> v1:172.21.15.6:6790/0 conn(0x7f989009b2f0 legacy=0x7f98900a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- 172.21.15.6:0/4283753636 shutdown_connections 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- 172.21.15.6:0/4283753636 >> 172.21.15.6:0/4283753636 conn(0x7f98900a1c90 msgr2=0x7f98900a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- 172.21.15.6:0/4283753636 shutdown_connections 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- 172.21.15.6:0/4283753636 wait complete. 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: build_initial_monmap 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: monmap: 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:21.104 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 Processor -- start 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- start start 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: init 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient(hunting): picked mon.noname-a con 0x7f989009b2f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient(hunting): picked mon.noname-b con 0x7f989009b6d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient(hunting): picked mon.noname-c con 0x7f9890136b70 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9890131e30 con 0x7f989009b2f0 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9890130dd0 con 0x7f989009b6d0 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9890132110 con 0x7f9890136b70 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f988effd640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f989009b2f0 legacy=0x7f98901cb6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient(hunting): authenticate will time out at 1522.898025s 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f988f7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9890136b70 0x7f98901d39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60834/0 (socket says 172.21.15.6:60834) 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f988f7fe640 1 -- 172.21.15.6:0/2720134271 learned_addr learned my addr 172.21.15.6:0/2720134271 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501077959 0 0) 0x7f9890132110 con 0x7f9890136b70 2024-09-16T05:44:21.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient(hunting): my global_id is 6803 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f987c003530 con 0x7f9890136b70 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 777141349 0 0) 0x7f987c003530 con 0x7f9890136b70 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9890132110 con 0x7f9890136b70 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9888001c80 con 0x7f9890136b70 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient(hunting): dump: 2024-09-16T05:44:21.107 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:21.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 250305804 0 0) 0x7f9890132110 con 0x7f9890136b70 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 >> v1:172.21.15.63:6789/0 conn(0x7f989009b6d0 legacy=0x7f9890133640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 >> v1:172.21.15.6:6789/0 conn(0x7f989009b2f0 legacy=0x7f98901cb6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient: _finish_hunting 0 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 monclient: found mon.c 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9890130910 con 0x7f9890136b70 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient: _finish_auth 0 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient: _check_auth_tickets 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:21.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 5 monclient: authenticate success, global_id 6803 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: _renew_subs 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9890133d50 con 0x7f9890136b70 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: _renew_subs 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f98901d52e0 con 0x7f9890136b70 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98880039a0 con 0x7f9890136b70 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9888003cf0 con 0x7f9890136b70 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f988d7fa640 4 set_mon_vals no callback set 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 10 monclient: dump: 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.111 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f98700065d0 con 0x7f9870002870 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.090+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f9888003f10 con 0x7f9890136b70 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f9888045260 con 0x7f9890136b70 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182775249 0 0) 0x7f98700065d0 con 0x7f9870002870 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient(hunting): my global_id is 6806 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9890132110 con 0x7f9870002870 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 834460082 0 0) 0x7f9890132110 con 0x7f9870002870 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98700065d0 con 0x7f9870002870 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9880001840 con 0x7f9870002870 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f988e7fc640 10 monclient: get_auth_request con 0x7f987c036980 auth_method 0 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:21.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient: dump: 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3348288127 0 0) 0x7f98700065d0 con 0x7f9870002870 2024-09-16T05:44:21.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f987000a160 con 0x7f9870002870 2024-09-16T05:44:21.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9880001b30 con 0x7f9870002870 2024-09-16T05:44:21.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:21.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:21.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 >> v1:172.21.15.6:6790/0 conn(0x7f9870002870 legacy=0x7f9870004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.094+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 >> v1:172.21.15.6:6790/0 conn(0x7f9870002870 legacy=0x7f9870004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98957b9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:44:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9870009e30 con 0x7f98700090f0 2024-09-16T05:44:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3878524510 0 0) 0x7f9870009e30 con 0x7f98700090f0 2024-09-16T05:44:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 10 monclient(hunting): my global_id is 6809 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98700065d0 con 0x7f98700090f0 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2155124518 0 0) 0x7f98700065d0 con 0x7f98700090f0 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9870009e30 con 0x7f98700090f0 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9880004af0 con 0x7f98700090f0 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 10 monclient: dump: 2024-09-16T05:44:21.118 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.119 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1817952512 0 0) 0x7f9870009e30 con 0x7f98700090f0 2024-09-16T05:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.098+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f987000c3a0 con 0x7f98700090f0 2024-09-16T05:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4285 (unknown 0 0 852458518) 0x7f9880006040 con 0x7f98700090f0 2024-09-16T05:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98777fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98777fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 >> v1:172.21.15.6:6790/0 conn(0x7f98700090f0 legacy=0x7f9870009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98777fe640 1 -- 172.21.15.6:0/2720134271 >> v1:172.21.15.6:6790/0 conn(0x7f98700090f0 legacy=0x7f9870009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:21.120 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:21.121 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:21.121 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:21.121 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:44:21.121 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 42, 2024-09-16T05:44:21.121 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 590154, 2024-09-16T05:44:21.121 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:44:21.122 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:44:21.122 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:44:21.122 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.122 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:44:21.122 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:21.122 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:21.123 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:21.124 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:44:21.125 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:44:21.126 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:21.127 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.128 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:21.131 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:44:21.132 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:21.133 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:44:21.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 7 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:21.135 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:21.136 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.136 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.136 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:44:21.136 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:21.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 >> v1:172.21.15.63:6812/3643207211 conn(0x7f987c036980 legacy=0x7f987c038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98957b9640 10 monclient: shutdown 2024-09-16T05:44:21.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98957b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:21.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 >> v1:172.21.15.6:6790/0 conn(0x7f9890136b70 legacy=0x7f98901d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f988f7fe640 1 -- 172.21.15.6:0/2720134271 reap_dead start 2024-09-16T05:44:21.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f988f7fe640 1 -- 172.21.15.6:0/2720134271 reap_dead start 2024-09-16T05:44:21.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 shutdown_connections 2024-09-16T05:44:21.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 >> 172.21.15.6:0/2720134271 conn(0x7f98900a1c90 msgr2=0x7f9890134f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:21.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 shutdown_connections 2024-09-16T05:44:21.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.102+0000 7f98957b9640 1 -- 172.21.15.6:0/2720134271 wait complete. 2024-09-16T05:44:21.138 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:44:21.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:44:21.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:44:21.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient: get_monmap_and_config 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient: build_initial_monmap 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient: monmap: 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:44:21.255878+0000 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:44:21.255878+0000 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:44:21.255 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 1 Processor -- start 2024-09-16T05:44:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 1 -- start start 2024-09-16T05:44:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient: init 2024-09-16T05:44:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:44:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient(hunting): picked mon.noname-a con 0x7fdee809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient(hunting): picked mon.noname-b con 0x7fdee809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:21.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient(hunting): picked mon.noname-c con 0x7fdee80a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee80a1370 con 0x7fdee809aa30 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee809ead0 con 0x7fdee80a0da0 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee809edb0 con 0x7fdee809e230 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient(hunting): authenticate will time out at 1523.062460s 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeede80640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fdee809aa30 legacy=0x7fdee809ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeed67f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdee809e230 0x7fdee81399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60846/0 (socket says 172.21.15.6:60846) 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeed67f640 1 -- 172.21.15.6:0/3459801014 learned_addr learned my addr 172.21.15.6:0/3459801014 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3180122406 0 0) 0x7fdee809edb0 con 0x7fdee809e230 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient(hunting): my global_id is 6812 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fded0003530 con 0x7fdee809e230 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1963109135 0 0) 0x7fded0003530 con 0x7fdee809e230 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 >> v1:172.21.15.63:6789/0 conn(0x7fdee80a0da0 legacy=0x7fdee813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 >> v1:172.21.15.6:6789/0 conn(0x7fdee809aa30 legacy=0x7fdee809ae10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: _finish_hunting 0 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 monclient: found mon.noname-b 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee809dd50 con 0x7fdee809e230 2024-09-16T05:44:21.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: _finish_auth 0 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: _check_auth_tickets 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdee4001620 con 0x7fdee809e230 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 5 monclient: authenticate success, global_id 6812 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: dump: 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.267 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdee4001910 con 0x7fdee809e230 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 1 -- 172.21.15.6:0/3459801014 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdee4001d20 con 0x7fdee809e230 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdeece7e640 10 monclient: dump: 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.269 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.270 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.270 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 10 monclient: get_monmap_and_config success 2024-09-16T05:44:21.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.254+0000 7fdef00e4640 4 set_mon_vals no callback set 2024-09-16T05:44:21.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: shutdown 2024-09-16T05:44:21.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:21.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- 172.21.15.6:0/3459801014 >> v1:172.21.15.6:6790/0 conn(0x7fdee809e230 legacy=0x7fdee81399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- 172.21.15.6:0/3459801014 shutdown_connections 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- 172.21.15.6:0/3459801014 >> 172.21.15.6:0/3459801014 conn(0x7fdee80a1650 msgr2=0x7fdee80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- 172.21.15.6:0/3459801014 shutdown_connections 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- 172.21.15.6:0/3459801014 wait complete. 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: build_initial_monmap 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: monmap: 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:44:21.271 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 Processor -- start 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- start start 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: init 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: _reopen_session rank -1 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient(hunting): picked mon.noname-a con 0x7fdee809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient(hunting): picked mon.noname-b con 0x7fdee809e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient(hunting): picked mon.noname-c con 0x7fdee80a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee81c9030 con 0x7fdee809aa30 2024-09-16T05:44:21.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdeede80640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fdee809aa30 legacy=0x7fdee813d040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee8135330 con 0x7fdee809e230 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee80a0170 con 0x7fdee80a0da0 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient(hunting): _renew_subs 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient(hunting): authenticate will time out at 1523.065769s 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdeee681640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdee80a0da0 0x7fdee81d15c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60848/0 (socket says 172.21.15.6:60848) 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdeee681640 1 -- 172.21.15.6:0/2648047592 learned_addr learned my addr 172.21.15.6:0/2648047592 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2350107077 0 0) 0x7fdee80a0170 con 0x7fdee80a0da0 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): my global_id is 6815 2024-09-16T05:44:21.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fded4003530 con 0x7fdee80a0da0 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 220468858 0 0) 0x7fded4003530 con 0x7fdee80a0da0 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdee80a0170 con 0x7fdee80a0da0 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdee0001950 con 0x7fdee80a0da0 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): dump: 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.274 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2100415637 0 0) 0x7fdee8135330 con 0x7fdee809e230 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient(hunting): my global_id is 134173 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fded4003530 con 0x7fdee809e230 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3111365542 0 0) 0x7fdee80a0170 con 0x7fdee80a0da0 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 >> v1:172.21.15.63:6789/0 conn(0x7fdee809e230 legacy=0x7fdee81cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 >> v1:172.21.15.6:6789/0 conn(0x7fdee809aa30 legacy=0x7fdee813d040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 monclient: found mon.c 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee8138800 con 0x7fdee80a0da0 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient: _finish_auth 0 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 5 monclient: authenticate success, global_id 6815 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: _renew_subs 2024-09-16T05:44:21.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdee8137060 con 0x7fdee80a0da0 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: _renew_subs 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdee813d750 con 0x7fdee80a0da0 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdee0001d40 con 0x7fdee80a0da0 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdee0003af0 con 0x7fdee80a0da0 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 10 monclient: dump: 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.277 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdedffff640 4 set_mon_vals no callback set 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:21.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdec80065d0 con 0x7fdec8002870 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdee0003c90 con 0x7fdee80a0da0 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.258+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fdee0044ee0 con 0x7fdee80a0da0 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdeed67f640 10 monclient: get_auth_request con 0x7fded4036ad0 auth_method 0 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895463278 0 0) 0x7fdec80065d0 con 0x7fdec8002870 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient(hunting): my global_id is 134176 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdee80a0170 con 0x7fdec8002870 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2252944532 0 0) 0x7fdee80a0170 con 0x7fdec8002870 2024-09-16T05:44:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdec80065d0 con 0x7fdec8002870 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fded80042f0 con 0x7fdec8002870 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient: dump: 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.280 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.281 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1263403887 0 0) 0x7fdec80065d0 con 0x7fdec8002870 2024-09-16T05:44:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdec800a160 con 0x7fdec8002870 2024-09-16T05:44:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fded8006700 con 0x7fdec8002870 2024-09-16T05:44:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:44:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:44:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 >> v1:172.21.15.63:6789/0 conn(0x7fdec8002870 legacy=0x7fdec8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.262+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 >> v1:172.21.15.63:6789/0 conn(0x7fdec8002870 legacy=0x7fdec8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:21.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:44:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdec8009e50 con 0x7fdec8009110 2024-09-16T05:44:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2719878822 0 0) 0x7fdec8009e50 con 0x7fdec8009110 2024-09-16T05:44:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:44:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:44:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:44:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient(hunting): my global_id is 134179 2024-09-16T05:44:21.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdec80065d0 con 0x7fdec8009110 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 54200493 0 0) 0x7fdec80065d0 con 0x7fdec8009110 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdec8009e50 con 0x7fdec8009110 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fded80049b0 con 0x7fdec8009110 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient: dump: 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.284 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2943621310 0 0) 0x7fdec8009e50 con 0x7fdec8009110 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdec800c3e0 con 0x7fdec8009110 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3952 (unknown 0 0 2997100261) 0x7fded8006970 con 0x7fdec8009110 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:44:21.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:44:21.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 >> v1:172.21.15.63:6789/0 conn(0x7fdec8009110 legacy=0x7fdec80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdede7fc640 1 -- 172.21.15.6:0/2648047592 >> v1:172.21.15.63:6789/0 conn(0x7fdec8009110 legacy=0x7fdec80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:44:21.286 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:44:21.286 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:21.286 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:21.286 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:44:21.286 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 42, 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 18863, 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:21.287 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:44:21.288 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:44:21.290 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:44:21.291 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:44:21.292 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:44:21.293 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:44:21.294 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:44:21.295 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:44:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:44:21.298 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 >> v1:172.21.15.63:6812/3643207211 conn(0x7fded4036ad0 legacy=0x7fded4038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 10 monclient: shutdown 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 >> v1:172.21.15.6:6790/0 conn(0x7fdee80a0da0 legacy=0x7fdee81d15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdeee681640 1 -- 172.21.15.6:0/2648047592 reap_dead start 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdeee681640 1 -- 172.21.15.6:0/2648047592 reap_dead start 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 shutdown_connections 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 >> 172.21.15.6:0/2648047592 conn(0x7fdee80a1650 msgr2=0x7fdee80a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:44:21.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 shutdown_connections 2024-09-16T05:44:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:44:21.270+0000 7fdef00e4640 1 -- 172.21.15.6:0/2648047592 wait complete. 2024-09-16T05:44:21.301 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:44:21.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:45:51.302 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:45:51.303 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T05:45:51.303 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T05:45:51.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T05:45:51.307 INFO:tasks.ceph.mon.a:Started 2024-09-16T05:45:51.307 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:45:51.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:45:51.401 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:51.401+0000 7f36eb991e00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 396 -> 0 (sync_force? mon rebuild?) 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient: get_monmap_and_config 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient: build_initial_monmap 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient: monmap: 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:51.444561+0000 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:51.444561+0000 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.444 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:51.444 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:51.444 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:51.444 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 1 Processor -- start 2024-09-16T05:45:51.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 1 -- start start 2024-09-16T05:45:51.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient: init 2024-09-16T05:45:51.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:51.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:45:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient(hunting): picked mon.noname-c con 0x7f158c0a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient(hunting): picked mon.noname-a con 0x7f158c0a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient(hunting): picked mon.noname-b con 0x7f158c0a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158c06e520 con 0x7f158c0a4290 2024-09-16T05:45:51.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158c0a0780 con 0x7f158c0a37a0 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158c0a4c70 con 0x7f158c0a1a00 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 10 monclient(hunting): authenticate will time out at 1613.250987s 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158ad9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f158c0a1a00 0x7f158c13f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57454/0 (socket says 172.21.15.6:57454) 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158ad9d640 1 -- 172.21.15.6:0/2620971073 learned_addr learned my addr 172.21.15.6:0/2620971073 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3374414099 0 0) 0x7f158c0a4c70 con 0x7f158c0a1a00 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient(hunting): my global_id is 6818 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1574003530 con 0x7f158c0a1a00 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 743933116 0 0) 0x7f1574003530 con 0x7f158c0a1a00 2024-09-16T05:45:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 >> v1:172.21.15.63:6789/0 conn(0x7f158c0a37a0 legacy=0x7f158c0a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 >> v1:172.21.15.6:6789/0 conn(0x7f158c0a4290 legacy=0x7f158c13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient: _finish_hunting 0 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 1 monclient: found mon.noname-b 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158c0a65f0 con 0x7f158c0a1a00 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient: _finish_auth 0 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient: _check_auth_tickets 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:51.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f15840019a0 con 0x7f158c0a1a00 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f158959a640 10 monclient: dump: 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.456 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 5 monclient: authenticate success, global_id 6818 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.441+0000 7f1590d49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1584001d00 con 0x7f158c0a1a00 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f158959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 4 set_mon_vals no callback set 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f158959a640 1 -- 172.21.15.6:0/2620971073 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f15840020d0 con 0x7f158c0a1a00 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f158959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f158959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f158959a640 10 monclient: dump: 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.457 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: shutdown 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- 172.21.15.6:0/2620971073 >> v1:172.21.15.6:6790/0 conn(0x7f158c0a1a00 legacy=0x7f158c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- 172.21.15.6:0/2620971073 shutdown_connections 2024-09-16T05:45:51.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- 172.21.15.6:0/2620971073 >> 172.21.15.6:0/2620971073 conn(0x7f158c0a58a0 msgr2=0x7f158c0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- 172.21.15.6:0/2620971073 shutdown_connections 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- 172.21.15.6:0/2620971073 wait complete. 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: build_initial_monmap 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: monmap: 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:51.459 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 Processor -- start 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- start start 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: init 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient(hunting): picked mon.noname-a con 0x7f158c0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient(hunting): picked mon.noname-c con 0x7f158c0a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient(hunting): picked mon.noname-b con 0x7f158c0a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158c13d890 con 0x7f158c0a1a00 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158c13f2a0 con 0x7f158c0a4290 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158c13fba0 con 0x7f158c0a37a0 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient(hunting): authenticate will time out at 1613.254426s 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1589d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f158c0a37a0 0x7f158c13bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57466/0 (socket says 172.21.15.6:57466) 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f158a59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f158c0a1a00 0x7f158c13b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38864/0 (socket says 172.21.15.6:38864) 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1589d9b640 1 -- 172.21.15.6:0/3220264354 learned_addr learned my addr 172.21.15.6:0/3220264354 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1795891019 0 0) 0x7f158c13fba0 con 0x7f158c0a37a0 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): my global_id is 6821 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f156c003560 con 0x7f158c0a37a0 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3104806190 0 0) 0x7f156c003560 con 0x7f158c0a37a0 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f158c13fba0 con 0x7f158c0a37a0 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f15800018c0 con 0x7f158c0a37a0 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): dump: 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.462 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2408989648 0 0) 0x7f158c13f2a0 con 0x7f158c0a4290 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient(hunting): my global_id is 134185 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f156c003560 con 0x7f158c0a4290 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2263705059 0 0) 0x7f158c13fba0 con 0x7f158c0a37a0 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 >> v1:172.21.15.63:6789/0 conn(0x7f158c0a4290 legacy=0x7f158c1d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 >> v1:172.21.15.6:6789/0 conn(0x7f158c0a1a00 legacy=0x7f158c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient: _finish_hunting 0 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 monclient: found mon.c 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158c0a8340 con 0x7f158c0a37a0 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient: _finish_auth 0 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient: _check_auth_tickets 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 5 monclient: authenticate success, global_id 6821 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: _renew_subs 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f158c0a0100 con 0x7f158c0a37a0 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: _renew_subs 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f158c13f6c0 con 0x7f158c0a37a0 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1580001cf0 con 0x7f158c0a37a0 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1580003af0 con 0x7f158c0a37a0 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f157affd640 10 monclient: dump: 2024-09-16T05:45:51.465 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.445+0000 7f1588d99640 4 set_mon_vals no callback set 2024-09-16T05:45:51.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.449+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1580003c90 con 0x7f158c0a37a0 2024-09-16T05:45:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.449+0000 7f1590d49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.449+0000 7f1590d49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.449+0000 7f1590d49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.449+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f1580044f00 con 0x7f158c0a37a0 2024-09-16T05:45:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.449+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1570005090 con 0x7f158c0a37a0 2024-09-16T05:45:51.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.449+0000 7f158a59c640 10 monclient: get_auth_request con 0x7f156c036c10 auth_method 0 2024-09-16T05:45:51.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.453+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1580015530 con 0x7f158c0a37a0 2024-09-16T05:45:51.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.453+0000 7f157affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:51.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.453+0000 7f157affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:51.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f1590d49640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:45:51.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f1590d49640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:45:51.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f1590d49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1570005380 con 0x7f158c0a37a0 2024-09-16T05:45:51.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f157affd640 1 -- 172.21.15.6:0/3220264354 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1454480338) 0x7f1580014e30 con 0x7f158c0a37a0 2024-09-16T05:45:51.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f157affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:45:51.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f157affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:51.691 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:45:51.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 >> v1:172.21.15.63:6812/3643207211 conn(0x7f156c036c10 legacy=0x7f156c0390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f1590d49640 10 monclient: shutdown 2024-09-16T05:45:51.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f1590d49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:51.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.689+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 >> v1:172.21.15.6:6790/0 conn(0x7f158c0a37a0 legacy=0x7f158c13bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.693+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 shutdown_connections 2024-09-16T05:45:51.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.693+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 >> 172.21.15.6:0/3220264354 conn(0x7f158c0a58a0 msgr2=0x7f158c0a73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:51.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.693+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 shutdown_connections 2024-09-16T05:45:51.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.693+0000 7f1590d49640 1 -- 172.21.15.6:0/3220264354 wait complete. 2024-09-16T05:45:51.708 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:45:51.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:45:51.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:45:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: get_monmap_and_config 2024-09-16T05:45:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: build_initial_monmap 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: monmap: 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:51.822988+0000 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:51.822988+0000 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:51.822 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 Processor -- start 2024-09-16T05:45:51.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- start start 2024-09-16T05:45:51.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: init 2024-09-16T05:45:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:45:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient(hunting): picked mon.noname-a con 0x7fb02009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient(hunting): picked mon.noname-c con 0x7fb02009e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient(hunting): picked mon.noname-b con 0x7fb0200a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0200a1370 con 0x7fb02009aa30 2024-09-16T05:45:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb02009ead0 con 0x7fb02009e230 2024-09-16T05:45:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb02009edb0 con 0x7fb0200a0da0 2024-09-16T05:45:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient(hunting): authenticate will time out at 1613.629509s 2024-09-16T05:45:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb026b03640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb0200a0da0 0x7fb02013d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57470/0 (socket says 172.21.15.6:57470) 2024-09-16T05:45:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb026302640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb02009aa30 0x7fb02009ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38872/0 (socket says 172.21.15.6:38872) 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb026b03640 1 -- 172.21.15.6:0/1758271102 learned_addr learned my addr 172.21.15.6:0/1758271102 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161099142 0 0) 0x7fb0200a1370 con 0x7fb02009aa30 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient(hunting): my global_id is 144099 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb00c003530 con 0x7fb02009aa30 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315106002 0 0) 0x7fb02009edb0 con 0x7fb0200a0da0 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient(hunting): my global_id is 6824 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0200a1370 con 0x7fb0200a0da0 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 771817718 0 0) 0x7fb00c003530 con 0x7fb02009aa30 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 >> v1:172.21.15.6:6790/0 conn(0x7fb0200a0da0 legacy=0x7fb02013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 >> v1:172.21.15.63:6789/0 conn(0x7fb02009e230 legacy=0x7fb0201399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: _finish_hunting 0 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 monclient: found mon.noname-a 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02009dd50 con 0x7fb02009aa30 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: _finish_auth 0 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: _check_auth_tickets 2024-09-16T05:45:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb010001620 con 0x7fb02009aa30 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: dump: 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 5 monclient: authenticate success, global_id 144099 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb010001810 con 0x7fb02009aa30 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 1 -- 172.21.15.6:0/1758271102 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb010001c20 con 0x7fb02009aa30 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 4 set_mon_vals no callback set 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb025300640 10 monclient: dump: 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: shutdown 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- 172.21.15.6:0/1758271102 >> v1:172.21.15.6:6789/0 conn(0x7fb02009aa30 legacy=0x7fb02009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- 172.21.15.6:0/1758271102 shutdown_connections 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- 172.21.15.6:0/1758271102 >> 172.21.15.6:0/1758271102 conn(0x7fb0200a1650 msgr2=0x7fb0200a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- 172.21.15.6:0/1758271102 shutdown_connections 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- 172.21.15.6:0/1758271102 wait complete. 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: build_initial_monmap 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: monmap: 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:51.838 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 Processor -- start 2024-09-16T05:45:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 1 -- start start 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.821+0000 7fb028566640 10 monclient: init 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient(hunting): picked mon.noname-a con 0x7fb02009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient(hunting): picked mon.noname-b con 0x7fb02009e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient(hunting): picked mon.noname-c con 0x7fb0200a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb02013b610 con 0x7fb02009aa30 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb02013c740 con 0x7fb02009e230 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb02009c990 con 0x7fb0200a0da0 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient(hunting): authenticate will time out at 1613.632462s 2024-09-16T05:45:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb026b03640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb0200a0da0 0x7fb0201d3930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57480/0 (socket says 172.21.15.6:57480) 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb026302640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb02009aa30 0x7fb02009bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38882/0 (socket says 172.21.15.6:38882) 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb026b03640 1 -- 172.21.15.6:0/4197152905 learned_addr learned my addr 172.21.15.6:0/4197152905 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482457060 0 0) 0x7fb02009c990 con 0x7fb0200a0da0 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): my global_id is 6827 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb004003530 con 0x7fb0200a0da0 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1243254927 0 0) 0x7fb02009dd50 con 0x7fb02009aa30 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): my global_id is 144102 2024-09-16T05:45:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb02009c990 con 0x7fb02009aa30 2024-09-16T05:45:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 617576165 0 0) 0x7fb004003530 con 0x7fb0200a0da0 2024-09-16T05:45:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb02009dd50 con 0x7fb0200a0da0 2024-09-16T05:45:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb0180016b0 con 0x7fb0200a0da0 2024-09-16T05:45:51.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): dump: 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.842 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.843 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 736971100 0 0) 0x7fb02009c990 con 0x7fb02009aa30 2024-09-16T05:45:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb004003530 con 0x7fb02009aa30 2024-09-16T05:45:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb010001ba0 con 0x7fb02009aa30 2024-09-16T05:45:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:51.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): dump: 2024-09-16T05:45:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.844 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3988796728 0 0) 0x7fb02013c740 con 0x7fb02009e230 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): my global_id is 134191 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb02009c990 con 0x7fb02009e230 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2315303915 0 0) 0x7fb02009dd50 con 0x7fb0200a0da0 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 >> v1:172.21.15.63:6789/0 conn(0x7fb02009e230 legacy=0x7fb02009c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 >> v1:172.21.15.6:6789/0 conn(0x7fb02009aa30 legacy=0x7fb02009bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 monclient: found mon.c 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0200740c0 con 0x7fb0200a0da0 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: _finish_auth 0 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 5 monclient: authenticate success, global_id 6827 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient: _renew_subs 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb02009d7d0 con 0x7fb0200a0da0 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient: _renew_subs 2024-09-16T05:45:51.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:51.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb02013cd00 con 0x7fb0200a0da0 2024-09-16T05:45:51.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb018001960 con 0x7fb0200a0da0 2024-09-16T05:45:51.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:51.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb018004fc0 con 0x7fb0200a0da0 2024-09-16T05:45:51.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:51.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:51.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb024aff640 4 set_mon_vals no callback set 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: dump: 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:51.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb0080065d0 con 0x7fb008002870 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb01800fd60 con 0x7fb0200a0da0 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fb0180451c0 con 0x7fb0200a0da0 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872591510 0 0) 0x7fb0080065d0 con 0x7fb008002870 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient(hunting): my global_id is 144105 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb02009dd50 con 0x7fb008002870 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb025b01640 10 monclient: get_auth_request con 0x7fb004036bf0 auth_method 0 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2974658499 0 0) 0x7fb02009dd50 con 0x7fb008002870 2024-09-16T05:45:51.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0080065d0 con 0x7fb008002870 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb010004890 con 0x7fb008002870 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: dump: 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.850 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.851 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.851 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.851 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.851 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.851 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1267093432 0 0) 0x7fb0080065d0 con 0x7fb008002870 2024-09-16T05:45:51.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb00800a160 con 0x7fb008002870 2024-09-16T05:45:51.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb010004b40 con 0x7fb008002870 2024-09-16T05:45:51.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:51.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:51.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 >> v1:172.21.15.6:6789/0 conn(0x7fb008002870 legacy=0x7fb008004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.825+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 >> v1:172.21.15.6:6789/0 conn(0x7fb008002870 legacy=0x7fb008004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:45:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb008009e30 con 0x7fb0080090f0 2024-09-16T05:45:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818579932 0 0) 0x7fb008009e30 con 0x7fb0080090f0 2024-09-16T05:45:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:51.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient(hunting): my global_id is 144108 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0080065d0 con 0x7fb0080090f0 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3674368873 0 0) 0x7fb0080065d0 con 0x7fb0080090f0 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb008009e30 con 0x7fb0080090f0 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb010005100 con 0x7fb0080090f0 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient: dump: 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:51.855 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4103482713 0 0) 0x7fb008009e30 con 0x7fb0080090f0 2024-09-16T05:45:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb00800c3a0 con 0x7fb0080090f0 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3961 (unknown 0 0 1845350468) 0x7fb0100063b0 con 0x7fb0080090f0 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 >> v1:172.21.15.6:6789/0 conn(0x7fb0080090f0 legacy=0x7fb008009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb016ffd640 1 -- 172.21.15.6:0/4197152905 >> v1:172.21.15.6:6789/0 conn(0x7fb0080090f0 legacy=0x7fb008009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:45:51.857 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 46, 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 447, 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:51.858 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:51.859 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:51.860 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:45:51.861 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:45:51.862 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:45:51.863 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:45:51.863 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:51.863 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:45:51.863 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:51.863 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:51.863 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:45:51.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:51.865 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:51.867 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:45:51.868 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:51.869 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:51.870 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:51.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.870 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:45:51.870 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:45:51.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:51.870 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:45:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:51.871 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:51.872 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:51.872 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:51.872 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:51.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:51.872 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:45:51.872 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:45:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb004036bf0 legacy=0x7fb0040390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 10 monclient: shutdown 2024-09-16T05:45:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 >> v1:172.21.15.6:6790/0 conn(0x7fb0200a0da0 legacy=0x7fb0201d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb026b03640 1 -- 172.21.15.6:0/4197152905 reap_dead start 2024-09-16T05:45:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb026b03640 1 -- 172.21.15.6:0/4197152905 reap_dead start 2024-09-16T05:45:51.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 shutdown_connections 2024-09-16T05:45:51.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 >> 172.21.15.6:0/4197152905 conn(0x7fb0200a1650 msgr2=0x7fb02013de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:51.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 shutdown_connections 2024-09-16T05:45:51.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.833+0000 7fb028566640 1 -- 172.21.15.6:0/4197152905 wait complete. 2024-09-16T05:45:51.875 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:45:51.875 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:45:51.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:45:51.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient: get_monmap_and_config 2024-09-16T05:45:51.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient: build_initial_monmap 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient: monmap: 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:51.999216+0000 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:51.999216+0000 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:51.998 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 Processor -- start 2024-09-16T05:45:52.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- start start 2024-09-16T05:45:52.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient: init 2024-09-16T05:45:52.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient(hunting): picked mon.noname-a con 0x7fd898134880 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient(hunting): picked mon.noname-c con 0x7fd8980a0f20 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient(hunting): picked mon.noname-b con 0x7fd89813ae70 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8980a3180 con 0x7fd898134880 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89809f2a0 con 0x7fd8980a0f20 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8980a0260 con 0x7fd89813ae70 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient(hunting): authenticate will time out at 1613.805980s 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8977fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd898134880 0x7fd898134c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38904/0 (socket says 172.21.15.6:38904) 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8977fe640 1 -- 172.21.15.6:0/4263366674 learned_addr learned my addr 172.21.15.6:0/4263366674 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163759497 0 0) 0x7fd8900014a0 con 0x7fd898134880 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): my global_id is 144111 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd884003530 con 0x7fd898134880 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276256049 0 0) 0x7fd8980a0260 con 0x7fd89813ae70 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): my global_id is 6830 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8900014a0 con 0x7fd89813ae70 2024-09-16T05:45:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545973520 0 0) 0x7fd89809f2a0 con 0x7fd8980a0f20 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient(hunting): my global_id is 134194 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8980a0260 con 0x7fd8980a0f20 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4095537913 0 0) 0x7fd884003530 con 0x7fd898134880 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 >> v1:172.21.15.6:6790/0 conn(0x7fd89813ae70 legacy=0x7fd89813d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 >> v1:172.21.15.63:6789/0 conn(0x7fd8980a0f20 legacy=0x7fd898139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: _finish_hunting 0 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 monclient: found mon.noname-a 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:52.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8980a0540 con 0x7fd898134880 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: _finish_auth 0 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: _check_auth_tickets 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd890001b70 con 0x7fd898134880 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: dump: 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 5 monclient: authenticate success, global_id 144111 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd890001e20 con 0x7fd898134880 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 1 -- 172.21.15.6:0/4263366674 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd890004c30 con 0x7fd898134880 2024-09-16T05:45:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd8967fc640 10 monclient: dump: 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 4 set_mon_vals no callback set 2024-09-16T05:45:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 10 monclient: shutdown 2024-09-16T05:45:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- 172.21.15.6:0/4263366674 >> v1:172.21.15.6:6789/0 conn(0x7fd898134880 legacy=0x7fd898134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- 172.21.15.6:0/4263366674 shutdown_connections 2024-09-16T05:45:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- 172.21.15.6:0/4263366674 >> 172.21.15.6:0/4263366674 conn(0x7fd8980a37e0 msgr2=0x7fd8980a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- 172.21.15.6:0/4263366674 shutdown_connections 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:51.997+0000 7fd89dbc9640 1 -- 172.21.15.6:0/4263366674 wait complete. 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: build_initial_monmap 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: monmap: 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 1 Processor -- start 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 1 -- start start 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: init 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient(hunting): picked mon.noname-c con 0x7fd8980a0f20 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient(hunting): picked mon.noname-a con 0x7fd898134880 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient(hunting): picked mon.noname-b con 0x7fd89813ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89806eae0 con 0x7fd898134880 2024-09-16T05:45:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89813b3f0 con 0x7fd89813ae70 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8981cc830 con 0x7fd8980a0f20 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient(hunting): authenticate will time out at 1613.809149s 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd8977fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd8980a0f20 0x7fd8981cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57494/0 (socket says 172.21.15.6:57494) 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd8977fe640 1 -- 172.21.15.6:0/544713597 learned_addr learned my addr 172.21.15.6:0/544713597 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3981216472 0 0) 0x7fd89806eae0 con 0x7fd898134880 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): my global_id is 144114 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd878003530 con 0x7fd898134880 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2245742280 0 0) 0x7fd8981cc830 con 0x7fd8980a0f20 2024-09-16T05:45:52.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): my global_id is 6833 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89806eae0 con 0x7fd8980a0f20 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1355351068 0 0) 0x7fd878003530 con 0x7fd898134880 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8981cc830 con 0x7fd898134880 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8800018c0 con 0x7fd898134880 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:52.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): dump: 2024-09-16T05:45:52.018 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.018 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.018 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.018 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.018 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.018 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.019 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4144262869 0 0) 0x7fd89806eae0 con 0x7fd8980a0f20 2024-09-16T05:45:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd878003530 con 0x7fd8980a0f20 2024-09-16T05:45:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd890004b70 con 0x7fd8980a0f20 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): dump: 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1754657821 0 0) 0x7fd89813b3f0 con 0x7fd89813ae70 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): my global_id is 134197 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89806eae0 con 0x7fd89813ae70 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 642942196 0 0) 0x7fd8981cc830 con 0x7fd898134880 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 >> v1:172.21.15.6:6790/0 conn(0x7fd8980a0f20 legacy=0x7fd8981cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 >> v1:172.21.15.63:6789/0 conn(0x7fd89813ae70 legacy=0x7fd8981d3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 monclient: found mon.a 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89813e5a0 con 0x7fd898134880 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: _finish_auth 0 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 5 monclient: authenticate success, global_id 144114 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: _renew_subs 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd89806dfe0 con 0x7fd898134880 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: _renew_subs 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd89813a160 con 0x7fd898134880 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd880003620 con 0x7fd898134880 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd880003970 con 0x7fd898134880 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: dump: 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd895ffb640 4 set_mon_vals no callback set 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd87c0065d0 con 0x7fd87c002870 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 769815205 0 0) 0x7fd87c0065d0 con 0x7fd87c002870 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient(hunting): my global_id is 6836 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8981cc830 con 0x7fd87c002870 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1034493240 0 0) 0x7fd8981cc830 con 0x7fd87c002870 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd87c0065d0 con 0x7fd87c002870 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd890001e20 con 0x7fd87c002870 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 10 monclient: dump: 2024-09-16T05:45:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd88000e710 con 0x7fd898134880 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fd880043b40 con 0x7fd898134880 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 810063256 0 0) 0x7fd87c0065d0 con 0x7fd87c002870 2024-09-16T05:45:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.001+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd87c00a160 con 0x7fd87c002870 2024-09-16T05:45:52.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.005+0000 7fd897fff640 10 monclient: get_auth_request con 0x7fd878036a30 auth_method 0 2024-09-16T05:45:52.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.005+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd8900075b0 con 0x7fd87c002870 2024-09-16T05:45:52.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.005+0000 7fd88ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:52.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.005+0000 7fd88ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:52.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.005+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 >> v1:172.21.15.6:6790/0 conn(0x7fd87c002870 legacy=0x7fd87c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.005+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 >> v1:172.21.15.6:6790/0 conn(0x7fd87c002870 legacy=0x7fd87c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:52.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd87c009e60 con 0x7fd87c009120 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 866877845 0 0) 0x7fd87c009e60 con 0x7fd87c009120 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient(hunting): my global_id is 6839 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd87c0065d0 con 0x7fd87c009120 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1032621685 0 0) 0x7fd87c0065d0 con 0x7fd87c009120 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd87c009e60 con 0x7fd87c009120 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd890005bd0 con 0x7fd87c009120 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:52.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient: dump: 2024-09-16T05:45:52.032 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.032 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1274049156 0 0) 0x7fd87c009e60 con 0x7fd87c009120 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd87c00c3e0 con 0x7fd87c009120 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 707248622) 0x7fd890006ed0 con 0x7fd87c009120 2024-09-16T05:45:52.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:45:52.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:52.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 >> v1:172.21.15.6:6790/0 conn(0x7fd87c009120 legacy=0x7fd87c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd88ffff640 1 -- 172.21.15.6:0/544713597 >> v1:172.21.15.6:6790/0 conn(0x7fd87c009120 legacy=0x7fd87c0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:52.034 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:45:52.034 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:45:52.034 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:45:52.034 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:45:52.034 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 46, 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 681066, 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:52.035 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:45:52.036 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:52.037 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:45:52.038 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:45:52.039 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:45:52.039 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:45:52.039 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:45:52.039 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:45:52.039 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:45:52.039 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:52.039 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:45:52.039 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:52.040 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:52.040 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:52.040 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:52.040 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:52.040 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:52.040 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:52.040 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:52.041 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:45:52.042 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:52.043 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:52.043 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:52.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.043 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:45:52.043 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:45:52.044 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:52.044 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:52.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.044 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:52.044 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:45:52.044 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:52.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.045 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.045 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:45:52.045 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:45:52.045 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:52.046 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:52.046 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:52.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.046 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:45:52.046 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:45:52.047 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:52.047 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:52.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.047 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:52.047 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:45:52.047 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:52.048 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.048 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.049 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:45:52.049 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:45:52.049 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:52.049 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:52.049 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:52.049 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.050 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.050 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:45:52.050 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:45:52.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.050 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:52.051 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:52.051 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:52.051 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.052 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.052 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.053 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:45:52.054 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:45:52.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd878036a30 legacy=0x7fd878038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 10 monclient: shutdown 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 >> v1:172.21.15.6:6789/0 conn(0x7fd898134880 legacy=0x7fd8981cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd897fff640 1 -- 172.21.15.6:0/544713597 reap_dead start 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd897fff640 1 -- 172.21.15.6:0/544713597 reap_dead start 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 shutdown_connections 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 >> 172.21.15.6:0/544713597 conn(0x7fd8980a37e0 msgr2=0x7fd8980a4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 shutdown_connections 2024-09-16T05:45:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.009+0000 7fd89dbc9640 1 -- 172.21.15.6:0/544713597 wait complete. 2024-09-16T05:45:52.057 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:45:52.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:45:52.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.169+0000 7f5ecd655640 10 monclient: get_monmap_and_config 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.169+0000 7f5ecd655640 10 monclient: build_initial_monmap 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.169+0000 7f5ecd655640 10 monclient: monmap: 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:52.173741+0000 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:52.173741+0000 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.173 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:52.174 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.169+0000 7f5ecd655640 1 Processor -- start 2024-09-16T05:45:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.169+0000 7f5ecd655640 1 -- start start 2024-09-16T05:45:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.169+0000 7f5ecd655640 10 monclient: init 2024-09-16T05:45:52.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): picked mon.noname-b con 0x7f5ec809aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): picked mon.noname-c con 0x7f5ec809e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): picked mon.noname-a con 0x7f5ec80a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ec80a1370 con 0x7f5ec80a0da0 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ec809ead0 con 0x7f5ec809e230 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ec809edb0 con 0x7f5ec809aa30 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): authenticate will time out at 1613.980309s 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec6ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5ec809aa30 0x7f5ec809ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57508/0 (socket says 172.21.15.6:57508) 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec6ffd640 1 -- 172.21.15.6:0/2033286710 learned_addr learned my addr 172.21.15.6:0/2033286710 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ec80a0da0 0x7f5ec813d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38926/0 (socket says 172.21.15.6:38926) 2024-09-16T05:45:52.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1790608308 0 0) 0x7f5ec80a1370 con 0x7f5ec80a0da0 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient(hunting): my global_id is 144117 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5eb4003530 con 0x7f5ec80a0da0 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2149861564 0 0) 0x7f5ec809edb0 con 0x7f5ec809aa30 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient(hunting): my global_id is 6842 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ec80a1370 con 0x7f5ec809aa30 2024-09-16T05:45:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1125054964 0 0) 0x7f5eb4003530 con 0x7f5ec80a0da0 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 >> v1:172.21.15.6:6790/0 conn(0x7f5ec809aa30 legacy=0x7f5ec809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 >> v1:172.21.15.63:6789/0 conn(0x7f5ec809e230 legacy=0x7f5ec81399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 monclient: found mon.noname-a 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec809dd50 con 0x7f5ec80a0da0 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 5 monclient: authenticate success, global_id 144117 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec0001620 con 0x7f5ec80a0da0 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: dump: 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ec0001810 con 0x7f5ec80a0da0 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 4 set_mon_vals no callback set 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 1 -- 172.21.15.6:0/2033286710 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec0001c20 con 0x7f5ec80a0da0 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec5ffb640 10 monclient: dump: 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: shutdown 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- 172.21.15.6:0/2033286710 >> v1:172.21.15.6:6789/0 conn(0x7f5ec80a0da0 legacy=0x7f5ec813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- 172.21.15.6:0/2033286710 shutdown_connections 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- 172.21.15.6:0/2033286710 >> 172.21.15.6:0/2033286710 conn(0x7f5ec80a1650 msgr2=0x7f5ec80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- 172.21.15.6:0/2033286710 shutdown_connections 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- 172.21.15.6:0/2033286710 wait complete. 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: build_initial_monmap 2024-09-16T05:45:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: monmap: 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 Processor -- start 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- start start 2024-09-16T05:45:52.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: init 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): picked mon.noname-a con 0x7f5ec809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): picked mon.noname-c con 0x7f5ec809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): picked mon.noname-b con 0x7f5ec80a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ec811e660 con 0x7f5ec809aa30 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ec811cf30 con 0x7f5ec80a0da0 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ec811bfc0 con 0x7f5ec809e230 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ecd655640 10 monclient(hunting): authenticate will time out at 1613.983181s 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec67fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5ec809e230 0x7f5ec811b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57518/0 (socket says 172.21.15.6:57518) 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ec67fc640 1 -- 172.21.15.6:0/1865784834 learned_addr learned my addr 172.21.15.6:0/1865784834 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:52.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1801646725 0 0) 0x7f5ec811bfc0 con 0x7f5ec809e230 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): my global_id is 6845 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ea8003530 con 0x7f5ec809e230 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723206145 0 0) 0x7f5ec811e660 con 0x7f5ec809aa30 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): my global_id is 144120 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ec811bfc0 con 0x7f5ec809aa30 2024-09-16T05:45:52.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 377709778 0 0) 0x7f5ea8003530 con 0x7f5ec809e230 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ec811e660 con 0x7f5ec809e230 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5eb8001cc0 con 0x7f5ec809e230 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): dump: 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.190 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 596600539 0 0) 0x7f5ec811bfc0 con 0x7f5ec809aa30 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ea8003530 con 0x7f5ec809aa30 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5eb0001180 con 0x7f5ec809aa30 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): dump: 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.191 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 868424773 0 0) 0x7f5ec811cf30 con 0x7f5ec80a0da0 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 10 monclient(hunting): my global_id is 134203 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.173+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ec811bfc0 con 0x7f5ec80a0da0 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 925755927 0 0) 0x7f5ea8003530 con 0x7f5ec809aa30 2024-09-16T05:45:52.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 >> v1:172.21.15.6:6790/0 conn(0x7f5ec809e230 legacy=0x7f5ec811b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 >> v1:172.21.15.63:6789/0 conn(0x7f5ec80a0da0 legacy=0x7f5ec81d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 monclient: found mon.a 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec806e5d0 con 0x7f5ec809aa30 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: _finish_auth 0 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 5 monclient: authenticate success, global_id 144120 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 10 monclient: _renew_subs 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:52.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec80740c0 con 0x7f5ec809aa30 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 10 monclient: _renew_subs 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ec811e940 con 0x7f5ec809aa30 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5eb00047f0 con 0x7f5ec809aa30 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5eb0004b40 con 0x7f5ec809aa30 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ec57fa640 4 set_mon_vals no callback set 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: dump: 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.194 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5eac0065d0 con 0x7f5eac002870 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5eb000f8e0 con 0x7f5ec809aa30 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f5eb0044d20 con 0x7f5ec809aa30 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ec77fe640 10 monclient: get_auth_request con 0x7f5ea8036bf0 auth_method 0 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973935321 0 0) 0x7f5eac0065d0 con 0x7f5eac002870 2024-09-16T05:45:52.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient(hunting): my global_id is 134206 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ea8003530 con 0x7f5eac002870 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2600772934 0 0) 0x7f5ea8003530 con 0x7f5eac002870 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5eac0065d0 con 0x7f5eac002870 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5eb8003db0 con 0x7f5eac002870 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: dump: 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.196 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1281011538 0 0) 0x7f5eac0065d0 con 0x7f5eac002870 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5eac00a160 con 0x7f5eac002870 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5eb8005b50 con 0x7f5eac002870 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 >> v1:172.21.15.63:6789/0 conn(0x7f5eac002870 legacy=0x7f5eac004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.177+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 >> v1:172.21.15.63:6789/0 conn(0x7f5eac002870 legacy=0x7f5eac004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5eac009e30 con 0x7f5eac0090f0 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292009615 0 0) 0x7f5eac009e30 con 0x7f5eac0090f0 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient(hunting): my global_id is 134209 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5eac0065d0 con 0x7f5eac0090f0 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 443955502 0 0) 0x7f5eac0065d0 con 0x7f5eac0090f0 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5eac009e30 con 0x7f5eac0090f0 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5eb8004430 con 0x7f5eac0090f0 2024-09-16T05:45:52.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient: dump: 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2983268966 0 0) 0x7f5eac009e30 con 0x7f5eac0090f0 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5eac00c3a0 con 0x7f5eac0090f0 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3962 (unknown 0 0 3748896039) 0x7f5eb8004290 con 0x7f5eac0090f0 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 >> v1:172.21.15.63:6789/0 conn(0x7f5eac0090f0 legacy=0x7f5eac009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ebf7fe640 1 -- 172.21.15.6:0/1865784834 >> v1:172.21.15.63:6789/0 conn(0x7f5eac0090f0 legacy=0x7f5eac009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:45:52.201 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 46, 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 109778, 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:52.202 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:52.203 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:45:52.204 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:45:52.205 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.206 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.207 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.208 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:45:52.209 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:52.210 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:45:52.211 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:52.212 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:52.212 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:52.212 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:52.212 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:52.212 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:52.212 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:52.212 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:45:52.212 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:45:52.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5ea8036bf0 legacy=0x7f5ea80390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 10 monclient: shutdown 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 >> v1:172.21.15.6:6789/0 conn(0x7f5ec809aa30 legacy=0x7f5ec811b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ec77fe640 1 -- 172.21.15.6:0/1865784834 reap_dead start 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ec77fe640 1 -- 172.21.15.6:0/1865784834 reap_dead start 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 shutdown_connections 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 >> 172.21.15.6:0/1865784834 conn(0x7f5ec80a1650 msgr2=0x7f5ec813de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 shutdown_connections 2024-09-16T05:45:52.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.185+0000 7f5ecd655640 1 -- 172.21.15.6:0/1865784834 wait complete. 2024-09-16T05:45:52.214 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:45:52.215 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:45:52.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:45:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.325+0000 7fb1dd844640 10 monclient: get_monmap_and_config 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.325+0000 7fb1dd844640 10 monclient: build_initial_monmap 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.325+0000 7fb1dd844640 10 monclient: monmap: 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:52.330129+0000 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:52.330129+0000 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:52.329 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 Processor -- start 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- start start 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient: init 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d812dcf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d80a1680 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient(hunting): picked mon.noname-a con 0x7fb1d809ba40 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d809f170 con 0x7fb1d809ba40 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d809f6c0 con 0x7fb1d80a1680 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d809f3e0 con 0x7fb1d812dcf0 2024-09-16T05:45:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient(hunting): authenticate will time out at 1614.136959s 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d6ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb1d812dcf0 0x7fb1d812e0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57530/0 (socket says 172.21.15.6:57530) 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d6ffd640 1 -- 172.21.15.6:0/1304532134 learned_addr learned my addr 172.21.15.6:0/1304532134 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1260212562 0 0) 0x7fb1d809f170 con 0x7fb1d809ba40 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): my global_id is 144123 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1c0003530 con 0x7fb1d809ba40 2024-09-16T05:45:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2966891436 0 0) 0x7fb1d809f3e0 con 0x7fb1d812dcf0 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): my global_id is 6848 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d809f170 con 0x7fb1d812dcf0 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3216590018 0 0) 0x7fb1d809f6c0 con 0x7fb1d80a1680 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient(hunting): my global_id is 134212 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d809f3e0 con 0x7fb1d80a1680 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3154720188 0 0) 0x7fb1d809f170 con 0x7fb1d812dcf0 2024-09-16T05:45:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 >> v1:172.21.15.63:6789/0 conn(0x7fb1d80a1680 legacy=0x7fb1d8139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d77fe640 1 -- 172.21.15.6:0/1304532134 >> v1:172.21.15.6:6789/0 conn(0x7fb1d809ba40 legacy=0x7fb1d813ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d77fe640 1 -- 172.21.15.6:0/1304532134 >> v1:172.21.15.6:6789/0 conn(0x7fb1d809ba40 legacy=0x7fb1d813ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 >> v1:172.21.15.6:6789/0 conn(0x7fb1d809ba40 legacy=0x7fb1d813ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d77fe640 1 --1- 172.21.15.6:0/1304532134 >> v1:172.21.15.6:6789/0 conn(0x7fb1d809ba40 0x7fb1d813ce40 :-1 s=OPENED pgs=917 cs=1 l=1).handle_message read tag failed 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d77fe640 1 --1- 172.21.15.6:0/1304532134 >> v1:172.21.15.6:6789/0 conn(0x7fb1d809ba40 0x7fb1d813ce40 :-1 s=OPENED pgs=917 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 monclient: found mon.noname-b 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d812e7e0 con 0x7fb1d812dcf0 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:45:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: ms_handle_reset stray mon v1:172.21.15.6:6789/0 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d0001980 con 0x7fb1d812dcf0 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: dump: 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 5 monclient: authenticate success, global_id 6848 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1d0001c70 con 0x7fb1d812dcf0 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 1 -- 172.21.15.6:0/1304532134 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d0002040 con 0x7fb1d812dcf0 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1d5ffb640 10 monclient: dump: 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 4 set_mon_vals no callback set 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient: shutdown 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- 172.21.15.6:0/1304532134 >> v1:172.21.15.6:6790/0 conn(0x7fb1d812dcf0 legacy=0x7fb1d812e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- 172.21.15.6:0/1304532134 shutdown_connections 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- 172.21.15.6:0/1304532134 >> 172.21.15.6:0/1304532134 conn(0x7fb1d80a1f00 msgr2=0x7fb1d80a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- 172.21.15.6:0/1304532134 shutdown_connections 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- 172.21.15.6:0/1304532134 wait complete. 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient: build_initial_monmap 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient: monmap: 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 Processor -- start 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 1 -- start start 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.329+0000 7fb1dd844640 10 monclient: init 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient(hunting): picked mon.noname-c con 0x7fb1d809ba40 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient(hunting): picked mon.noname-b con 0x7fb1d812dcf0 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient(hunting): picked mon.noname-a con 0x7fb1d8073440 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d8137700 con 0x7fb1d8073440 2024-09-16T05:45:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d81379e0 con 0x7fb1d812dcf0 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d813e0b0 con 0x7fb1d809ba40 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient(hunting): authenticate will time out at 1614.140291s 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1d6ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb1d809ba40 0x7fb1d8072620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57534/0 (socket says 172.21.15.6:57534) 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1d6ffd640 1 -- 172.21.15.6:0/1817565206 learned_addr learned my addr 172.21.15.6:0/1817565206 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677425165 0 0) 0x7fb1d813e0b0 con 0x7fb1d809ba40 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): my global_id is 6851 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1c4003530 con 0x7fb1d809ba40 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2991286599 0 0) 0x7fb1d8137700 con 0x7fb1d8073440 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): my global_id is 144126 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d813e0b0 con 0x7fb1d8073440 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3375110892 0 0) 0x7fb1c4003530 con 0x7fb1d809ba40 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1d8137700 con 0x7fb1d809ba40 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d0001940 con 0x7fb1d809ba40 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): dump: 2024-09-16T05:45:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.348 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 36153688 0 0) 0x7fb1d813e0b0 con 0x7fb1d8073440 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1c4003530 con 0x7fb1d8073440 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1c8001910 con 0x7fb1d8073440 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): dump: 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 168497829 0 0) 0x7fb1d81379e0 con 0x7fb1d812dcf0 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient(hunting): my global_id is 134215 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d813e0b0 con 0x7fb1d812dcf0 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 559547941 0 0) 0x7fb1d8137700 con 0x7fb1d809ba40 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 >> v1:172.21.15.63:6789/0 conn(0x7fb1d812dcf0 legacy=0x7fb1d8072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 >> v1:172.21.15.6:6789/0 conn(0x7fb1d8073440 legacy=0x7fb1d81d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient: _finish_hunting 0 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 monclient: found mon.c 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d80a4760 con 0x7fb1d809ba40 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient: _finish_auth 0 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient: _check_auth_tickets 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 5 monclient: authenticate success, global_id 6851 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient: _renew_subs 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d809c420 con 0x7fb1d809ba40 2024-09-16T05:45:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient: _renew_subs 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1d8137140 con 0x7fb1d809ba40 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1d0001c70 con 0x7fb1d809ba40 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d0005210 con 0x7fb1d809ba40 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1d57fa640 4 set_mon_vals no callback set 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 10 monclient: dump: 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:52.352 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b8005090 con 0x7fb1d809ba40 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb1d0005430 con 0x7fb1d809ba40 2024-09-16T05:45:52.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fb1d00453f0 con 0x7fb1d809ba40 2024-09-16T05:45:52.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.333+0000 7fb1d67fc640 10 monclient: get_auth_request con 0x7fb1c4036bd0 auth_method 0 2024-09-16T05:45:52.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.337+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fb1d0015b80 con 0x7fb1d809ba40 2024-09-16T05:45:52.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.337+0000 7fb1bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:52.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.337+0000 7fb1bffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.561+0000 7fb1dd844640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:45:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.561+0000 7fb1dd844640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:45:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.561+0000 7fb1dd844640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.561+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb1b8005b90 con 0x7fb1d809ba40 2024-09-16T05:45:52.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.565+0000 7fb1bffff640 1 -- 172.21.15.6:0/1817565206 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fb1d0015480 con 0x7fb1d809ba40 2024-09-16T05:45:52.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.565+0000 7fb1bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:45:52.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.565+0000 7fb1bffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:52.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.565+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.565+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=10,kv=11,logm=56} crc {auth=1143493741,config=3541143443,health=881976245,kv=2361287224,logm=2413510537}) 2024-09-16T05:45:52.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.565+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=1,health=11,kv=10,logm=55} crc {auth=2676643498,config=555582061,health=3778872965,kv=897214189,logm=903600391}) 2024-09-16T05:45:52.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.565+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.565+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=10,kv=11,logm=56} crc {auth=1143493741,config=3541143443,health=881976245,kv=2361287224,logm=2413510537}) 2024-09-16T05:45:52.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.565+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=10,kv=11,logm=53} crc {auth=2791312942,config=3541143443,health=1212683227,kv=2361287224,logm=359633835}) 2024-09-16T05:45:52.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.569+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb1c4036bd0 legacy=0x7fb1c4039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.569+0000 7fb1dd844640 10 monclient: shutdown 2024-09-16T05:45:52.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.569+0000 7fb1dd844640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:52.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.569+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 >> v1:172.21.15.6:6790/0 conn(0x7fb1d809ba40 legacy=0x7fb1d8072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:52.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158394045}) 2024-09-16T05:45:52.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4291512526}) 2024-09-16T05:45:52.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.569+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 shutdown_connections 2024-09-16T05:45:52.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.569+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 >> 172.21.15.6:0/1817565206 conn(0x7fb1d80a1f00 msgr2=0x7fb1d812f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:52.571 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158394045}) 2024-09-16T05:45:52.571 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3369045577}) 2024-09-16T05:45:52.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.569+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 shutdown_connections 2024-09-16T05:45:52.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:52.569+0000 7fb1dd844640 1 -- 172.21.15.6:0/1817565206 wait complete. 2024-09-16T05:45:52.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187502858}) 2024-09-16T05:45:52.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315341716}) 2024-09-16T05:45:52.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187502858}) 2024-09-16T05:45:52.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.569+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419444186}) 2024-09-16T05:45:52.573 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177067815}) 2024-09-16T05:45:52.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=764994421}) 2024-09-16T05:45:52.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177067815}) 2024-09-16T05:45:52.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=299741423}) 2024-09-16T05:45:52.575 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.575 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407743739}) 2024-09-16T05:45:52.575 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417233596}) 2024-09-16T05:45:52.575 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.575 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407743739}) 2024-09-16T05:45:52.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2937706282}) 2024-09-16T05:45:52.577 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.577 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=33952557}) 2024-09-16T05:45:52.577 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.573+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3965511028}) 2024-09-16T05:45:52.577 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.577 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=33952557}) 2024-09-16T05:45:52.577 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3452167683}) 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642665813}) 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=622082466}) 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642665813}) 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=869814439}) 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167813040}) 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820930680}) 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167813040}) 2024-09-16T05:45:52.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.577+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343724507}) 2024-09-16T05:45:52.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145972015}) 2024-09-16T05:45:52.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2643323193}) 2024-09-16T05:45:52.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145972015}) 2024-09-16T05:45:52.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385343269}) 2024-09-16T05:45:52.584 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.584 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302927737}) 2024-09-16T05:45:52.584 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489655056}) 2024-09-16T05:45:52.584 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.584 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3302927737}) 2024-09-16T05:45:52.584 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.581+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136338321}) 2024-09-16T05:45:52.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48015}) 2024-09-16T05:45:52.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504979856}) 2024-09-16T05:45:52.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48015}) 2024-09-16T05:45:52.586 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=455760124}) 2024-09-16T05:45:52.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:45:52.587 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843660622}) 2024-09-16T05:45:52.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12756574}) 2024-09-16T05:45:52.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843660622}) 2024-09-16T05:45:52.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.585+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923677434}) 2024-09-16T05:45:52.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.589+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.589+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852787922}) 2024-09-16T05:45:52.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.589+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555280659}) 2024-09-16T05:45:52.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.589+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.589+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1852787922}) 2024-09-16T05:45:52.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.589+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252496268}) 2024-09-16T05:45:52.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237775337}) 2024-09-16T05:45:52.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4261583215}) 2024-09-16T05:45:52.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237775337}) 2024-09-16T05:45:52.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233331943}) 2024-09-16T05:45:52.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310755845}) 2024-09-16T05:45:52.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1917627281}) 2024-09-16T05:45:52.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310755845}) 2024-09-16T05:45:52.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.601+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291030613}) 2024-09-16T05:45:52.606 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259479533}) 2024-09-16T05:45:52.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992263178}) 2024-09-16T05:45:52.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259479533}) 2024-09-16T05:45:52.607 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3983131665}) 2024-09-16T05:45:52.608 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.608 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481473540}) 2024-09-16T05:45:52.608 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751841154}) 2024-09-16T05:45:52.608 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481473540}) 2024-09-16T05:45:52.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.605+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1577823984}) 2024-09-16T05:45:52.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1239317904}) 2024-09-16T05:45:52.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013911495}) 2024-09-16T05:45:52.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1239317904}) 2024-09-16T05:45:52.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=562770242}) 2024-09-16T05:45:52.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314297049}) 2024-09-16T05:45:52.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334510017}) 2024-09-16T05:45:52.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314297049}) 2024-09-16T05:45:52.612 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.609+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3974402021}) 2024-09-16T05:45:52.614 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.614 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784011440}) 2024-09-16T05:45:52.614 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996800371}) 2024-09-16T05:45:52.614 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.614 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784011440}) 2024-09-16T05:45:52.614 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2263767888}) 2024-09-16T05:45:52.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719879711}) 2024-09-16T05:45:52.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874219383}) 2024-09-16T05:45:52.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.616 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719879711}) 2024-09-16T05:45:52.616 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989087417}) 2024-09-16T05:45:52.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649085679}) 2024-09-16T05:45:52.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1972520654}) 2024-09-16T05:45:52.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.613+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649085679}) 2024-09-16T05:45:52.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103025936}) 2024-09-16T05:45:52.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654469134}) 2024-09-16T05:45:52.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270004784}) 2024-09-16T05:45:52.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654469134}) 2024-09-16T05:45:52.620 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556396906}) 2024-09-16T05:45:52.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1010436318}) 2024-09-16T05:45:52.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3504286500}) 2024-09-16T05:45:52.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1010436318}) 2024-09-16T05:45:52.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.617+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110670731}) 2024-09-16T05:45:52.623 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.623 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542303749}) 2024-09-16T05:45:52.623 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1630704666}) 2024-09-16T05:45:52.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542303749}) 2024-09-16T05:45:52.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408495652}) 2024-09-16T05:45:52.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=895933777}) 2024-09-16T05:45:52.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420445368}) 2024-09-16T05:45:52.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=895933777}) 2024-09-16T05:45:52.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.621+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=54199020}) 2024-09-16T05:45:52.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583945137}) 2024-09-16T05:45:52.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1964074156}) 2024-09-16T05:45:52.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583945137}) 2024-09-16T05:45:52.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3801836736}) 2024-09-16T05:45:52.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407989434}) 2024-09-16T05:45:52.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=91352242}) 2024-09-16T05:45:52.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407989434}) 2024-09-16T05:45:52.629 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.625+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409371645}) 2024-09-16T05:45:52.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=149876657}) 2024-09-16T05:45:52.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2163824671}) 2024-09-16T05:45:52.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=149876657}) 2024-09-16T05:45:52.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=209666620}) 2024-09-16T05:45:52.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829851171}) 2024-09-16T05:45:52.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=554585630}) 2024-09-16T05:45:52.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829851171}) 2024-09-16T05:45:52.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.629+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1203605840}) 2024-09-16T05:45:52.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167328185}) 2024-09-16T05:45:52.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1464612798}) 2024-09-16T05:45:52.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167328185}) 2024-09-16T05:45:52.634 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305502443}) 2024-09-16T05:45:52.635 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.635 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114531629}) 2024-09-16T05:45:52.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3357936194}) 2024-09-16T05:45:52.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114531629}) 2024-09-16T05:45:52.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.633+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=793499757}) 2024-09-16T05:45:52.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676185111}) 2024-09-16T05:45:52.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343446595}) 2024-09-16T05:45:52.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676185111}) 2024-09-16T05:45:52.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4049100456}) 2024-09-16T05:45:52.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240512406}) 2024-09-16T05:45:52.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457931151}) 2024-09-16T05:45:52.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240512406}) 2024-09-16T05:45:52.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.637+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=388859330}) 2024-09-16T05:45:52.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023696101}) 2024-09-16T05:45:52.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4205937023}) 2024-09-16T05:45:52.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023696101}) 2024-09-16T05:45:52.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2169818122}) 2024-09-16T05:45:52.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748910668}) 2024-09-16T05:45:52.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=994400964}) 2024-09-16T05:45:52.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748910668}) 2024-09-16T05:45:52.644 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.641+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2559557080}) 2024-09-16T05:45:52.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266791589}) 2024-09-16T05:45:52.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1890084765}) 2024-09-16T05:45:52.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266791589}) 2024-09-16T05:45:52.646 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613635939}) 2024-09-16T05:45:52.647 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244865197}) 2024-09-16T05:45:52.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1299253111}) 2024-09-16T05:45:52.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244865197}) 2024-09-16T05:45:52.648 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.645+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2569909858}) 2024-09-16T05:45:52.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.649 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281848467}) 2024-09-16T05:45:52.650 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3861998883}) 2024-09-16T05:45:52.650 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.650 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281848467}) 2024-09-16T05:45:52.650 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=83370247}) 2024-09-16T05:45:52.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123411221}) 2024-09-16T05:45:52.651 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3520365596}) 2024-09-16T05:45:52.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123411221}) 2024-09-16T05:45:52.652 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=631676078}) 2024-09-16T05:45:52.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.649+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3816043172}) 2024-09-16T05:45:52.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851912612}) 2024-09-16T05:45:52.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3816043172}) 2024-09-16T05:45:52.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266913020}) 2024-09-16T05:45:52.655 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.655 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200251040}) 2024-09-16T05:45:52.655 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1220966879}) 2024-09-16T05:45:52.655 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.655 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200251040}) 2024-09-16T05:45:52.655 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.653+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3290530792}) 2024-09-16T05:45:52.657 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151055396}) 2024-09-16T05:45:52.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020091919}) 2024-09-16T05:45:52.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151055396}) 2024-09-16T05:45:52.658 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2550002636}) 2024-09-16T05:45:52.659 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.659 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983505422}) 2024-09-16T05:45:52.659 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1099420265}) 2024-09-16T05:45:52.659 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983505422}) 2024-09-16T05:45:52.660 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.657+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1895010130}) 2024-09-16T05:45:52.661 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.661 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878935842}) 2024-09-16T05:45:52.661 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678786744}) 2024-09-16T05:45:52.661 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878935842}) 2024-09-16T05:45:52.662 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669016128}) 2024-09-16T05:45:52.663 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.663 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2368570333}) 2024-09-16T05:45:52.663 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4108929537}) 2024-09-16T05:45:52.663 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.663 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2368570333}) 2024-09-16T05:45:52.663 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.661+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392269575}) 2024-09-16T05:45:52.665 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.665 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=2,osd_metadata=2} crc {logm=3335137031,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2214786167,mon_config_key=3041421033,monmap=3140514009,osd_metadata=1495888149}) 2024-09-16T05:45:52.666 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=54,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6} crc {logm=4003682939,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2214786167,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417}) 2024-09-16T05:45:52.666 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.666 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=2,osd_metadata=2} crc {logm=3335137031,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2214786167,mon_config_key=3041421033,monmap=3140514009,osd_metadata=1495888149}) 2024-09-16T05:45:52.666 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=52,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=3594357525,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2214786167,mon_config_key=1983187342,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393}) 2024-09-16T05:45:52.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=2,osdmap=94} crc {osd_metadata=1847309184,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=2388383861}) 2024-09-16T05:45:52.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=93} crc {osd_metadata=3695383094,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=3278943164}) 2024-09-16T05:45:52.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=2,osdmap=94} crc {osd_metadata=1847309184,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=2388383861}) 2024-09-16T05:45:52.668 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.665+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=2,osdmap=94} crc {osd_metadata=147319237,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1549613996}) 2024-09-16T05:45:52.671 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.669+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.671 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.669+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=973638763}) 2024-09-16T05:45:52.671 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.669+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3734152899}) 2024-09-16T05:45:52.671 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.669+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.671 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.669+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=973638763}) 2024-09-16T05:45:52.671 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.669+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2903444882}) 2024-09-16T05:45:52.674 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.673+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.674 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.673+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=994607771}) 2024-09-16T05:45:52.674 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.673+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1729628314}) 2024-09-16T05:45:52.674 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.673+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.675 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.673+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=994607771}) 2024-09-16T05:45:52.675 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.673+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2707201791}) 2024-09-16T05:45:52.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.681+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.684 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.681+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=4025277506}) 2024-09-16T05:45:52.685 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.681+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=3261150659}) 2024-09-16T05:45:52.685 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.681+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:45:52.685 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.681+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=4025277506}) 2024-09-16T05:45:52.685 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:45:52.681+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=65} crc {osdmap=18745782}) 2024-09-16T05:45:53.588 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:45:53.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:45:53.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient: get_monmap_and_config 2024-09-16T05:45:53.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient: build_initial_monmap 2024-09-16T05:45:53.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient: monmap: 2024-09-16T05:45:53.720 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:53.720 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:53.720 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:53.720580+0000 2024-09-16T05:45:53.720 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:53.720580+0000 2024-09-16T05:45:53.720 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:53.720 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:53.720 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:53.721 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:53.721 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:53.721 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:53.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 1 Processor -- start 2024-09-16T05:45:53.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 1 -- start start 2024-09-16T05:45:53.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient: init 2024-09-16T05:45:53.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient(hunting): picked mon.noname-b con 0x7ff75c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient(hunting): picked mon.noname-c con 0x7ff75c09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient(hunting): picked mon.noname-a con 0x7ff75c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75c0a1370 con 0x7ff75c0a0da0 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75c09ead0 con 0x7ff75c09e230 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75c09edb0 con 0x7ff75c09aa30 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff763824640 10 monclient(hunting): authenticate will time out at 1615.527323s 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff7615c0640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff75c09aa30 0x7ff75c09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57544/0 (socket says 172.21.15.6:57544) 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff761dc1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff75c0a0da0 0x7ff75c13d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38984/0 (socket says 172.21.15.6:38984) 2024-09-16T05:45:53.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff7615c0640 1 -- 172.21.15.6:0/420042842 learned_addr learned my addr 172.21.15.6:0/420042842 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2649632980 0 0) 0x7ff75c0a1370 con 0x7ff75c0a0da0 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff753fff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff753fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.717+0000 7ff753fff640 10 monclient(hunting): my global_id is 144129 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff748003530 con 0x7ff75c0a0da0 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676620687 0 0) 0x7ff75c09edb0 con 0x7ff75c09aa30 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient(hunting): _negotiate 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:53.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient(hunting): my global_id is 6854 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff75c0a1370 con 0x7ff75c09aa30 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 820411959 0 0) 0x7ff748003530 con 0x7ff75c0a0da0 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 >> v1:172.21.15.6:6790/0 conn(0x7ff75c09aa30 legacy=0x7ff75c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 >> v1:172.21.15.63:6789/0 conn(0x7ff75c09e230 legacy=0x7ff75c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: _finish_hunting 0 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 monclient: found mon.noname-a 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:45:53.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75c09dd50 con 0x7ff75c0a0da0 2024-09-16T05:45:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: _finish_auth 0 2024-09-16T05:45:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: _check_auth_tickets 2024-09-16T05:45:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff754001620 con 0x7ff75c0a0da0 2024-09-16T05:45:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 5 monclient: authenticate success, global_id 144129 2024-09-16T05:45:53.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: dump: 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:53.739 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff7540018d0 con 0x7ff75c0a0da0 2024-09-16T05:45:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 1 -- 172.21.15.6:0/420042842 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff754001ce0 con 0x7ff75c0a0da0 2024-09-16T05:45:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:53.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:53.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff753fff640 10 monclient: dump: 2024-09-16T05:45:53.741 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:53.741 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:53.741 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.741 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.741 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:53.741 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:53.741 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 4 set_mon_vals no callback set 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: shutdown 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- 172.21.15.6:0/420042842 >> v1:172.21.15.6:6789/0 conn(0x7ff75c0a0da0 legacy=0x7ff75c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- 172.21.15.6:0/420042842 shutdown_connections 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- 172.21.15.6:0/420042842 >> 172.21.15.6:0/420042842 conn(0x7ff75c0a1650 msgr2=0x7ff75c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- 172.21.15.6:0/420042842 shutdown_connections 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- 172.21.15.6:0/420042842 wait complete. 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: build_initial_monmap 2024-09-16T05:45:53.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: monmap: 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 Processor -- start 2024-09-16T05:45:53.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- start start 2024-09-16T05:45:53.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: init 2024-09-16T05:45:53.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:53.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:45:53.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient(hunting): picked mon.noname-a con 0x7ff75c09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:53.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient(hunting): picked mon.noname-b con 0x7ff75c09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:53.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient(hunting): picked mon.noname-c con 0x7ff75c0a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:53.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75c13ccd0 con 0x7ff75c09aa30 2024-09-16T05:45:53.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75c09c990 con 0x7ff75c09e230 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75c13ae80 con 0x7ff75c0a0da0 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient(hunting): authenticate will time out at 1615.530767s 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff7615c0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff75c09aa30 0x7ff75c09bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38986/0 (socket says 172.21.15.6:38986) 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff7615c0640 1 -- 172.21.15.6:0/1290900092 learned_addr learned my addr 172.21.15.6:0/1290900092 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2990129390 0 0) 0x7ff75c13ae80 con 0x7ff75c0a0da0 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): my global_id is 6857 2024-09-16T05:45:53.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff740003530 con 0x7ff75c0a0da0 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4232549246 0 0) 0x7ff75c13ccd0 con 0x7ff75c09aa30 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): my global_id is 144132 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff75c13ae80 con 0x7ff75c09aa30 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3275833971 0 0) 0x7ff75c09c990 con 0x7ff75c09e230 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): my global_id is 134221 2024-09-16T05:45:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff75c13ccd0 con 0x7ff75c09e230 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2156683073 0 0) 0x7ff740003530 con 0x7ff75c0a0da0 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff75c09c990 con 0x7ff75c0a0da0 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff754001a80 con 0x7ff75c0a0da0 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): dump: 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:53.747 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2982022558 0 0) 0x7ff75c13ae80 con 0x7ff75c09aa30 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff740003530 con 0x7ff75c09aa30 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff758001b90 con 0x7ff75c09aa30 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient(hunting): dump: 2024-09-16T05:45:53.748 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2983121554 0 0) 0x7ff75c09c990 con 0x7ff75c0a0da0 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 >> v1:172.21.15.63:6789/0 conn(0x7ff75c09e230 legacy=0x7ff75c09c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:53.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 >> v1:172.21.15.6:6789/0 conn(0x7ff75c09aa30 legacy=0x7ff75c09bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 monclient: found mon.c 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75c13b760 con 0x7ff75c0a0da0 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient: _finish_auth 0 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:53.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 5 monclient: authenticate success, global_id 6857 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: _renew_subs 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff75c0740c0 con 0x7ff75c0a0da0 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: _renew_subs 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.721+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff75c13d0f0 con 0x7ff75c0a0da0 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff7540018d0 con 0x7ff75c0a0da0 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff751ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff752ffd640 4 set_mon_vals no callback set 2024-09-16T05:45:53.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff754004db0 con 0x7ff75c0a0da0 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff751ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff751ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff751ffb640 10 monclient: dump: 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:53.752 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff763824640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff763824640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff763824640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff73c005090 con 0x7ff75c0a0da0 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff75400fb50 con 0x7ff75c0a0da0 2024-09-16T05:45:53.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7ff754044da0 con 0x7ff75c0a0da0 2024-09-16T05:45:53.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.725+0000 7ff7615c0640 10 monclient: get_auth_request con 0x7ff740036b40 auth_method 0 2024-09-16T05:45:53.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.729+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ff754015430 con 0x7ff75c0a0da0 2024-09-16T05:45:53.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.729+0000 7ff751ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:53.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.729+0000 7ff751ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:53.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.953+0000 7ff763824640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:45:53.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:45:53.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:53.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff73c005380 con 0x7ff75c0a0da0 2024-09-16T05:45:53.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff751ffb640 1 -- 172.21.15.6:0/1290900092 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 1372026756) 0x7ff754014d30 con 0x7ff75c0a0da0 2024-09-16T05:45:53.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff751ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:45:53.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff751ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:53.959 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:45:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff740036b40 legacy=0x7ff740039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 10 monclient: shutdown 2024-09-16T05:45:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 >> v1:172.21.15.6:6790/0 conn(0x7ff75c0a0da0 legacy=0x7ff75c1d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 shutdown_connections 2024-09-16T05:45:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 >> 172.21.15.6:0/1290900092 conn(0x7ff75c0a1650 msgr2=0x7ff75c071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 shutdown_connections 2024-09-16T05:45:53.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:53.957+0000 7ff763824640 1 -- 172.21.15.6:0/1290900092 wait complete. 2024-09-16T05:45:53.977 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:45:53.977 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:45:53.977 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:45:53.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:45:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient: get_monmap_and_config 2024-09-16T05:45:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient: build_initial_monmap 2024-09-16T05:45:54.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient: monmap: 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:54.104117+0000 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:54.104117+0000 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:54.103 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 1 Processor -- start 2024-09-16T05:45:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 1 -- start start 2024-09-16T05:45:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient: init 2024-09-16T05:45:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:45:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient(hunting): picked mon.noname-a con 0x7f05900a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient(hunting): picked mon.noname-b con 0x7f05900a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient(hunting): picked mon.noname-c con 0x7f05900a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059006e520 con 0x7f05900a37a0 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05900a0780 con 0x7f05900a1a00 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05900a4c70 con 0x7f05900a4290 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0597bbc640 10 monclient(hunting): authenticate will time out at 1615.910899s 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0595958640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f05900a37a0 0x7f05900a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38994/0 (socket says 172.21.15.6:38994) 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0595958640 1 -- 172.21.15.6:0/927946876 learned_addr learned my addr 172.21.15.6:0/927946876 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0595157640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f05900a4290 0x7f059013bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57562/0 (socket says 172.21.15.6:57562) 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251348911 0 0) 0x7f059006e520 con 0x7f05900a37a0 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): my global_id is 144135 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f057c003530 con 0x7f05900a37a0 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234836428 0 0) 0x7f05900a4c70 con 0x7f05900a4290 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): my global_id is 6860 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f059006e520 con 0x7f05900a4290 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2919992572 0 0) 0x7f05900a0780 con 0x7f05900a1a00 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient(hunting): my global_id is 134224 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05900a4c70 con 0x7f05900a1a00 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3033520099 0 0) 0x7f059006e520 con 0x7f05900a4290 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 >> v1:172.21.15.63:6789/0 conn(0x7f05900a1a00 legacy=0x7f059013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 >> v1:172.21.15.6:6789/0 conn(0x7f05900a37a0 legacy=0x7f05900a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient: _finish_hunting 0 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 monclient: found mon.noname-b 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05900a65f0 con 0x7f05900a4290 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient: _finish_auth 0 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient: _check_auth_tickets 2024-09-16T05:45:54.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f05800014b0 con 0x7f05900a4290 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.101+0000 7f0594956640 10 monclient: dump: 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.110 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 5 monclient: authenticate success, global_id 6860 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0580001790 con 0x7f05900a4290 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0594956640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0594956640 1 -- 172.21.15.6:0/927946876 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0580001ba0 con 0x7f05900a4290 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 4 set_mon_vals no callback set 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0594956640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0594956640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0594956640 10 monclient: dump: 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.111 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: shutdown 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- 172.21.15.6:0/927946876 >> v1:172.21.15.6:6790/0 conn(0x7f05900a4290 legacy=0x7f059013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- 172.21.15.6:0/927946876 shutdown_connections 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- 172.21.15.6:0/927946876 >> 172.21.15.6:0/927946876 conn(0x7f05900a58a0 msgr2=0x7f05900a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- 172.21.15.6:0/927946876 shutdown_connections 2024-09-16T05:45:54.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- 172.21.15.6:0/927946876 wait complete. 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: build_initial_monmap 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: monmap: 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:54.113 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 Processor -- start 2024-09-16T05:45:54.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- start start 2024-09-16T05:45:54.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: init 2024-09-16T05:45:54.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:54.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient(hunting): picked mon.noname-a con 0x7f05900a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient(hunting): picked mon.noname-b con 0x7f05900a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient(hunting): picked mon.noname-c con 0x7f05900a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05901c8c50 con 0x7f05900a1a00 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05900a0100 con 0x7f05900a37a0 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059013ccd0 con 0x7f05900a4290 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient(hunting): authenticate will time out at 1615.914309s 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0595958640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f05900a1a00 0x7f05901c7e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39004/0 (socket says 172.21.15.6:39004) 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0595958640 1 -- 172.21.15.6:0/3048152564 learned_addr learned my addr 172.21.15.6:0/3048152564 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1799487311 0 0) 0x7f05901c8c50 con 0x7f05900a1a00 2024-09-16T05:45:54.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): my global_id is 144138 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0574003530 con 0x7f05900a1a00 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2852428731 0 0) 0x7f059013ccd0 con 0x7f05900a4290 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): my global_id is 6863 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05901c8c50 con 0x7f05900a4290 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2736891644 0 0) 0x7f0574003530 con 0x7f05900a1a00 2024-09-16T05:45:54.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f059013ccd0 con 0x7f05900a1a00 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f058c001390 con 0x7f05900a1a00 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): dump: 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2416195751 0 0) 0x7f05900a0100 con 0x7f05900a37a0 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): my global_id is 134227 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0574003530 con 0x7f05900a37a0 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1848012504 0 0) 0x7f05901c8c50 con 0x7f05900a4290 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05900a0100 con 0x7f05900a4290 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0588001bc0 con 0x7f05900a4290 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:54.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient(hunting): dump: 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3093229702 0 0) 0x7f059013ccd0 con 0x7f05900a1a00 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 >> v1:172.21.15.6:6790/0 conn(0x7f05900a4290 legacy=0x7f05901d38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 >> v1:172.21.15.63:6789/0 conn(0x7f05900a37a0 legacy=0x7f05901c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient: _finish_hunting 0 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 monclient: found mon.a 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059013efc0 con 0x7f05900a1a00 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient: _finish_auth 0 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient: _check_auth_tickets 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 5 monclient: authenticate success, global_id 144138 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: _renew_subs 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f059013da50 con 0x7f05900a1a00 2024-09-16T05:45:54.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: _renew_subs 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f059013f910 con 0x7f05900a1a00 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f058c0015f0 con 0x7f05900a1a00 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f058c004c40 con 0x7f05900a1a00 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0587fff640 4 set_mon_vals no callback set 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f05867fc640 10 monclient: dump: 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.105+0000 7f0597bbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f05780065d0 con 0x7f0578002870 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f058c00f9e0 con 0x7f05900a1a00 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f058c044ec0 con 0x7f05900a1a00 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1512260095 0 0) 0x7f05780065d0 con 0x7f0578002870 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient(hunting): my global_id is 144141 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f059013ccd0 con 0x7f0578002870 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1216952576 0 0) 0x7f059013ccd0 con 0x7f0578002870 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f0596159640 10 monclient: get_auth_request con 0x7f0574036b60 auth_method 0 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05780065d0 con 0x7f0578002870 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0580001790 con 0x7f0578002870 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient: dump: 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1325516354 0 0) 0x7f05780065d0 con 0x7f0578002870 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f057800a160 con 0x7f0578002870 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f05800014d0 con 0x7f0578002870 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 >> v1:172.21.15.6:6789/0 conn(0x7f0578002870 legacy=0x7f0578004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.109+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 >> v1:172.21.15.6:6789/0 conn(0x7f0578002870 legacy=0x7f0578004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:54.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f0597bbc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0578009e50 con 0x7f0578009110 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894254453 0 0) 0x7f0578009e50 con 0x7f0578009110 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 10 monclient(hunting): my global_id is 144144 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05780065d0 con 0x7f0578009110 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 320707932 0 0) 0x7f05780065d0 con 0x7f0578009110 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0578009e50 con 0x7f0578009110 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0580004b80 con 0x7f0578009110 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:54.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.113+0000 7f05867fc640 10 monclient: dump: 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3763960278 0 0) 0x7f0578009e50 con 0x7f0578009110 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f057800c3e0 con 0x7f0578009110 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3962 (unknown 0 0 3367486535) 0x7f0580004b80 con 0x7f0578009110 2024-09-16T05:45:54.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f05867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:45:54.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f05867fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:54.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 >> v1:172.21.15.6:6789/0 conn(0x7f0578009110 legacy=0x7f05780095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f05867fc640 1 -- 172.21.15.6:0/3048152564 >> v1:172.21.15.6:6789/0 conn(0x7f0578009110 legacy=0x7f05780095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:54.131 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:45:54.131 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:45:54.131 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:45:54.131 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:45:54.131 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 46, 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 2731, 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.132 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.133 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.134 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:45:54.135 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.136 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.137 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.139 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.141 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:54.142 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:45:54.143 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.143 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.143 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:45:54.143 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:45:54.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0574036b60 legacy=0x7f0574039020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0597bbc640 10 monclient: shutdown 2024-09-16T05:45:54.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0597bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:54.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 >> v1:172.21.15.6:6789/0 conn(0x7f05900a1a00 legacy=0x7f05901c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0596159640 1 -- 172.21.15.6:0/3048152564 reap_dead start 2024-09-16T05:45:54.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0596159640 1 -- 172.21.15.6:0/3048152564 reap_dead start 2024-09-16T05:45:54.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 shutdown_connections 2024-09-16T05:45:54.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 >> 172.21.15.6:0/3048152564 conn(0x7f05900a58a0 msgr2=0x7f059013fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:54.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 shutdown_connections 2024-09-16T05:45:54.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.117+0000 7f0597bbc640 1 -- 172.21.15.6:0/3048152564 wait complete. 2024-09-16T05:45:54.145 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:45:54.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:45:54.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:45:54.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: get_monmap_and_config 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: build_initial_monmap 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: monmap: 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:54.263088+0000 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:54.263088+0000 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:54.262 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:54.263 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 Processor -- start 2024-09-16T05:45:54.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- start start 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: init 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0c0a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient(hunting): picked mon.noname-c con 0x7f0d0c0a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0c0a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d0c0a33a0 con 0x7f0d0c0a2190 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d0c0a07a0 con 0x7f0d0c0a2570 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d0c0a2d90 con 0x7f0d0c0a2980 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient(hunting): authenticate will time out at 1616.069699s 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d10fe4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0d0c0a2190 0x7f0d0c0a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39030/0 (socket says 172.21.15.6:39030) 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d10fe4640 1 -- 172.21.15.6:0/1164110815 learned_addr learned my addr 172.21.15.6:0/1164110815 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 177439192 0 0) 0x7f0d0c0a33a0 con 0x7f0d0c0a2190 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient(hunting): my global_id is 144147 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cec003530 con 0x7f0d0c0a2190 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815559943 0 0) 0x7f0d0c0a2d90 con 0x7f0d0c0a2980 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient(hunting): my global_id is 6866 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d0c0a33a0 con 0x7f0d0c0a2980 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 538718994 0 0) 0x7f0cec003530 con 0x7f0d0c0a2190 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 >> v1:172.21.15.6:6790/0 conn(0x7f0d0c0a2980 legacy=0x7f0d0c13f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 >> v1:172.21.15.63:6789/0 conn(0x7f0d0c0a2570 legacy=0x7f0d0c13b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 monclient: found mon.noname-a 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0c0a66e0 con 0x7f0d0c0a2190 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:45:54.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0cf40019a0 con 0x7f0d0c0a2190 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: dump: 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.274 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 5 monclient: authenticate success, global_id 144147 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0cf4001cc0 con 0x7f0d0c0a2190 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 1 -- 172.21.15.6:0/1164110815 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0cf4002090 con 0x7f0d0c0a2190 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 4 set_mon_vals no callback set 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d0b7fe640 10 monclient: dump: 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.275 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: shutdown 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- 172.21.15.6:0/1164110815 >> v1:172.21.15.6:6789/0 conn(0x7f0d0c0a2190 legacy=0x7f0d0c0a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- 172.21.15.6:0/1164110815 shutdown_connections 2024-09-16T05:45:54.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- 172.21.15.6:0/1164110815 >> 172.21.15.6:0/1164110815 conn(0x7f0d0c0a9850 msgr2=0x7f0d0c0a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- 172.21.15.6:0/1164110815 shutdown_connections 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 1 -- 172.21.15.6:0/1164110815 wait complete. 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: build_initial_monmap 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.261+0000 7f0d13248640 10 monclient: monmap: 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 1 Processor -- start 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 1 -- start start 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient: init 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient(hunting): picked mon.noname-c con 0x7f0d0c0a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0c0a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0c0a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:54.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d0c1240a0 con 0x7f0d0c0a2980 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d0c124380 con 0x7f0d0c0a2570 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d0c0a0100 con 0x7f0d0c0a2190 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient(hunting): authenticate will time out at 1616.072898s 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d117e5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0d0c0a2980 0x7f0d0c1217d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39044/0 (socket says 172.21.15.6:39044) 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d117e5640 1 -- 172.21.15.6:0/2468662308 learned_addr learned my addr 172.21.15.6:0/2468662308 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d10fe4640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0d0c0a2190 0x7f0d0c1209b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57576/0 (socket says 172.21.15.6:57576) 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1509060857 0 0) 0x7f0d0c1240a0 con 0x7f0d0c0a2980 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): my global_id is 144150 2024-09-16T05:45:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cf0003560 con 0x7f0d0c0a2980 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839316034 0 0) 0x7f0d0c0a0100 con 0x7f0d0c0a2190 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): my global_id is 6869 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d0c1240a0 con 0x7f0d0c0a2190 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 235310067 0 0) 0x7f0cf0003560 con 0x7f0d0c0a2980 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d0c0a0100 con 0x7f0d0c0a2980 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0d040018e0 con 0x7f0d0c0a2980 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): dump: 2024-09-16T05:45:54.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2263656094 0 0) 0x7f0d0c1240a0 con 0x7f0d0c0a2190 2024-09-16T05:45:54.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0cf0003560 con 0x7f0d0c0a2190 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0cf4001960 con 0x7f0d0c0a2190 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): dump: 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.282 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410191302 0 0) 0x7f0d0c124380 con 0x7f0d0c0a2570 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): my global_id is 134233 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d0c1240a0 con 0x7f0d0c0a2570 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1035664457 0 0) 0x7f0d0c0a0100 con 0x7f0d0c0a2980 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 >> v1:172.21.15.6:6790/0 conn(0x7f0d0c0a2190 legacy=0x7f0d0c1209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 >> v1:172.21.15.63:6789/0 conn(0x7f0d0c0a2570 legacy=0x7f0d0c1210c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: _finish_hunting 0 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 monclient: found mon.a 2024-09-16T05:45:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0c0a8080 con 0x7f0d0c0a2980 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: _finish_auth 0 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: _check_auth_tickets 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 5 monclient: authenticate success, global_id 144150 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient: _renew_subs 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0c122ed0 con 0x7f0d0c0a2980 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient: _renew_subs 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:45:54.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0d0c123110 con 0x7f0d0c0a2980 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d04001c70 con 0x7f0d0c0a2980 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0d04003af0 con 0x7f0d0c0a2980 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d0affd640 4 set_mon_vals no callback set 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: dump: 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.285 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ce40065d0 con 0x7f0ce4002870 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0d04003c90 con 0x7f0d0c0a2980 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f0d04044e60 con 0x7f0d0c0a2980 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961311514 0 0) 0x7f0ce40065d0 con 0x7f0ce4002870 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient(hunting): my global_id is 6872 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d0c0a0100 con 0x7f0ce4002870 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 375250765 0 0) 0x7f0d0c0a0100 con 0x7f0ce4002870 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d0bfff640 10 monclient: get_auth_request con 0x7f0cf0036bb0 auth_method 0 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ce40065d0 con 0x7f0ce4002870 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0cf40051f0 con 0x7f0ce4002870 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 10 monclient: dump: 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.287 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2461912504 0 0) 0x7f0ce40065d0 con 0x7f0ce4002870 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.265+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ce400a160 con 0x7f0ce4002870 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.269+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0cf4007410 con 0x7f0ce4002870 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.269+0000 7f0d097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.269+0000 7f0d097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.269+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 >> v1:172.21.15.6:6790/0 conn(0x7f0ce4002870 legacy=0x7f0ce4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.269+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 >> v1:172.21.15.6:6790/0 conn(0x7f0ce4002870 legacy=0x7f0ce4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:45:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ce4009e30 con 0x7f0ce40090f0 2024-09-16T05:45:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477350670 0 0) 0x7f0ce4009e30 con 0x7f0ce40090f0 2024-09-16T05:45:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient(hunting): my global_id is 6875 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ce40065d0 con 0x7f0ce40090f0 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 221571719 0 0) 0x7f0ce40065d0 con 0x7f0ce40090f0 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ce4009e30 con 0x7f0ce40090f0 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0cf4006aa0 con 0x7f0ce40090f0 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient: dump: 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.291 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1603310204 0 0) 0x7f0ce4009e30 con 0x7f0ce40090f0 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ce400c3a0 con 0x7f0ce40090f0 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 3679559928) 0x7f0cf4006880 con 0x7f0ce40090f0 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 >> v1:172.21.15.6:6790/0 conn(0x7f0ce40090f0 legacy=0x7f0ce4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d097fa640 1 -- 172.21.15.6:0/2468662308 >> v1:172.21.15.6:6790/0 conn(0x7f0ce40090f0 legacy=0x7f0ce4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 46, 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 683330, 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:45:54.293 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.294 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.295 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:45:54.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.297 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:45:54.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:45:54.299 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:45:54.300 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.301 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.302 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:54.303 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:54.304 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:54.305 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:54.305 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.305 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.305 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:45:54.305 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:45:54.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0cf0036bb0 legacy=0x7f0cf0039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 10 monclient: shutdown 2024-09-16T05:45:54.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:54.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 >> v1:172.21.15.6:6789/0 conn(0x7f0d0c0a2980 legacy=0x7f0d0c1217d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d117e5640 1 -- 172.21.15.6:0/2468662308 reap_dead start 2024-09-16T05:45:54.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d117e5640 1 -- 172.21.15.6:0/2468662308 reap_dead start 2024-09-16T05:45:54.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 shutdown_connections 2024-09-16T05:45:54.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 >> 172.21.15.6:0/2468662308 conn(0x7f0d0c0a9850 msgr2=0x7f0d0c12f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:54.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 shutdown_connections 2024-09-16T05:45:54.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.273+0000 7f0d13248640 1 -- 172.21.15.6:0/2468662308 wait complete. 2024-09-16T05:45:54.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:45:54.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:45:54.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:45:54.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient: get_monmap_and_config 2024-09-16T05:45:54.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient: build_initial_monmap 2024-09-16T05:45:54.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient: monmap: 2024-09-16T05:45:54.434 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:45:54.427312+0000 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:45:54.427312+0000 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 Processor -- start 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- start start 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient: init 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient(hunting): picked mon.noname-a con 0x7fd6f409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient(hunting): picked mon.noname-c con 0x7fd6f40a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f40a1370 con 0x7fd6f409e230 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f409ead0 con 0x7fd6f40a0da0 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f409edb0 con 0x7fd6f409aa30 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient(hunting): authenticate will time out at 1616.233977s 2024-09-16T05:45:54.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd6f409aa30 0x7fd6f409ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57592/0 (socket says 172.21.15.6:57592) 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f3fff640 1 -- 172.21.15.6:0/1301378946 learned_addr learned my addr 172.21.15.6:0/1301378946 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd6f409e230 0x7fd6f41399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39056/0 (socket says 172.21.15.6:39056) 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4087934413 0 0) 0x7fd6f40a1370 con 0x7fd6f409e230 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient(hunting): my global_id is 144153 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6e0003530 con 0x7fd6f409e230 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2812847420 0 0) 0x7fd6f409edb0 con 0x7fd6f409aa30 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient(hunting): my global_id is 6878 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f40a1370 con 0x7fd6f409aa30 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1606851543 0 0) 0x7fd6e0003530 con 0x7fd6f409e230 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 >> v1:172.21.15.6:6790/0 conn(0x7fd6f409aa30 legacy=0x7fd6f409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 >> v1:172.21.15.63:6789/0 conn(0x7fd6f40a0da0 legacy=0x7fd6f413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 monclient: found mon.noname-a 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f409dd50 con 0x7fd6f409e230 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:45:54.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd6dc0019a0 con 0x7fd6f409e230 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: dump: 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.439 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 5 monclient: authenticate success, global_id 144153 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6dc001c00 con 0x7fd6f409e230 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient: get_monmap_and_config success 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 4 set_mon_vals no callback set 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 1 -- 172.21.15.6:0/1301378946 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd6dc001f50 con 0x7fd6f409e230 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6f2ffd640 10 monclient: dump: 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.440 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 10 monclient: shutdown 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- 172.21.15.6:0/1301378946 >> v1:172.21.15.6:6789/0 conn(0x7fd6f409e230 legacy=0x7fd6f41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- 172.21.15.6:0/1301378946 shutdown_connections 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- 172.21.15.6:0/1301378946 >> 172.21.15.6:0/1301378946 conn(0x7fd6f40a1650 msgr2=0x7fd6f40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- 172.21.15.6:0/1301378946 shutdown_connections 2024-09-16T05:45:54.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.425+0000 7fd6fa772640 1 -- 172.21.15.6:0/1301378946 wait complete. 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: build_initial_monmap 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: monmap: 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 1 Processor -- start 2024-09-16T05:45:54.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 1 -- start start 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: init 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: _reopen_session rank -1 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient(hunting): picked mon.noname-c con 0x7fd6f409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient(hunting): picked mon.noname-b con 0x7fd6f409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient(hunting): picked mon.noname-a con 0x7fd6f40a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f413d560 con 0x7fd6f40a0da0 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f40740c0 con 0x7fd6f409e230 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6f406ea80 con 0x7fd6f409aa30 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient(hunting): _renew_subs 2024-09-16T05:45:54.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient(hunting): authenticate will time out at 1616.237145s 2024-09-16T05:45:54.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd6f409aa30 0x7fd6f409bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:57598/0 (socket says 172.21.15.6:57598) 2024-09-16T05:45:54.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f8d0f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd6f40a0da0 0x7fd6f41d37b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39070/0 (socket says 172.21.15.6:39070) 2024-09-16T05:45:54.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f3fff640 1 -- 172.21.15.6:0/1123878669 learned_addr learned my addr 172.21.15.6:0/1123878669 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901687719 0 0) 0x7fd6f406ea80 con 0x7fd6f409aa30 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): my global_id is 6881 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6d4003530 con 0x7fd6f409aa30 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2568598018 0 0) 0x7fd6f409ead0 con 0x7fd6f40a0da0 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): my global_id is 144156 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f406ea80 con 0x7fd6f40a0da0 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 209577310 0 0) 0x7fd6d4003530 con 0x7fd6f409aa30 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6f409ead0 con 0x7fd6f409aa30 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd6e4001970 con 0x7fd6f409aa30 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): dump: 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3324863511 0 0) 0x7fd6f406ea80 con 0x7fd6f40a0da0 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6d4003530 con 0x7fd6f40a0da0 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd6ec001840 con 0x7fd6f40a0da0 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:45:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): dump: 2024-09-16T05:45:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.448 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710765174 0 0) 0x7fd6f409dd50 con 0x7fd6f409e230 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): my global_id is 134239 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f406ea80 con 0x7fd6f409e230 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3353937560 0 0) 0x7fd6f409ead0 con 0x7fd6f409aa30 2024-09-16T05:45:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 >> v1:172.21.15.63:6789/0 conn(0x7fd6f409e230 legacy=0x7fd6f409c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 >> v1:172.21.15.6:6789/0 conn(0x7fd6f40a0da0 legacy=0x7fd6f41d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 monclient: found mon.c 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f413ce60 con 0x7fd6f409aa30 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient: _finish_auth 0 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 5 monclient: authenticate success, global_id 6881 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: _renew_subs 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f413d140 con 0x7fd6f409aa30 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: _renew_subs 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd6f409cbc0 con 0x7fd6f409aa30 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6e4001d00 con 0x7fd6f409aa30 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd6e4004c40 con 0x7fd6f409aa30 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f27fc640 4 set_mon_vals no callback set 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient: dump: 2024-09-16T05:45:54.451 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd6e400f9e0 con 0x7fd6f409aa30 2024-09-16T05:45:54.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:54.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd6d80065d0 con 0x7fd6d8002870 2024-09-16T05:45:54.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fd6e4044ed0 con 0x7fd6f409aa30 2024-09-16T05:45:54.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f8d0f640 10 monclient: get_auth_request con 0x7fd6d4036af0 auth_method 0 2024-09-16T05:45:54.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1546573950 0 0) 0x7fd6d80065d0 con 0x7fd6d8002870 2024-09-16T05:45:54.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 10 monclient(hunting): my global_id is 134242 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.429+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f409ead0 con 0x7fd6d8002870 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2779278761 0 0) 0x7fd6f409ead0 con 0x7fd6d8002870 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6d80065d0 con 0x7fd6d8002870 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd6dc0018d0 con 0x7fd6d8002870 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 10 monclient: dump: 2024-09-16T05:45:54.454 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 557587071 0 0) 0x7fd6d80065d0 con 0x7fd6d8002870 2024-09-16T05:45:54.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6d800a160 con 0x7fd6d8002870 2024-09-16T05:45:54.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd6dc001920 con 0x7fd6d8002870 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 >> v1:172.21.15.63:6789/0 conn(0x7fd6d8002870 legacy=0x7fd6d8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.433+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 >> v1:172.21.15.63:6789/0 conn(0x7fd6d8002870 legacy=0x7fd6d8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.437+0000 7fd6fa772640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.437+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd6d8009e50 con 0x7fd6d8009110 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 275751919 0 0) 0x7fd6d8009e50 con 0x7fd6d8009110 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:45:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:45:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient(hunting): my global_id is 134245 2024-09-16T05:45:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6d80065d0 con 0x7fd6d8009110 2024-09-16T05:45:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4062804734 0 0) 0x7fd6d80065d0 con 0x7fd6d8009110 2024-09-16T05:45:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6d8009e50 con 0x7fd6d8009110 2024-09-16T05:45:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd6dc005000 con 0x7fd6d8009110 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient: dump: 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:45:54.460 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:45:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:45:54.461 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:45:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3130023417 0 0) 0x7fd6d8009e50 con 0x7fd6d8009110 2024-09-16T05:45:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6d800c3e0 con 0x7fd6d8009110 2024-09-16T05:45:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3962 (unknown 0 0 4251604616) 0x7fd6dc0051f0 con 0x7fd6d8009110 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 >> v1:172.21.15.63:6789/0 conn(0x7fd6d8009110 legacy=0x7fd6d80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f0ff9640 1 -- 172.21.15.6:0/1123878669 >> v1:172.21.15.63:6789/0 conn(0x7fd6d8009110 legacy=0x7fd6d80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:45:54.462 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 46, 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 112030, 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:45:54.463 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.464 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.465 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:45:54.466 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:45:54.467 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.470 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:45:54.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.472 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:45:54.473 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:45:54.474 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:45:54.474 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:45:54.474 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:45:54.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:45:54.474 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:45:54.474 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:45:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd6d4036af0 legacy=0x7fd6d4038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6fa772640 10 monclient: shutdown 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6fa772640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 >> v1:172.21.15.6:6790/0 conn(0x7fd6f409aa30 legacy=0x7fd6f409bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f8d0f640 1 -- 172.21.15.6:0/1123878669 reap_dead start 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6f8d0f640 1 -- 172.21.15.6:0/1123878669 reap_dead start 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 shutdown_connections 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 >> 172.21.15.6:0/1123878669 conn(0x7fd6f40a1650 msgr2=0x7fd6f406eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 shutdown_connections 2024-09-16T05:45:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:45:54.441+0000 7fd6fa772640 1 -- 172.21.15.6:0/1123878669 wait complete. 2024-09-16T05:45:54.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:45:54.477 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:45:54.477 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T05:45:54.477 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:45:54.477 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:45:54.477 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:45:54.478 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:45:54.502 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:45:54.502 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T05:45:54.503 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T05:45:54.503 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:46:09.504 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T05:46:09.505 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T05:46:09.505 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:46:09.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:46:09.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient: get_monmap_and_config 2024-09-16T05:46:09.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient: build_initial_monmap 2024-09-16T05:46:09.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient: monmap: 2024-09-16T05:46:09.639 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:46:09.639 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:09.639 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:46:09.640740+0000 2024-09-16T05:46:09.640 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:46:09.640740+0000 2024-09-16T05:46:09.640 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:46:09.640 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:09.640 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:46:09.640 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:46:09.640 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:46:09.640 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:09.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 1 Processor -- start 2024-09-16T05:46:09.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 1 -- start start 2024-09-16T05:46:09.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient: init 2024-09-16T05:46:09.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient: _reopen_session rank -1 2024-09-16T05:46:09.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient(hunting): picked mon.noname-b con 0x7f84180a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient(hunting): picked mon.noname-c con 0x7f84180a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient(hunting): picked mon.noname-a con 0x7f84180a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f841806e520 con 0x7f84180a1a00 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84180a0780 con 0x7f84180a4290 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84180a4c70 con 0x7f84180a37a0 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient(hunting): _renew_subs 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f841dbaa640 10 monclient(hunting): authenticate will time out at 1631.447681s 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f8416ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f841813bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f8417fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f84180a1a00 0x7f841813f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56920/0 (socket says 172.21.15.6:56920) 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f84177fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f84180a37a0 0x7f84180a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:59142/0 (socket says 172.21.15.6:59142) 2024-09-16T05:46:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.637+0000 7f8417fff640 1 -- 172.21.15.6:0/2060145596 learned_addr learned my addr 172.21.15.6:0/2060145596 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:46:09.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:09.841+0000 7f8416ffd640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f841813bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:10.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:10.245+0000 7f8416ffd640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f841813bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:11.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:11.049+0000 7f8416ffd640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f841813bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:12.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 10 monclient(hunting): tick 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 1 monclient(hunting): continuing hunt 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.6:6790/0 conn(0x7f84180a37a0 legacy=0x7f84180a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f841813bab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.6:6789/0 conn(0x7f84180a1a00 legacy=0x7f841813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 10 monclient(hunting): picked mon.noname-a con 0x7f83f80036d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.637+0000 7f84157fa640 10 monclient(hunting): picked mon.noname-b con 0x7f84180a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.641+0000 7f84157fa640 10 monclient(hunting): picked mon.noname-c con 0x7f84180a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:46:12.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.641+0000 7f84157fa640 1 -- 172.21.15.6:0/2060145596 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84180a65f0 con 0x7f83f80036d0 2024-09-16T05:46:12.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.641+0000 7f84157fa640 1 -- 172.21.15.6:0/2060145596 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84180a0780 con 0x7f84180a4290 2024-09-16T05:46:12.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.641+0000 7f84157fa640 1 -- 172.21.15.6:0/2060145596 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83f80103b0 con 0x7f84180a37a0 2024-09-16T05:46:12.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.641+0000 7f84157fa640 10 monclient(hunting): _renew_subs 2024-09-16T05:46:12.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.641+0000 7f8417fff640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f83f800cc00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:12.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:12.845+0000 7f8417fff640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f83f800cc00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:13.249+0000 7f8417fff640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f83f800cc00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:14.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.053+0000 7f8417fff640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f83f800cc00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:14.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063021429 0 0) 0x7f83f80103b0 con 0x7f84180a37a0 2024-09-16T05:46:14.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient(hunting): _negotiate 2024-09-16T05:46:14.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:14.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:14.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient(hunting): my global_id is 6884 2024-09-16T05:46:14.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8404003530 con 0x7f84180a37a0 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1565968253 0 0) 0x7f8404003530 con 0x7f84180a37a0 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f83f800cc00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.6:6789/0 conn(0x7f83f80036d0 legacy=0x7f83f8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: _finish_hunting 0 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 monclient: found mon.noname-b 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f8011600 con 0x7f84180a37a0 2024-09-16T05:46:14.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: _finish_auth 0 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: _check_auth_tickets 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f84000044e0 con 0x7f84180a37a0 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: dump: 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.554 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f841dbaa640 5 monclient: authenticate success, global_id 6884 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f841dbaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8400004740 con 0x7f84180a37a0 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 1 -- 172.21.15.6:0/2060145596 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8400004b90 con 0x7f84180a37a0 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f841dbaa640 10 monclient: get_monmap_and_config success 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f841dbaa640 4 set_mon_vals no callback set 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:14.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.549+0000 7f84167fc640 10 monclient: dump: 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.556 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient: shutdown 2024-09-16T05:46:14.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:46:14.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- 172.21.15.6:0/2060145596 >> v1:172.21.15.6:6790/0 conn(0x7f84180a37a0 legacy=0x7f83f8009400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- 172.21.15.6:0/2060145596 shutdown_connections 2024-09-16T05:46:14.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- 172.21.15.6:0/2060145596 >> 172.21.15.6:0/2060145596 conn(0x7f84180a58a0 msgr2=0x7f84180a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:46:14.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- 172.21.15.6:0/2060145596 shutdown_connections 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- 172.21.15.6:0/2060145596 wait complete. 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient: build_initial_monmap 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient: monmap: 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:46:14.561 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 Processor -- start 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- start start 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient: init 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient: _reopen_session rank -1 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient(hunting): picked mon.noname-c con 0x7f83f80036d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient(hunting): picked mon.noname-a con 0x7f84180a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient(hunting): picked mon.noname-b con 0x7f84180a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8418126310 con 0x7f84180a37a0 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8418127aa0 con 0x7f84180a4290 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8418137600 con 0x7f83f80036d0 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient(hunting): _renew_subs 2024-09-16T05:46:14.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f841dbaa640 10 monclient(hunting): authenticate will time out at 1636.363228s 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f8417fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f84181d4200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f8416ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f84180a37a0 0x7f84181d0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56950/0 (socket says 172.21.15.6:56950) 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f84177fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f83f80036d0 0x7f84181cbaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:59148/0 (socket says 172.21.15.6:59148) 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f8416ffd640 1 -- 172.21.15.6:0/645591195 learned_addr learned my addr 172.21.15.6:0/645591195 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144911334 0 0) 0x7f8418137600 con 0x7f83f80036d0 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f840ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f840ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f840ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f840ffff640 10 monclient(hunting): my global_id is 6887 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.553+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83f80063d0 con 0x7f83f80036d0 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 501626784 0 0) 0x7f83f80063d0 con 0x7f83f80036d0 2024-09-16T05:46:14.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8418137600 con 0x7f83f80036d0 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f84080017b0 con 0x7f83f80036d0 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient(hunting): dump: 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.564 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 362227906 0 0) 0x7f8418137600 con 0x7f83f80036d0 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 >> v1:172.21.15.63:6789/0 conn(0x7f84180a4290 legacy=0x7f84181d4200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 >> v1:172.21.15.6:6789/0 conn(0x7f84180a37a0 legacy=0x7f84181d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 monclient: found mon.c 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84180a8630 con 0x7f83f80036d0 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient: _finish_auth 0 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:46:14.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 5 monclient: authenticate success, global_id 6887 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 10 monclient: _renew_subs 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f84181274e0 con 0x7f83f80036d0 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 10 monclient: _renew_subs 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8418124bc0 con 0x7f83f80036d0 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f84080047f0 con 0x7f83f80036d0 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8408004b40 con 0x7f83f80036d0 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:46:14.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 10 monclient: dump: 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f8415ffb640 4 set_mon_vals no callback set 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:46:14.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83fc005090 con 0x7f83f80036d0 2024-09-16T05:46:14.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8408004ce0 con 0x7f83f80036d0 2024-09-16T05:46:14.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f8408044d30 con 0x7f83f80036d0 2024-09-16T05:46:14.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.557+0000 7f8416ffd640 10 monclient: get_auth_request con 0x7f83f8037380 auth_method 0 2024-09-16T05:46:14.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.565+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8408018fb0 con 0x7f83f80036d0 2024-09-16T05:46:14.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.565+0000 7f840ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:46:14.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.565+0000 7f840ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:46:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.821+0000 7f841dbaa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:46:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.821+0000 7f841dbaa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:46:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.821+0000 7f841dbaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.821+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f83fc005380 con 0x7f83f80036d0 2024-09-16T05:46:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.821+0000 7f840ffff640 1 -- 172.21.15.6:0/645591195 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 2803614208) 0x7f840801d870 con 0x7f83f80036d0 2024-09-16T05:46:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.821+0000 7f840ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:46:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.821+0000 7f840ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:46:14.824 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":50,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:46:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.825+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 >> v1:172.21.15.63:6812/3643207211 conn(0x7f83f8037380 legacy=0x7f83f8039840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.825+0000 7f841dbaa640 10 monclient: shutdown 2024-09-16T05:46:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.825+0000 7f841dbaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:46:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.825+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 >> v1:172.21.15.6:6790/0 conn(0x7f83f80036d0 legacy=0x7f84181cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.825+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 shutdown_connections 2024-09-16T05:46:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.825+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 >> 172.21.15.6:0/645591195 conn(0x7f84180a58a0 msgr2=0x7f841809d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:46:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.825+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 shutdown_connections 2024-09-16T05:46:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.825+0000 7f841dbaa640 1 -- 172.21.15.6:0/645591195 wait complete. 2024-09-16T05:46:14.842 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:46:14.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:46:14.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:46:14.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient: get_monmap_and_config 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient: build_initial_monmap 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient: monmap: 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:46:14.967113+0000 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:46:14.967113+0000 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:46:14.966 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 Processor -- start 2024-09-16T05:46:14.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- start start 2024-09-16T05:46:14.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient: init 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient: _reopen_session rank -1 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient(hunting): picked mon.noname-b con 0x7fd6240a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient(hunting): picked mon.noname-c con 0x7fd6240a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient(hunting): picked mon.noname-a con 0x7fd6240a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6240a33a0 con 0x7fd6240a2980 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6240a07a0 con 0x7fd6240a2570 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6240a2d90 con 0x7fd6240a2190 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient(hunting): _renew_subs 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient(hunting): authenticate will time out at 1636.773746s 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd622ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd6240a2190 0x7fd6240a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:59150/0 (socket says 172.21.15.6:59150) 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd622ffd640 1 -- 172.21.15.6:0/2816104340 learned_addr learned my addr 172.21.15.6:0/2816104340 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:46:14.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6227fc640 1 -- 172.21.15.6:0/2816104340 >> v1:172.21.15.63:6789/0 conn(0x7fd6240a2570 legacy=0x7fd62413b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2305115022 0 0) 0x7fd6240a2d90 con 0x7fd6240a2190 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient(hunting): my global_id is 6890 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd604003530 con 0x7fd6240a2190 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2153500497 0 0) 0x7fd6240a33a0 con 0x7fd6240a2980 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient(hunting): my global_id is 144159 2024-09-16T05:46:14.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6240a2d90 con 0x7fd6240a2980 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 901543209 0 0) 0x7fd6240a2d90 con 0x7fd6240a2980 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 >> v1:172.21.15.6:6790/0 conn(0x7fd6240a2190 legacy=0x7fd6240a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 >> v1:172.21.15.63:6789/0 conn(0x7fd6240a2570 legacy=0x7fd62413b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 monclient: found mon.noname-a 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6240a66e0 con 0x7fd6240a2980 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: _finish_auth 0 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd614001600 con 0x7fd6240a2980 2024-09-16T05:46:14.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: dump: 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 5 monclient: authenticate success, global_id 144159 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6140018e0 con 0x7fd6240a2980 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 1 -- 172.21.15.6:0/2816104340 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd614001cf0 con 0x7fd6240a2980 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient: get_monmap_and_config success 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 4 set_mon_vals no callback set 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd621ffb640 10 monclient: dump: 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.978 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 10 monclient: shutdown 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- 172.21.15.6:0/2816104340 >> v1:172.21.15.6:6789/0 conn(0x7fd6240a2980 legacy=0x7fd62413f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- 172.21.15.6:0/2816104340 shutdown_connections 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- 172.21.15.6:0/2816104340 >> 172.21.15.6:0/2816104340 conn(0x7fd6240a9850 msgr2=0x7fd6240a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- 172.21.15.6:0/2816104340 shutdown_connections 2024-09-16T05:46:14.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.965+0000 7fd6292c3640 1 -- 172.21.15.6:0/2816104340 wait complete. 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: build_initial_monmap 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: monmap: 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 1 Processor -- start 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 1 -- start start 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: init 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: _reopen_session rank -1 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient(hunting): picked mon.noname-b con 0x7fd6240a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient(hunting): picked mon.noname-c con 0x7fd6240a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient(hunting): picked mon.noname-a con 0x7fd6240a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6240a8260 con 0x7fd6240a2980 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6240a7d30 con 0x7fd6240a2190 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd62413f750 con 0x7fd6240a2570 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient(hunting): _renew_subs 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient(hunting): authenticate will time out at 1636.776960s 2024-09-16T05:46:14.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6227fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd6240a2570 0x7fd6241d01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:59158/0 (socket says 172.21.15.6:59158) 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6227fc640 1 -- 172.21.15.6:0/1298107008 learned_addr learned my addr 172.21.15.6:0/1298107008 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd622ffd640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.63:6789/0 conn(0x7fd6240a2190 legacy=0x7fd624138e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258669593 0 0) 0x7fd62413f750 con 0x7fd6240a2570 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): my global_id is 6893 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd608003530 con 0x7fd6240a2570 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1172791816 0 0) 0x7fd6240a8260 con 0x7fd6240a2980 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): my global_id is 144162 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd62413f750 con 0x7fd6240a2980 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3812671890 0 0) 0x7fd608003530 con 0x7fd6240a2570 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6240a8260 con 0x7fd6240a2570 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd60c0041e0 con 0x7fd6240a2570 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): dump: 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.983 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3520513579 0 0) 0x7fd62413f750 con 0x7fd6240a2980 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd608003530 con 0x7fd6240a2980 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd614001c30 con 0x7fd6240a2980 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): dump: 2024-09-16T05:46:14.984 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 723820761 0 0) 0x7fd6240a8260 con 0x7fd6240a2570 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.63:6789/0 conn(0x7fd6240a2190 legacy=0x7fd624138e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.6:6789/0 conn(0x7fd6240a2980 legacy=0x7fd6241d38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 monclient: found mon.c 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6241207d0 con 0x7fd6240a2570 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 5 monclient: authenticate success, global_id 6893 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: _renew_subs 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6241204f0 con 0x7fd6240a2570 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: _renew_subs 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:46:14.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd6241cbcc0 con 0x7fd6240a2570 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd60c004490 con 0x7fd6240a2570 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd60c004860 con 0x7fd6240a2570 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6217fa640 4 set_mon_vals no callback set 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: dump: 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.987 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd60c004a80 con 0x7fd6240a2570 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd5fc0065d0 con 0x7fd5fc002870 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fd60c044fc0 con 0x7fd6240a2570 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129872438 0 0) 0x7fd5fc0065d0 con 0x7fd5fc002870 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:14.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient(hunting): my global_id is 144165 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6240a8260 con 0x7fd5fc002870 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd6237fe640 10 monclient: get_auth_request con 0x7fd6080369d0 auth_method 0 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1749916722 0 0) 0x7fd6240a8260 con 0x7fd5fc002870 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5fc0065d0 con 0x7fd5fc002870 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd61c001ca0 con 0x7fd5fc002870 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 10 monclient: dump: 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2765626103 0 0) 0x7fd5fc0065d0 con 0x7fd5fc002870 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.969+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5fc00a160 con 0x7fd5fc002870 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.973+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd61c001ca0 con 0x7fd5fc002870 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.973+0000 7fd61b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.973+0000 7fd61b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.973+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.6:6789/0 conn(0x7fd5fc002870 legacy=0x7fd5fc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.973+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.6:6789/0 conn(0x7fd5fc002870 legacy=0x7fd5fc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:46:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:46:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd5fc009e30 con 0x7fd5fc0090f0 2024-09-16T05:46:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606604127 0 0) 0x7fd5fc009e30 con 0x7fd5fc0090f0 2024-09-16T05:46:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:46:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient(hunting): my global_id is 144168 2024-09-16T05:46:14.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5fc0065d0 con 0x7fd5fc0090f0 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2663461788 0 0) 0x7fd5fc0065d0 con 0x7fd5fc0090f0 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5fc009e30 con 0x7fd5fc0090f0 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd61c0051a0 con 0x7fd5fc0090f0 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient: dump: 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:14.993 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3847152577 0 0) 0x7fd5fc009e30 con 0x7fd5fc0090f0 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd5fc00c3a0 con 0x7fd5fc0090f0 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3952 (unknown 0 0 803126678) 0x7fd61c0073c0 con 0x7fd5fc0090f0 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.6:6789/0 conn(0x7fd5fc0090f0 legacy=0x7fd5fc009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd61b7fe640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.6:6789/0 conn(0x7fd5fc0090f0 legacy=0x7fd5fc009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:46:14.994 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 50, 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 23591, 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:46:14.995 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:46:14.996 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:46:14.997 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:46:14.998 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:46:14.999 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:46:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:46:15.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.002 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:46:15.003 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:46:15.005 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.006 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.006 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:46:15.006 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:46:15.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd6080369d0 legacy=0x7fd608038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 10 monclient: shutdown 2024-09-16T05:46:15.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:46:15.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 >> v1:172.21.15.6:6790/0 conn(0x7fd6240a2570 legacy=0x7fd6241d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6237fe640 1 -- 172.21.15.6:0/1298107008 reap_dead start 2024-09-16T05:46:15.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6237fe640 1 -- 172.21.15.6:0/1298107008 reap_dead start 2024-09-16T05:46:15.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 shutdown_connections 2024-09-16T05:46:15.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 >> 172.21.15.6:0/1298107008 conn(0x7fd6240a9850 msgr2=0x7fd62406d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:46:15.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 shutdown_connections 2024-09-16T05:46:15.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:14.977+0000 7fd6292c3640 1 -- 172.21.15.6:0/1298107008 wait complete. 2024-09-16T05:46:15.008 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:46:15.008 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:46:15.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:46:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient: get_monmap_and_config 2024-09-16T05:46:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient: build_initial_monmap 2024-09-16T05:46:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient: monmap: 2024-09-16T05:46:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:46:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:46:15.127614+0000 2024-09-16T05:46:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:46:15.127614+0000 2024-09-16T05:46:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:46:15.127 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.127 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:46:15.127 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:46:15.127 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:46:15.127 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 Processor -- start 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- start start 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient: init 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient: _reopen_session rank -1 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient(hunting): picked mon.noname-a con 0x7f116809bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient(hunting): picked mon.noname-b con 0x7f116809bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient(hunting): picked mon.noname-c con 0x7f11681397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11680a1350 con 0x7f116809bb40 2024-09-16T05:46:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11680a10d0 con 0x7f11681397f0 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f116809d940 con 0x7f116809bf20 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient(hunting): _renew_subs 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient(hunting): authenticate will time out at 1636.934209s 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116e32e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f116809bb40 0x7f116809e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57008/0 (socket says 172.21.15.6:57008) 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116e32e640 1 -- 172.21.15.6:0/2052454443 learned_addr learned my addr 172.21.15.6:0/2052454443 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116db2d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f116809bf20 0x7f116809c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:59170/0 (socket says 172.21.15.6:59170) 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116eb2f640 1 -- 172.21.15.6:0/2052454443 >> v1:172.21.15.63:6789/0 conn(0x7f11681397f0 legacy=0x7f116813bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277581651 0 0) 0x7f11680a1350 con 0x7f116809bb40 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient(hunting): _negotiate 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient(hunting): my global_id is 144171 2024-09-16T05:46:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1154003530 con 0x7f116809bb40 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 580732380 0 0) 0x7f116809d940 con 0x7f116809bf20 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient(hunting): _negotiate 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient(hunting): my global_id is 6896 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11680a1350 con 0x7f116809bf20 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 895028046 0 0) 0x7f1154003530 con 0x7f116809bb40 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 >> v1:172.21.15.6:6790/0 conn(0x7f116809bf20 legacy=0x7f116809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 >> v1:172.21.15.63:6789/0 conn(0x7f11681397f0 legacy=0x7f116813bbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: _finish_hunting 0 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 monclient: found mon.noname-a 2024-09-16T05:46:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116809dc20 con 0x7f116809bb40 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: _finish_auth 0 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: _check_auth_tickets 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11500019a0 con 0x7f116809bb40 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 5 monclient: authenticate success, global_id 144171 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: dump: 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1150001c50 con 0x7f116809bb40 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient: get_monmap_and_config success 2024-09-16T05:46:15.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 1 -- 172.21.15.6:0/2052454443 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1150002060 con 0x7f116809bb40 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 4 set_mon_vals no callback set 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f116d32c640 10 monclient: dump: 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:15.139 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 10 monclient: shutdown 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- 172.21.15.6:0/2052454443 >> v1:172.21.15.6:6789/0 conn(0x7f116809bb40 legacy=0x7f116809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- 172.21.15.6:0/2052454443 shutdown_connections 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- 172.21.15.6:0/2052454443 >> 172.21.15.6:0/2052454443 conn(0x7f11680a1650 msgr2=0x7f11680a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- 172.21.15.6:0/2052454443 shutdown_connections 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.125+0000 7f1170592640 1 -- 172.21.15.6:0/2052454443 wait complete. 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: build_initial_monmap 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: monmap: 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:46:15.140 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 1 Processor -- start 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 1 -- start start 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: init 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: _reopen_session rank -1 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:46:15.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient(hunting): picked mon.noname-c con 0x7f116809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient(hunting): picked mon.noname-a con 0x7f116809bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient(hunting): picked mon.noname-b con 0x7f11681397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11680740c0 con 0x7f116809bf20 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11680a3c20 con 0x7f11681397f0 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f116811eb00 con 0x7f116809bb40 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient(hunting): _renew_subs 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient(hunting): authenticate will time out at 1636.937339s 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f116db2d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f116809bf20 0x7f11681d07f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57010/0 (socket says 172.21.15.6:57010) 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f116db2d640 1 -- 172.21.15.6:0/820304217 learned_addr learned my addr 172.21.15.6:0/820304217 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f116e32e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f116809bb40 0x7f11681cb810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:59180/0 (socket says 172.21.15.6:59180) 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f116eb2f640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.63:6789/0 conn(0x7f11681397f0 legacy=0x7f11681d3f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731549169 0 0) 0x7f11680740c0 con 0x7f116809bf20 2024-09-16T05:46:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): my global_id is 144174 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1148003530 con 0x7f116809bf20 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979001785 0 0) 0x7f116811eb00 con 0x7f116809bb40 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): my global_id is 6899 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11680740c0 con 0x7f116809bb40 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2470896293 0 0) 0x7f1148003530 con 0x7f116809bf20 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f116811eb00 con 0x7f116809bf20 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 15026274 0 0) 0x7f11680740c0 con 0x7f116809bb40 2024-09-16T05:46:15.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1148003530 con 0x7f116809bb40 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1158001840 con 0x7f116809bf20 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): dump: 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:15.144 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1150001fa0 con 0x7f116809bb40 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): dump: 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 158759534 0 0) 0x7f116811eb00 con 0x7f116809bf20 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.6:6790/0 conn(0x7f116809bb40 legacy=0x7f11681cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.63:6789/0 conn(0x7f11681397f0 legacy=0x7f11681d3f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 monclient: found mon.a 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116811d080 con 0x7f116809bf20 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: _finish_auth 0 2024-09-16T05:46:15.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 5 monclient: authenticate success, global_id 144174 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: _renew_subs 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f116811e820 con 0x7f116809bf20 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: _renew_subs 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f116811c1b0 con 0x7f116809bf20 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11580047f0 con 0x7f116809bf20 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1158004b40 con 0x7f116809bf20 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:15.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: dump: 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f116cb2b640 4 set_mon_vals no callback set 2024-09-16T05:46:15.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f11680008e0 con 0x7f116813d300 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3303377888 0 0) 0x7f11680008e0 con 0x7f116813d300 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient(hunting): my global_id is 6902 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f116811eb00 con 0x7f116813d300 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4232934934 0 0) 0x7f116811eb00 con 0x7f116813d300 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11680008e0 con 0x7f116813d300 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f11500051b0 con 0x7f116813d300 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:46:15.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 10 monclient: dump: 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.129+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f115800f8e0 con 0x7f116809bf20 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f1158044d30 con 0x7f116809bf20 2024-09-16T05:46:15.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2660460293 0 0) 0x7f11680008e0 con 0x7f116813d300 2024-09-16T05:46:15.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f116811c4a0 con 0x7f116813d300 2024-09-16T05:46:15.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f116eb2f640 10 monclient: get_auth_request con 0x7f1148036a00 auth_method 0 2024-09-16T05:46:15.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1150005480 con 0x7f116813d300 2024-09-16T05:46:15.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f1166ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:46:15.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f1166ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:46:15.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.6:6790/0 conn(0x7f116813d300 legacy=0x7f116806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.133+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.6:6790/0 conn(0x7f116813d300 legacy=0x7f116806ded0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:46:15.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1170592640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f116811e260 con 0x7f11681cbf20 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096126727 0 0) 0x7f116811e260 con 0x7f11681cbf20 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient(hunting): my global_id is 6905 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11680008e0 con 0x7f11681cbf20 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 40872329 0 0) 0x7f11680008e0 con 0x7f11681cbf20 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f116811e260 con 0x7f11681cbf20 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1150005b50 con 0x7f11681cbf20 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:46:15.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient: dump: 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:46:15.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1127977666 0 0) 0x7f116811e260 con 0x7f11681cbf20 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f11681cc330 con 0x7f11681cbf20 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4285 (unknown 0 0 1662448236) 0x7f1150005e00 con 0x7f11681cbf20 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.6:6790/0 conn(0x7f11681cbf20 legacy=0x7f116806e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.137+0000 7f1166ffd640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.6:6790/0 conn(0x7f11681cbf20 legacy=0x7f116806e5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:46:15.155 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 50, 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 704193, 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:46:15.156 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:46:15.157 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.158 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:46:15.159 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:46:15.160 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:46:15.161 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:46:15.162 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:46:15.163 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:46:15.164 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:46:15.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:15.166 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:46:15.167 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:46:15.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:46:15.168 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:46:15.168 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:46:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1148036a00 legacy=0x7f1148038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f1170592640 10 monclient: shutdown 2024-09-16T05:46:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f1170592640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:46:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 >> v1:172.21.15.6:6789/0 conn(0x7f116809bf20 legacy=0x7f11681d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:46:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f116eb2f640 1 -- 172.21.15.6:0/820304217 reap_dead start 2024-09-16T05:46:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f116eb2f640 1 -- 172.21.15.6:0/820304217 reap_dead start 2024-09-16T05:46:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 shutdown_connections 2024-09-16T05:46:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 >> 172.21.15.6:0/820304217 conn(0x7f11680a1650 msgr2=0x7f11680a4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:46:15.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 shutdown_connections 2024-09-16T05:46:15.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:46:15.141+0000 7f1170592640 1 -- 172.21.15.6:0/820304217 wait complete. 2024-09-16T05:46:15.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:46:15.170 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:47:19.188 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=110 2024-09-16T05:47:19.188 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 110 -ne 110 ]] 2024-09-16T05:47:19.188 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:47:19.189 INFO:tasks.workunit.client.0.smithi063.stderr:+ export CEPH_ARGS= 2024-09-16T05:47:19.189 INFO:tasks.workunit.client.0.smithi063.stderr:+ CEPH_ARGS= 2024-09-16T05:47:19.189 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph auth del client.foo 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient: get_monmap_and_config 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient: build_initial_monmap 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient: monmap: 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:47:19.305362+0000 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:47:19.305362+0000 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:47:19.305 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:47:19.306 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:47:19.306 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:47:19.306 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 1 Processor -- start 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 1 -- start start 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient: init 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient: _reopen_session rank -1 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient(hunting): picked mon.noname-c con 0x7febd80a3a90 addr v1:172.21.15.63:6789/0 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient(hunting): picked mon.noname-a con 0x7febd80a4580 addr v1:172.21.15.6:6789/0 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient(hunting): picked mon.noname-b con 0x7febd813d510 addr v1:172.21.15.6:6790/0 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd80a0620 con 0x7febd80a4580 2024-09-16T05:47:19.308 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd80a1d30 con 0x7febd80a3a90 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd80a4fb0 con 0x7febd813d510 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient(hunting): _renew_subs 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 10 monclient(hunting): authenticate will time out at 1696.082475s 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7febd80a3a90 legacy=0x7febd80a3e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febdcafc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7febd813d510 0x7febd813f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:36736/0 (socket says 172.21.15.63:36736) 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febdcafc640 1 -- 172.21.15.63:0/585855147 learned_addr learned my addr 172.21.15.63:0/585855147 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1172926599 0 0) 0x7febd80a4fb0 con 0x7febd813d510 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:47:19.309 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient(hunting): my global_id is 6908 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febc0003530 con 0x7febd813d510 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330506818 0 0) 0x7febcc001120 con 0x7febd80a4580 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient(hunting): my global_id is 144177 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febd80a4fb0 con 0x7febd80a4580 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1609796181 0 0) 0x7febd80a4fb0 con 0x7febd80a4580 2024-09-16T05:47:19.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 >> v1:172.21.15.6:6790/0 conn(0x7febd813d510 legacy=0x7febd813f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:47:19.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 >> v1:172.21.15.63:6789/0 conn(0x7febd80a3a90 legacy=0x7febd80a3e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:47:19.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:47:19.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 monclient: found mon.noname-a 2024-09-16T05:47:19.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:47:19.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:47:19.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd80a4ce0 con 0x7febd80a4580 2024-09-16T05:47:19.311 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febcc0017d0 con 0x7febd80a4580 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febd6ffd640 10 monclient: dump: 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.312 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 5 monclient: authenticate success, global_id 144177 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.301+0000 7febde55f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febcc001ab0 con 0x7febd80a4580 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:47:19.313 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd6ffd640 1 -- 172.21.15.63:0/585855147 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febcc003620 con 0x7febd80a4580 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd6ffd640 10 monclient: dump: 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:47:19.314 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient: get_monmap_and_config success 2024-09-16T05:47:19.315 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 4 set_mon_vals no callback set 2024-09-16T05:47:19.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient: shutdown 2024-09-16T05:47:19.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:47:19.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- 172.21.15.63:0/585855147 >> v1:172.21.15.6:6789/0 conn(0x7febd80a4580 legacy=0x7febd813bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:47:19.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- 172.21.15.63:0/585855147 shutdown_connections 2024-09-16T05:47:19.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- 172.21.15.63:0/585855147 >> 172.21.15.63:0/585855147 conn(0x7febd80a9ab0 msgr2=0x7febd80a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:47:19.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- 172.21.15.63:0/585855147 shutdown_connections 2024-09-16T05:47:19.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- 172.21.15.63:0/585855147 wait complete. 2024-09-16T05:47:19.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient: build_initial_monmap 2024-09-16T05:47:19.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient: monmap: 2024-09-16T05:47:19.320 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:47:19.320 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:47:19.321 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:47:19.321 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:47:19.321 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:47:19.321 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:47:19.321 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:47:19.323 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:47:19.324 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:47:19.324 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:47:19.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 Processor -- start 2024-09-16T05:47:19.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- start start 2024-09-16T05:47:19.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient: init 2024-09-16T05:47:19.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient: _reopen_session rank -1 2024-09-16T05:47:19.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:47:19.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient(hunting): picked mon.noname-b con 0x7febd80a3a90 addr v1:172.21.15.63:6789/0 2024-09-16T05:47:19.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient(hunting): picked mon.noname-a con 0x7febd80a4580 addr v1:172.21.15.6:6789/0 2024-09-16T05:47:19.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient(hunting): picked mon.noname-c con 0x7febd813d510 addr v1:172.21.15.6:6790/0 2024-09-16T05:47:19.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd81393d0 con 0x7febd80a4580 2024-09-16T05:47:19.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd813cbe0 con 0x7febd80a3a90 2024-09-16T05:47:19.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd80a0280 con 0x7febd813d510 2024-09-16T05:47:19.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient(hunting): _renew_subs 2024-09-16T05:47:19.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febde55f640 10 monclient(hunting): authenticate will time out at 1696.086499s 2024-09-16T05:47:19.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd7fff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7febd80a3a90 legacy=0x7febd809cd20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:47:19.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd77fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7febd80a4580 0x7febd809d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:53234/0 (socket says 172.21.15.63:53234) 2024-09-16T05:47:19.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd77fe640 1 -- 172.21.15.63:0/2857284858 learned_addr learned my addr 172.21.15.63:0/2857284858 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:47:19.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4179548594 0 0) 0x7febd80a0280 con 0x7febd813d510 2024-09-16T05:47:19.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:47:19.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:47:19.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:47:19.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 10 monclient(hunting): my global_id is 6911 2024-09-16T05:47:19.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febc4003530 con 0x7febd813d510 2024-09-16T05:47:19.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2236087527 0 0) 0x7febd81393d0 con 0x7febd80a4580 2024-09-16T05:47:19.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:47:19.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:47:19.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:47:19.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 10 monclient(hunting): my global_id is 144180 2024-09-16T05:47:19.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febd80a0280 con 0x7febd80a4580 2024-09-16T05:47:19.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.305+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3599376279 0 0) 0x7febc4003530 con 0x7febd813d510 2024-09-16T05:47:19.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febd81393d0 con 0x7febd813d510 2024-09-16T05:47:19.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febd00013f0 con 0x7febd813d510 2024-09-16T05:47:19.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:47:19.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:47:19.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient(hunting): dump: 2024-09-16T05:47:19.329 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:47:19.329 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:47:19.329 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.329 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.329 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:47:19.329 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:47:19.330 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:47:19.330 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:47:19.330 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:47:19.330 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:47:19.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:47:19.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1045173046 0 0) 0x7febd80a0280 con 0x7febd80a4580 2024-09-16T05:47:19.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febc4003530 con 0x7febd80a4580 2024-09-16T05:47:19.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febcc000ff0 con 0x7febd80a4580 2024-09-16T05:47:19.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:47:19.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:47:19.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient(hunting): dump: 2024-09-16T05:47:19.331 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:47:19.331 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:47:19.331 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.332 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.332 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:47:19.332 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:47:19.332 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:47:19.332 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:47:19.332 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:47:19.332 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:47:19.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4242870633 0 0) 0x7febd81393d0 con 0x7febd813d510 2024-09-16T05:47:19.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 >> v1:172.21.15.63:6789/0 conn(0x7febd80a3a90 legacy=0x7febd809cd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:47:19.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 >> v1:172.21.15.6:6789/0 conn(0x7febd80a4580 legacy=0x7febd809d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:47:19.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:47:19.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 monclient: found mon.c 2024-09-16T05:47:19.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:47:19.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:47:19.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd8137dd0 con 0x7febd813d510 2024-09-16T05:47:19.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient: _finish_auth 0 2024-09-16T05:47:19.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:47:19.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:47:19.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 5 monclient: authenticate success, global_id 6911 2024-09-16T05:47:19.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 10 monclient: _renew_subs 2024-09-16T05:47:19.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:47:19.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7febd81384d0 con 0x7febd813d510 2024-09-16T05:47:19.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 10 monclient: _renew_subs 2024-09-16T05:47:19.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:47:19.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7febd81390f0 con 0x7febd813d510 2024-09-16T05:47:19.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febd00019c0 con 0x7febd813d510 2024-09-16T05:47:19.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:47:19.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7febd0004d10 con 0x7febd813d510 2024-09-16T05:47:19.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:47:19.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:47:19.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd67fc640 4 set_mon_vals no callback set 2024-09-16T05:47:19.336 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 10 monclient: dump: 2024-09-16T05:47:19.336 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:47:19.336 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:47:19.336 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.336 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:47:19.336 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:47:19.337 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:47:19.337 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:47:19.337 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:47:19.337 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:47:19.337 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:47:19.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:47:19.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:47:19.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:47:19.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb8005090 con 0x7febd813d510 2024-09-16T05:47:19.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7febd000fab0 con 0x7febd813d510 2024-09-16T05:47:19.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7febd0044e20 con 0x7febd813d510 2024-09-16T05:47:19.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.309+0000 7febd7fff640 10 monclient: get_auth_request con 0x7febc4036c20 auth_method 0 2024-09-16T05:47:19.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.321+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7febd0015450 con 0x7febd813d510 2024-09-16T05:47:19.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.321+0000 7febd4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:47:19.339 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.321+0000 7febd4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:47:19.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.553+0000 7febde55f640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T05:47:19.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.553+0000 7febde55f640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T05:47:19.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.553+0000 7febde55f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:47:19.558 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.553+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) -- 0x7febb8005380 con 0x7febd813d510 2024-09-16T05:47:19.568 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.565+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v22) ==== 80+0+0 (unknown 1726994579 0 0) 0x7febd0014d50 con 0x7febd813d510 2024-09-16T05:47:19.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.565+0000 7febd4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T05:47:19.569 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.565+0000 7febd4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:47:19.730 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.725+0000 7febde55f640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T05:47:19.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.725+0000 7febde55f640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T05:47:19.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.725+0000 7febde55f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:47:19.731 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.725+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) -- 0x7febb8005770 con 0x7febd813d510 2024-09-16T05:47:19.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febd4ff9640 1 -- 172.21.15.63:0/2857284858 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v22) ==== 80+0+0 (unknown 1726994579 0 0) 0x7febd001d900 con 0x7febd813d510 2024-09-16T05:47:19.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febd4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T05:47:19.732 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febd4ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:47:19.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 >> v1:172.21.15.63:6812/3643207211 conn(0x7febc4036c20 legacy=0x7febc40390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:47:19.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febde55f640 10 monclient: shutdown 2024-09-16T05:47:19.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febde55f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:47:19.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 >> v1:172.21.15.6:6790/0 conn(0x7febd813d510 legacy=0x7febd81d36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:47:19.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 shutdown_connections 2024-09-16T05:47:19.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 >> 172.21.15.63:0/2857284858 conn(0x7febd80a9ab0 msgr2=0x7febd80a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:47:19.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 shutdown_connections 2024-09-16T05:47:19.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:47:19.729+0000 7febde55f640 1 -- 172.21.15.63:0/2857284858 wait complete. 2024-09-16T05:47:19.750 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13 2024-09-16T05:47:19.750 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-16T05:47:19.751 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:47:19.751 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T05:47:19.752 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T05:47:19.752 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T05:47:45.171 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:47:45.172 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:47:45.172 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:47:45.172 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:47:45.176 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:47:45.176 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T05:47:45.176 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T05:47:45.176 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:48:00.177 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T05:48:00.177 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T05:48:00.178 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:48:00.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:48:00.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient: get_monmap_and_config 2024-09-16T05:48:00.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient: build_initial_monmap 2024-09-16T05:48:00.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient: monmap: 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:00.313314+0000 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:00.313314+0000 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:00.312 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 1 Processor -- start 2024-09-16T05:48:00.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 1 -- start start 2024-09-16T05:48:00.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient: init 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd45409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd45409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd4540a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4540a1370 con 0x7fd4540a0da0 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd45409ead0 con 0x7fd45409aa30 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd45409edb0 con 0x7fd45409e230 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.308+0000 7fd45ba9d640 10 monclient(hunting): authenticate will time out at 1742.119911s 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45a03a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd4540a0da0 0x7fd45413d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58704/0 (socket says 172.21.15.6:58704) 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45a03a640 1 -- 172.21.15.6:0/1297015369 learned_addr learned my addr 172.21.15.6:0/1297015369 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998002968 0 0) 0x7fd4540a1370 con 0x7fd4540a0da0 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): my global_id is 144213 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd440003530 con 0x7fd4540a0da0 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819925738 0 0) 0x7fd45409edb0 con 0x7fd45409e230 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): my global_id is 6953 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4540a1370 con 0x7fd45409e230 2024-09-16T05:48:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3740132744 0 0) 0x7fd440003530 con 0x7fd4540a0da0 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 >> v1:172.21.15.6:6790/0 conn(0x7fd45409e230 legacy=0x7fd4541399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 >> v1:172.21.15.63:6789/0 conn(0x7fd45409aa30 legacy=0x7fd45409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _finish_hunting 0 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 monclient: found mon.noname-a 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45409dd50 con 0x7fd4540a0da0 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _finish_auth 0 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _check_auth_tickets 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd44c0019a0 con 0x7fd4540a0da0 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: dump: 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 5 monclient: authenticate success, global_id 144213 2024-09-16T05:48:00.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45a03a640 1 -- 172.21.15.6:0/1297015369 >> v1:172.21.15.6:6789/0 conn(0x7fd4540a0da0 legacy=0x7fd45413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45a03a640 1 -- 172.21.15.6:0/1297015369 >> v1:172.21.15.6:6789/0 conn(0x7fd4540a0da0 legacy=0x7fd45413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45a03a640 1 --1- 172.21.15.6:0/1297015369 >> v1:172.21.15.6:6789/0 conn(0x7fd4540a0da0 0x7fd45413d100 :-1 s=OPENED pgs=969 cs=1 l=1).handle_message read tag failed 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45a03a640 1 --1- 172.21.15.6:0/1297015369 >> v1:172.21.15.6:6789/0 conn(0x7fd4540a0da0 0x7fd45413d100 :-1 s=OPENED pgs=969 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: ms_handle_reset current mon v1:172.21.15.6:6789/0 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 >> v1:172.21.15.6:6789/0 conn(0x7fd4540a0da0 legacy=0x7fd45413d100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): picked mon.c con 0x7fd45409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): picked mon.a con 0x7fd440004390 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): picked mon.b con 0x7fd45409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd440003530 con 0x7fd440004390 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd45409ead0 con 0x7fd45409aa30 2024-09-16T05:48:00.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd4540a1370 con 0x7fd45409e230 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985033216 0 0) 0x7fd4540a1370 con 0x7fd45409e230 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.6:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fd44000edf0 con 0x7fd45409e230 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255157753 0 0) 0x7fd440003530 con 0x7fd440004390 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.6:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fd4540a1370 con 0x7fd440004390 2024-09-16T05:48:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 1321892997 0 0) 0x7fd44000edf0 con 0x7fd45409e230 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 >> v1:172.21.15.63:6789/0 conn(0x7fd45409aa30 legacy=0x7fd44000d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 >> v1:172.21.15.6:6789/0 conn(0x7fd440004390 legacy=0x7fd440009a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _finish_hunting 0 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 monclient: found mon.c 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=2+}) -- 0x7fd45409edb0 con 0x7fd45409e230 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _finish_auth 0 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: _check_auth_tickets 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:00.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd450001780 con 0x7fd45409e230 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: dump: 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.328 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 1 -- 172.21.15.6:0/1297015369 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd450001780 con 0x7fd45409e230 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd458837640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 4 set_mon_vals no callback set 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient: shutdown 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- 172.21.15.6:0/1297015369 >> v1:172.21.15.6:6790/0 conn(0x7fd45409e230 legacy=0x7fd440003c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- 172.21.15.6:0/1297015369 shutdown_connections 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- 172.21.15.6:0/1297015369 >> 172.21.15.6:0/1297015369 conn(0x7fd4540a1650 msgr2=0x7fd4540a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- 172.21.15.6:0/1297015369 shutdown_connections 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- 172.21.15.6:0/1297015369 wait complete. 2024-09-16T05:48:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient: build_initial_monmap 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient: monmap: 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 Processor -- start 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- start start 2024-09-16T05:48:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient: init 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd440004390 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd45409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd45409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd454121f50 con 0x7fd440004390 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd454124240 con 0x7fd45409e230 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4541223f0 con 0x7fd45409aa30 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.312+0000 7fd45ba9d640 10 monclient(hunting): authenticate will time out at 1742.123968s 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd459839640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd440004390 0x7fd454133830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58722/0 (socket says 172.21.15.6:58722) 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd459038640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd45409aa30 0x7fd4541cdf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39522/0 (socket says 172.21.15.6:39522) 2024-09-16T05:48:00.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd459839640 1 -- 172.21.15.6:0/3212458705 learned_addr learned my addr 172.21.15.6:0/3212458705 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3637309752 0 0) 0x7fd4541223f0 con 0x7fd45409aa30 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): my global_id is 6956 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd438003560 con 0x7fd45409aa30 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 938517812 0 0) 0x7fd454121f50 con 0x7fd440004390 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): my global_id is 144216 2024-09-16T05:48:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4541223f0 con 0x7fd440004390 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2349709038 0 0) 0x7fd4541223f0 con 0x7fd440004390 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd454121f50 con 0x7fd440004390 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3544944105 0 0) 0x7fd438003560 con 0x7fd45409aa30 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4541223f0 con 0x7fd45409aa30 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd450001ad0 con 0x7fd440004390 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): dump: 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd444001a60 con 0x7fd45409aa30 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): dump: 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 992935205 0 0) 0x7fd454124240 con 0x7fd45409e230 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient(hunting): my global_id is 154111 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd438003560 con 0x7fd45409e230 2024-09-16T05:48:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 463556390 0 0) 0x7fd454121f50 con 0x7fd440004390 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 >> v1:172.21.15.6:6790/0 conn(0x7fd45409aa30 legacy=0x7fd4541cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 >> v1:172.21.15.63:6789/0 conn(0x7fd45409e230 legacy=0x7fd4541d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 monclient: found mon.a 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45406e800 con 0x7fd440004390 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 5 monclient: authenticate success, global_id 144216 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 10 monclient: _renew_subs 2024-09-16T05:48:00.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4541c9320 con 0x7fd440004390 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 10 monclient: _renew_subs 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd454133f40 con 0x7fd440004390 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd4500038b0 con 0x7fd440004390 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd450003c00 con 0x7fd440004390 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44bfff640 4 set_mon_vals no callback set 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 10 monclient: dump: 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd450003da0 con 0x7fd440004390 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd45411c8b0 con 0x7fd440004390 2024-09-16T05:48:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fd450044c80 con 0x7fd440004390 2024-09-16T05:48:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.316+0000 7fd459038640 10 monclient: get_auth_request con 0x7fd438036be0 auth_method 0 2024-09-16T05:48:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.320+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd4500153f0 con 0x7fd440004390 2024-09-16T05:48:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.320+0000 7fd44a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.320+0000 7fd44a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.544+0000 7fd45ba9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:48:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.544+0000 7fd45ba9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:00.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.544+0000 7fd45ba9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.544+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd454134230 con 0x7fd440004390 2024-09-16T05:48:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.544+0000 7fd44a7fc640 1 -- 172.21.15.6:0/3212458705 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 622508945) 0x7fd450014cf0 con 0x7fd440004390 2024-09-16T05:48:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.544+0000 7fd44a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:00.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.544+0000 7fd44a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:00.548 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:48:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.548+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd438036be0 legacy=0x7fd4380390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.548+0000 7fd45ba9d640 10 monclient: shutdown 2024-09-16T05:48:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.548+0000 7fd45ba9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.548+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 >> v1:172.21.15.6:6789/0 conn(0x7fd440004390 legacy=0x7fd454133830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.548+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 shutdown_connections 2024-09-16T05:48:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.548+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 >> 172.21.15.6:0/3212458705 conn(0x7fd4540a1650 msgr2=0x7fd4540a4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.548+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 shutdown_connections 2024-09-16T05:48:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.548+0000 7fd45ba9d640 1 -- 172.21.15.6:0/3212458705 wait complete. 2024-09-16T05:48:00.567 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:48:00.568 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:48:00.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:48:00.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient: get_monmap_and_config 2024-09-16T05:48:00.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient: build_initial_monmap 2024-09-16T05:48:00.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient: monmap: 2024-09-16T05:48:00.694 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:00.694 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.695 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:00.696127+0000 2024-09-16T05:48:00.695 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:00.696127+0000 2024-09-16T05:48:00.695 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:00.695 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.695 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:00.695 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:00.695 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:00.695 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 1 Processor -- start 2024-09-16T05:48:00.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 1 -- start start 2024-09-16T05:48:00.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient: init 2024-09-16T05:48:00.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient(hunting): picked mon.noname-b con 0x7f35b809aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient(hunting): picked mon.noname-a con 0x7f35b809e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient(hunting): picked mon.noname-c con 0x7f35b80a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b80a1370 con 0x7f35b809e230 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b809ead0 con 0x7f35b80a0da0 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b809edb0 con 0x7f35b809aa30 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bfc5c640 10 monclient(hunting): authenticate will time out at 1742.503216s 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bd9f8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f35b809aa30 0x7f35b809ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39532/0 (socket says 172.21.15.6:39532) 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bd9f8640 1 -- 172.21.15.6:0/4225971606 learned_addr learned my addr 172.21.15.6:0/4225971606 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2695602451 0 0) 0x7f35b80a1370 con 0x7f35b809e230 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 10 monclient(hunting): my global_id is 144219 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35a4003530 con 0x7f35b809e230 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1959232258 0 0) 0x7f35b809edb0 con 0x7f35b809aa30 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 10 monclient(hunting): my global_id is 6959 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.692+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35b80a1370 con 0x7f35b809aa30 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 365277554 0 0) 0x7f35a4003530 con 0x7f35b809e230 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 >> v1:172.21.15.6:6790/0 conn(0x7f35b809aa30 legacy=0x7f35b809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 >> v1:172.21.15.63:6789/0 conn(0x7f35b80a0da0 legacy=0x7f35b813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: _finish_hunting 0 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 1 monclient: found mon.noname-a 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b809dd50 con 0x7f35b809e230 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: _finish_auth 0 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: _check_auth_tickets 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35a8001550 con 0x7f35b809e230 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: dump: 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 5 monclient: authenticate success, global_id 144219 2024-09-16T05:48:00.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35a8001800 con 0x7f35b809e230 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 1 -- 172.21.15.6:0/4225971606 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35a8001c10 con 0x7f35b809e230 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bc9f6640 10 monclient: dump: 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.702 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.703 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.703 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.703 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:00.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 4 set_mon_vals no callback set 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient: shutdown 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- 172.21.15.6:0/4225971606 >> v1:172.21.15.6:6789/0 conn(0x7f35b809e230 legacy=0x7f35b81399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- 172.21.15.6:0/4225971606 shutdown_connections 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- 172.21.15.6:0/4225971606 >> 172.21.15.6:0/4225971606 conn(0x7f35b80a1650 msgr2=0x7f35b80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- 172.21.15.6:0/4225971606 shutdown_connections 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- 172.21.15.6:0/4225971606 wait complete. 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient: build_initial_monmap 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient: monmap: 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:00.708 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 Processor -- start 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- start start 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient: init 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:00.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient(hunting): picked mon.noname-c con 0x7f35b809aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient(hunting): picked mon.noname-b con 0x7f35b809e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient(hunting): picked mon.noname-a con 0x7f35b80a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b80a0290 con 0x7f35b80a0da0 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b81348e0 con 0x7f35b809e230 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b8133580 con 0x7f35b809aa30 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bfc5c640 10 monclient(hunting): authenticate will time out at 1742.506987s 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bd9f8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f35b809aa30 0x7f35b813d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39534/0 (socket says 172.21.15.6:39534) 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35bd9f8640 1 -- 172.21.15.6:0/1674648769 learned_addr learned my addr 172.21.15.6:0/1674648769 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3205707841 0 0) 0x7f35b8133580 con 0x7f35b809aa30 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): my global_id is 6962 2024-09-16T05:48:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f359c003530 con 0x7f35b809aa30 2024-09-16T05:48:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3152309883 0 0) 0x7f35b80a0290 con 0x7f35b80a0da0 2024-09-16T05:48:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): my global_id is 144222 2024-09-16T05:48:00.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35b8133580 con 0x7f35b80a0da0 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 56256107 0 0) 0x7f35b81348e0 con 0x7f35b809e230 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): my global_id is 154117 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35b80a0290 con 0x7f35b809e230 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1468577408 0 0) 0x7f359c003530 con 0x7f35b809aa30 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35b81348e0 con 0x7f35b809aa30 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35b4001a20 con 0x7f35b809aa30 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): dump: 2024-09-16T05:48:00.712 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 516970986 0 0) 0x7f35b8133580 con 0x7f35b80a0da0 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f359c003530 con 0x7f35b80a0da0 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35b0001b80 con 0x7f35b80a0da0 2024-09-16T05:48:00.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient(hunting): dump: 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2326581061 0 0) 0x7f35b81348e0 con 0x7f35b809aa30 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 >> v1:172.21.15.63:6789/0 conn(0x7f35b809e230 legacy=0x7f35b81ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 >> v1:172.21.15.6:6789/0 conn(0x7f35b80a0da0 legacy=0x7f35b81d1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 1 monclient: found mon.c 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.696+0000 7f35ae7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b8134fd0 con 0x7f35b809aa30 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: _finish_auth 0 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 5 monclient: authenticate success, global_id 6962 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 10 monclient: _renew_subs 2024-09-16T05:48:00.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35b81352b0 con 0x7f35b809aa30 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 10 monclient: _renew_subs 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f35b81340e0 con 0x7f35b809aa30 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35b4001c80 con 0x7f35b809aa30 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35b4004c40 con 0x7f35b809aa30 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: dump: 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.716 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35affff640 4 set_mon_vals no callback set 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35a00065d0 con 0x7f35a0002870 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f35b400f9e0 con 0x7f35b809aa30 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f35b4044f40 con 0x7f35b809aa30 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884847159 0 0) 0x7f35a00065d0 con 0x7f35a0002870 2024-09-16T05:48:00.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient(hunting): my global_id is 144225 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35b81348e0 con 0x7f35a0002870 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35be1f9640 10 monclient: get_auth_request con 0x7f359c036bd0 auth_method 0 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3839715853 0 0) 0x7f35b81348e0 con 0x7f35a0002870 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35a00065d0 con 0x7f35a0002870 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35a8001610 con 0x7f35a0002870 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: dump: 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.718 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4260237047 0 0) 0x7f35a00065d0 con 0x7f35a0002870 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35a000a160 con 0x7f35a0002870 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f35a8001a70 con 0x7f35a0002870 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:00.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 >> v1:172.21.15.6:6789/0 conn(0x7f35a0002870 legacy=0x7f35a0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.700+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 >> v1:172.21.15.6:6789/0 conn(0x7f35a0002870 legacy=0x7f35a0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35a0009e50 con 0x7f35a0009110 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058790809 0 0) 0x7f35a0009e50 con 0x7f35a0009110 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient(hunting): my global_id is 144228 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35a00065d0 con 0x7f35a0009110 2024-09-16T05:48:00.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 540989051 0 0) 0x7f35a00065d0 con 0x7f35a0009110 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35a0009e50 con 0x7f35a0009110 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f35a8003840 con 0x7f35a0009110 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient: dump: 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1480526130 0 0) 0x7f35a0009e50 con 0x7f35a0009110 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35a000c3e0 con 0x7f35a0009110 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3964 (unknown 0 0 1946397220) 0x7f35a8003a70 con 0x7f35a0009110 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 >> v1:172.21.15.6:6789/0 conn(0x7f35a0009110 legacy=0x7f35a00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35ae7fc640 1 -- 172.21.15.6:0/1674648769 >> v1:172.21.15.6:6789/0 conn(0x7f35a0009110 legacy=0x7f35a00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:48:00.723 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 58, 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 129322, 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:48:00.724 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.725 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.726 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:48:00.727 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:00.728 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:48:00.729 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:00.730 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:00.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:48:00.732 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.733 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:48:00.734 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:48:00.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 >> v1:172.21.15.63:6812/3643207211 conn(0x7f359c036bd0 legacy=0x7f359c039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 10 monclient: shutdown 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 >> v1:172.21.15.6:6790/0 conn(0x7f35b809aa30 legacy=0x7f35b813d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35be1f9640 1 -- 172.21.15.6:0/1674648769 reap_dead start 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35be1f9640 1 -- 172.21.15.6:0/1674648769 reap_dead start 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 shutdown_connections 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 >> 172.21.15.6:0/1674648769 conn(0x7f35b80a1650 msgr2=0x7f35b80a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 shutdown_connections 2024-09-16T05:48:00.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.708+0000 7f35bfc5c640 1 -- 172.21.15.6:0/1674648769 wait complete. 2024-09-16T05:48:00.736 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:48:00.737 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:48:00.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.856+0000 7f358f41c640 10 monclient: get_monmap_and_config 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.856+0000 7f358f41c640 10 monclient: build_initial_monmap 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.856+0000 7f358f41c640 10 monclient: monmap: 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:00.862774+0000 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:00.862774+0000 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.862 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:00.862 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:00.862 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:00.862 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 Processor -- start 2024-09-16T05:48:00.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- start start 2024-09-16T05:48:00.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient: init 2024-09-16T05:48:00.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:00.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:48:00.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient(hunting): picked mon.noname-b con 0x7f358809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:00.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient(hunting): picked mon.noname-c con 0x7f358809bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:00.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient(hunting): picked mon.noname-a con 0x7f35881397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:00.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35880a1350 con 0x7f35881397f0 2024-09-16T05:48:00.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35880a10d0 con 0x7f358809bf20 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358809d940 con 0x7f358809bb40 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient(hunting): authenticate will time out at 1742.669410s 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358d1b8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f358809bb40 0x7f358809e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39542/0 (socket says 172.21.15.6:39542) 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358d1b8640 1 -- 172.21.15.6:0/1748487615 learned_addr learned my addr 172.21.15.6:0/1748487615 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 542050527 0 0) 0x7f358809d940 con 0x7f358809bb40 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient(hunting): my global_id is 6965 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3574005740 con 0x7f358809bb40 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545663777 0 0) 0x7f35880a1350 con 0x7f35881397f0 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient(hunting): my global_id is 144231 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f358809d940 con 0x7f35881397f0 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1177715103 0 0) 0x7f358809d940 con 0x7f35881397f0 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 >> v1:172.21.15.6:6790/0 conn(0x7f358809bb40 legacy=0x7f358809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 >> v1:172.21.15.63:6789/0 conn(0x7f358809bf20 legacy=0x7f358809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:48:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 monclient: found mon.noname-a 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358809dc20 con 0x7f35881397f0 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: _finish_auth 0 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3580001620 con 0x7f35881397f0 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: dump: 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.868 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.869 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.869 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.869 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.869 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.869 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.869 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.872 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 5 monclient: authenticate success, global_id 144231 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3580001890 con 0x7f35881397f0 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 1 -- 172.21.15.6:0/1748487615 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3580001c20 con 0x7f35881397f0 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f357ffff640 10 monclient: dump: 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.873 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 4 set_mon_vals no callback set 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 10 monclient: shutdown 2024-09-16T05:48:00.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:00.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- 172.21.15.6:0/1748487615 >> v1:172.21.15.6:6789/0 conn(0x7f35881397f0 legacy=0x7f358813bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- 172.21.15.6:0/1748487615 shutdown_connections 2024-09-16T05:48:00.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- 172.21.15.6:0/1748487615 >> 172.21.15.6:0/1748487615 conn(0x7f35880a1650 msgr2=0x7f35880a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:00.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- 172.21.15.6:0/1748487615 shutdown_connections 2024-09-16T05:48:00.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.860+0000 7f358f41c640 1 -- 172.21.15.6:0/1748487615 wait complete. 2024-09-16T05:48:00.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: build_initial_monmap 2024-09-16T05:48:00.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: monmap: 2024-09-16T05:48:00.880 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:00.880 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 1 Processor -- start 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 1 -- start start 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: init 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:00.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient(hunting): picked mon.noname-c con 0x7f358809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient(hunting): picked mon.noname-a con 0x7f358809bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient(hunting): picked mon.noname-b con 0x7f35881397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358812f280 con 0x7f358809bf20 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35880a3e60 con 0x7f35881397f0 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358811d540 con 0x7f358809bb40 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient(hunting): authenticate will time out at 1742.673263s 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358d1b8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f358809bb40 0x7f358812eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39556/0 (socket says 172.21.15.6:39556) 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358d1b8640 1 -- 172.21.15.6:0/1853774513 learned_addr learned my addr 172.21.15.6:0/1853774513 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 563830393 0 0) 0x7f358811d540 con 0x7f358809bb40 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): my global_id is 6968 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f356c003530 con 0x7f358809bb40 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 976474462 0 0) 0x7f358812f280 con 0x7f358809bf20 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): my global_id is 144234 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f358811d540 con 0x7f358809bf20 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099670577 0 0) 0x7f358809dc20 con 0x7f35881397f0 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): my global_id is 154123 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f358812f280 con 0x7f35881397f0 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1503557612 0 0) 0x7f356c003530 con 0x7f358809bb40 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f358809dc20 con 0x7f358809bb40 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 711778754 0 0) 0x7f358811d540 con 0x7f358809bf20 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f356c003530 con 0x7f358809bf20 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3584001c20 con 0x7f358809bb40 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): dump: 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.884 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3578001800 con 0x7f358809bf20 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): dump: 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.885 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2100460269 0 0) 0x7f358809dc20 con 0x7f358809bb40 2024-09-16T05:48:00.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 >> v1:172.21.15.63:6789/0 conn(0x7f35881397f0 legacy=0x7f35881d3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 >> v1:172.21.15.6:6789/0 conn(0x7f358809bf20 legacy=0x7f35881d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 monclient: found mon.c 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35880a39a0 con 0x7f358809bb40 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient: _finish_auth 0 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 5 monclient: authenticate success, global_id 6968 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: _renew_subs 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:00.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f358811d260 con 0x7f358809bb40 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: _renew_subs 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f35881384d0 con 0x7f358809bb40 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3584004b70 con 0x7f358809bb40 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3584004ec0 con 0x7f358809bb40 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357f7fe640 4 set_mon_vals no callback set 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient: dump: 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.888 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35680065d0 con 0x7f3568002870 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f358400fc60 con 0x7f358809bb40 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f3584045060 con 0x7f358809bb40 2024-09-16T05:48:00.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3069664237 0 0) 0x7f35680065d0 con 0x7f3568002870 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 10 monclient(hunting): my global_id is 6971 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.864+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f358809dc20 con 0x7f3568002870 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f358d9b9640 10 monclient: get_auth_request con 0x7f356c036c50 auth_method 0 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2210123833 0 0) 0x7f358809dc20 con 0x7f3568002870 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35680065d0 con 0x7f3568002870 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3578003a30 con 0x7f3568002870 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 10 monclient: dump: 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.890 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2490858725 0 0) 0x7f35680065d0 con 0x7f3568002870 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f356800a160 con 0x7f3568002870 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3578003980 con 0x7f3568002870 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:00.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 >> v1:172.21.15.6:6790/0 conn(0x7f3568002870 legacy=0x7f3568004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.868+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 >> v1:172.21.15.6:6790/0 conn(0x7f3568002870 legacy=0x7f3568004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f358f41c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3568009e50 con 0x7f3568009110 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2084183688 0 0) 0x7f3568009e50 con 0x7f3568009110 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient(hunting): my global_id is 6974 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35680065d0 con 0x7f3568009110 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2826515581 0 0) 0x7f35680065d0 con 0x7f3568009110 2024-09-16T05:48:00.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3568009e50 con 0x7f3568009110 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3578004170 con 0x7f3568009110 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient: dump: 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:00.893 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1846838634 0 0) 0x7f3568009e50 con 0x7f3568009110 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f356800c3e0 con 0x7f3568009110 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 1237881528) 0x7f3578003f90 con 0x7f3568009110 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 >> v1:172.21.15.6:6790/0 conn(0x7f3568009110 legacy=0x7f35680095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.872+0000 7f357dffb640 1 -- 172.21.15.6:0/1853774513 >> v1:172.21.15.6:6790/0 conn(0x7f3568009110 legacy=0x7f35680095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:00.894 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 58, 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 809928, 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:00.895 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:00.896 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:00.897 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:48:00.898 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:00.899 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:00.900 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:00.901 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:48:00.902 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.903 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:48:00.904 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.905 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:00.906 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:48:00.907 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:48:00.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 >> v1:172.21.15.63:6812/3643207211 conn(0x7f356c036c50 legacy=0x7f356c039110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358f41c640 10 monclient: shutdown 2024-09-16T05:48:00.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358f41c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:00.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 >> v1:172.21.15.6:6790/0 conn(0x7f358809bb40 legacy=0x7f358812eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:00.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358d9b9640 1 -- 172.21.15.6:0/1853774513 reap_dead start 2024-09-16T05:48:00.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358d9b9640 1 -- 172.21.15.6:0/1853774513 reap_dead start 2024-09-16T05:48:00.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 shutdown_connections 2024-09-16T05:48:00.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 >> 172.21.15.6:0/1853774513 conn(0x7f35880a1650 msgr2=0x7f35880a2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:00.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 shutdown_connections 2024-09-16T05:48:00.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:00.876+0000 7f358f41c640 1 -- 172.21.15.6:0/1853774513 wait complete. 2024-09-16T05:48:00.909 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:48:00.909 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:48:00.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:48:01.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient: get_monmap_and_config 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient: build_initial_monmap 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient: monmap: 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:01.024587+0000 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:01.024587+0000 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:01.023 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:01.024 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:01.024 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 1 Processor -- start 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 1 -- start start 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient: init 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient(hunting): picked mon.noname-b con 0x7f8b4409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient(hunting): picked mon.noname-a con 0x7f8b4409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient(hunting): picked mon.noname-c con 0x7f8b440a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:01.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b440a1370 con 0x7f8b4409e230 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b4409ead0 con 0x7f8b440a0da0 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b4409edb0 con 0x7f8b4409aa30 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b4a313640 10 monclient(hunting): authenticate will time out at 1742.831193s 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b43fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8b4409aa30 0x7f8b4409ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39580/0 (socket says 172.21.15.6:39580) 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b437fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8b4409e230 0x7f8b441399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58798/0 (socket says 172.21.15.6:58798) 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b43fff640 1 -- 172.21.15.6:0/2900209128 learned_addr learned my addr 172.21.15.6:0/2900209128 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253391579 0 0) 0x7f8b4409edb0 con 0x7f8b4409aa30 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 10 monclient(hunting): my global_id is 6977 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b24003530 con 0x7f8b4409aa30 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619868097 0 0) 0x7f8b440a1370 con 0x7f8b4409e230 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 10 monclient(hunting): my global_id is 144237 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.020+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b4409edb0 con 0x7f8b4409e230 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 273331270 0 0) 0x7f8b4409edb0 con 0x7f8b4409e230 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 >> v1:172.21.15.6:6790/0 conn(0x7f8b4409aa30 legacy=0x7f8b4409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 >> v1:172.21.15.63:6789/0 conn(0x7f8b440a0da0 legacy=0x7f8b4413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 1 monclient: found mon.noname-a 2024-09-16T05:48:01.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4409dd50 con 0x7f8b4409e230 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: _finish_auth 0 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b2c001980 con 0x7f8b4409e230 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 5 monclient: authenticate success, global_id 144237 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:01.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: dump: 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b2c001bf0 con 0x7f8b4409e230 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:01.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 4 set_mon_vals no callback set 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 1 -- 172.21.15.6:0/2900209128 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b2c001fc0 con 0x7f8b4409e230 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b42ffd640 10 monclient: dump: 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.036 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: shutdown 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- 172.21.15.6:0/2900209128 >> v1:172.21.15.6:6789/0 conn(0x7f8b4409e230 legacy=0x7f8b441399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- 172.21.15.6:0/2900209128 shutdown_connections 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- 172.21.15.6:0/2900209128 >> 172.21.15.6:0/2900209128 conn(0x7f8b440a1650 msgr2=0x7f8b440a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- 172.21.15.6:0/2900209128 shutdown_connections 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- 172.21.15.6:0/2900209128 wait complete. 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: build_initial_monmap 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: monmap: 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 Processor -- start 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- start start 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: init 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:48:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient(hunting): picked mon.noname-c con 0x7f8b4409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient(hunting): picked mon.noname-a con 0x7f8b4409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient(hunting): picked mon.noname-b con 0x7f8b440a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b440740c0 con 0x7f8b4409e230 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b441336b0 con 0x7f8b440a0da0 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b440a02e0 con 0x7f8b4409aa30 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient(hunting): authenticate will time out at 1742.834544s 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b43fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8b4409aa30 0x7f8b4409bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39582/0 (socket says 172.21.15.6:39582) 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b43fff640 1 -- 172.21.15.6:0/2756848930 learned_addr learned my addr 172.21.15.6:0/2756848930 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 352888148 0 0) 0x7f8b4409dd50 con 0x7f8b4409e230 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): my global_id is 144240 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b28003560 con 0x7f8b4409e230 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485426656 0 0) 0x7f8b440a02e0 con 0x7f8b4409aa30 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): my global_id is 6980 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b4409dd50 con 0x7f8b4409aa30 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2544540561 0 0) 0x7f8b28003560 con 0x7f8b4409e230 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b440a02e0 con 0x7f8b4409e230 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b2c001f40 con 0x7f8b4409e230 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): dump: 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3059267668 0 0) 0x7f8b4409dd50 con 0x7f8b4409aa30 2024-09-16T05:48:01.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b28003560 con 0x7f8b4409aa30 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b340017e0 con 0x7f8b4409aa30 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): dump: 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.042 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315091906 0 0) 0x7f8b441336b0 con 0x7f8b440a0da0 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient(hunting): my global_id is 154129 2024-09-16T05:48:01.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b4409dd50 con 0x7f8b440a0da0 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1771754873 0 0) 0x7f8b440a02e0 con 0x7f8b4409e230 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 >> v1:172.21.15.6:6790/0 conn(0x7f8b4409aa30 legacy=0x7f8b4409bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 >> v1:172.21.15.63:6789/0 conn(0x7f8b440a0da0 legacy=0x7f8b4409c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 monclient: found mon.a 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b441333d0 con 0x7f8b4409e230 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient: _finish_auth 0 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 5 monclient: authenticate success, global_id 144240 2024-09-16T05:48:01.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: _renew_subs 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4413d770 con 0x7f8b4409e230 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: _renew_subs 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b441d4900 con 0x7f8b4409e230 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b2c001bf0 con 0x7f8b4409e230 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b2c005060 con 0x7f8b4409e230 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b427fc640 4 set_mon_vals no callback set 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.024+0000 7f8b40ff9640 10 monclient: dump: 2024-09-16T05:48:01.045 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.046 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8b2c00fe00 con 0x7f8b4409e230 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b4a313640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b1c0065d0 con 0x7f8b1c002870 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f8b2c045150 con 0x7f8b4409e230 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b488b0640 10 monclient: get_auth_request con 0x7f8b28036cd0 auth_method 0 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 742431366 0 0) 0x7f8b1c0065d0 con 0x7f8b1c002870 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient(hunting): my global_id is 154132 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b440a02e0 con 0x7f8b1c002870 2024-09-16T05:48:01.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3131179676 0 0) 0x7f8b440a02e0 con 0x7f8b1c002870 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b1c0065d0 con 0x7f8b1c002870 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b34004990 con 0x7f8b1c002870 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient: dump: 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.048 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.049 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.049 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.049 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.049 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.049 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3742194663 0 0) 0x7f8b1c0065d0 con 0x7f8b1c002870 2024-09-16T05:48:01.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b1c00a160 con 0x7f8b1c002870 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8b34004980 con 0x7f8b1c002870 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 >> v1:172.21.15.63:6789/0 conn(0x7f8b1c002870 legacy=0x7f8b1c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.028+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 >> v1:172.21.15.63:6789/0 conn(0x7f8b1c002870 legacy=0x7f8b1c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b4a313640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b1c009e50 con 0x7f8b1c009110 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557704654 0 0) 0x7f8b1c009e50 con 0x7f8b1c009110 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient(hunting): my global_id is 154135 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b1c0065d0 con 0x7f8b1c009110 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4122468637 0 0) 0x7f8b1c0065d0 con 0x7f8b1c009110 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b1c009e50 con 0x7f8b1c009110 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8b34004fd0 con 0x7f8b1c009110 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient: dump: 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.052 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3026430682 0 0) 0x7f8b1c009e50 con 0x7f8b1c009110 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8b1c00c3e0 con 0x7f8b1c009110 2024-09-16T05:48:01.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3961 (unknown 0 0 3820166806) 0x7f8b340051f0 con 0x7f8b1c009110 2024-09-16T05:48:01.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:48:01.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:01.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 >> v1:172.21.15.63:6789/0 conn(0x7f8b1c009110 legacy=0x7f8b1c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b40ff9640 1 -- 172.21.15.6:0/2756848930 >> v1:172.21.15.63:6789/0 conn(0x7f8b1c009110 legacy=0x7f8b1c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:01.054 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:48:01.054 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:01.054 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:01.054 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 58, 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 15779, 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:48:01.055 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:48:01.056 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.057 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:48:01.058 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:01.059 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:01.060 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:01.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:01.062 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:01.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:01.064 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:01.065 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:01.066 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:01.066 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:01.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:01.066 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:48:01.066 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:48:01.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8b28036cd0 legacy=0x7f8b28039190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b4a313640 10 monclient: shutdown 2024-09-16T05:48:01.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b4a313640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 >> v1:172.21.15.6:6789/0 conn(0x7f8b4409e230 legacy=0x7f8b4409c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b488b0640 1 -- 172.21.15.6:0/2756848930 reap_dead start 2024-09-16T05:48:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b488b0640 1 -- 172.21.15.6:0/2756848930 reap_dead start 2024-09-16T05:48:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 shutdown_connections 2024-09-16T05:48:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.036+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 >> 172.21.15.6:0/2756848930 conn(0x7f8b440a1650 msgr2=0x7f8b440a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.040+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 shutdown_connections 2024-09-16T05:48:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.040+0000 7f8b4a313640 1 -- 172.21.15.6:0/2756848930 wait complete. 2024-09-16T05:48:01.068 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:48:01.069 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:48:01.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:48:01.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.180+0000 7ffb3cd2c640 10 monclient: get_monmap_and_config 2024-09-16T05:48:01.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.180+0000 7ffb3cd2c640 10 monclient: build_initial_monmap 2024-09-16T05:48:01.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.180+0000 7ffb3cd2c640 10 monclient: monmap: 2024-09-16T05:48:01.184 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:01.184 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.184 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:01.185936+0000 2024-09-16T05:48:01.184 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:01.185936+0000 2024-09-16T05:48:01.184 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:01.185 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.185 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:01.185 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:01.185 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:01.185 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.180+0000 7ffb3cd2c640 1 Processor -- start 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.180+0000 7ffb3cd2c640 1 -- start start 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.180+0000 7ffb3cd2c640 10 monclient: init 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): picked mon.noname-c con 0x7ffb3809aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): picked mon.noname-a con 0x7ffb3809e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): picked mon.noname-b con 0x7ffb380a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:01.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb380a1370 con 0x7ffb3809e230 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb3809ead0 con 0x7ffb3809aa30 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb3809edb0 con 0x7ffb380a0da0 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): authenticate will time out at 1742.992522s 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb36d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ffb380a0da0 0x7ffb3813d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39590/0 (socket says 172.21.15.6:39590) 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb35d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffb3809e230 0x7ffb381399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58808/0 (socket says 172.21.15.6:58808) 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb36d9d640 1 -- 172.21.15.6:0/3613700105 learned_addr learned my addr 172.21.15.6:0/3613700105 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3312751174 0 0) 0x7ffb3809edb0 con 0x7ffb380a0da0 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): my global_id is 6983 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb20003530 con 0x7ffb380a0da0 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3938922762 0 0) 0x7ffb380a1370 con 0x7ffb3809e230 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): my global_id is 144243 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb3809edb0 con 0x7ffb3809e230 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1842871512 0 0) 0x7ffb3809ead0 con 0x7ffb3809aa30 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient(hunting): my global_id is 154138 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb380a1370 con 0x7ffb3809aa30 2024-09-16T05:48:01.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 578853333 0 0) 0x7ffb3809edb0 con 0x7ffb3809e230 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 >> v1:172.21.15.6:6790/0 conn(0x7ffb380a0da0 legacy=0x7ffb3813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 >> v1:172.21.15.63:6789/0 conn(0x7ffb3809aa30 legacy=0x7ffb3809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: _finish_hunting 0 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 monclient: found mon.noname-a 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3809dd50 con 0x7ffb3809e230 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: _finish_auth 0 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: _check_auth_tickets 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ffb2c001600 con 0x7ffb3809e230 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:01.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: dump: 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 5 monclient: authenticate success, global_id 144243 2024-09-16T05:48:01.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb2c001860 con 0x7ffb3809e230 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 1 -- 172.21.15.6:0/3613700105 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ffb2c001c30 con 0x7ffb3809e230 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3559a640 10 monclient: dump: 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.196 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 4 set_mon_vals no callback set 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: shutdown 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/3613700105 >> v1:172.21.15.6:6789/0 conn(0x7ffb3809e230 legacy=0x7ffb381399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/3613700105 shutdown_connections 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/3613700105 >> 172.21.15.6:0/3613700105 conn(0x7ffb380a1650 msgr2=0x7ffb380a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/3613700105 shutdown_connections 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/3613700105 wait complete. 2024-09-16T05:48:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: build_initial_monmap 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: monmap: 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 Processor -- start 2024-09-16T05:48:01.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- start start 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: init 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): picked mon.noname-a con 0x7ffb3809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): picked mon.noname-c con 0x7ffb3809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): picked mon.noname-b con 0x7ffb380a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb3811c210 con 0x7ffb3809aa30 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb3811c4f0 con 0x7ffb380a0da0 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb381351c0 con 0x7ffb3809e230 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3cd2c640 10 monclient(hunting): authenticate will time out at 1742.995807s 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3659c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffb3809aa30 0x7ffb3813b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58820/0 (socket says 172.21.15.6:58820) 2024-09-16T05:48:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.184+0000 7ffb3659c640 1 -- 172.21.15.6:0/1810525991 learned_addr learned my addr 172.21.15.6:0/1810525991 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3111710223 0 0) 0x7ffb3811c210 con 0x7ffb3809aa30 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): my global_id is 144246 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb18003530 con 0x7ffb3809aa30 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980763441 0 0) 0x7ffb381351c0 con 0x7ffb3809e230 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): my global_id is 6986 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb3811c210 con 0x7ffb3809e230 2024-09-16T05:48:01.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1328026350 0 0) 0x7ffb18003530 con 0x7ffb3809aa30 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb381351c0 con 0x7ffb3809aa30 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 558836530 0 0) 0x7ffb3811c210 con 0x7ffb3809e230 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb18003530 con 0x7ffb3809e230 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 784151721 0 0) 0x7ffb3811c4f0 con 0x7ffb380a0da0 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): my global_id is 154141 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb3811c210 con 0x7ffb380a0da0 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ffb280019a0 con 0x7ffb3809aa30 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:01.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): dump: 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:01.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ffb2c001500 con 0x7ffb3809e230 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient(hunting): dump: 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.203 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 79897526 0 0) 0x7ffb18003530 con 0x7ffb3809e230 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 >> v1:172.21.15.63:6789/0 conn(0x7ffb380a0da0 legacy=0x7ffb381d1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 >> v1:172.21.15.6:6789/0 conn(0x7ffb3809aa30 legacy=0x7ffb3813b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 monclient: found mon.c 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3806e660 con 0x7ffb3809e230 2024-09-16T05:48:01.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient: _finish_auth 0 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 5 monclient: authenticate success, global_id 6986 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 10 monclient: _renew_subs 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffb380740c0 con 0x7ffb3809e230 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 10 monclient: _renew_subs 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ffb3813c1a0 con 0x7ffb3809e230 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb2c004b40 con 0x7ffb3809e230 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ffb2c004e90 con 0x7ffb3809e230 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb34d99640 4 set_mon_vals no callback set 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 10 monclient: dump: 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:01.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ffb2c0016e0 con 0x7ffb3809e230 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb1c005090 con 0x7ffb3809e230 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7ffb2c044c20 con 0x7ffb3809e230 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.188+0000 7ffb3659c640 10 monclient: get_auth_request con 0x7ffb18036c00 auth_method 0 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.192+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7ffb2c0151f0 con 0x7ffb3809e230 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.192+0000 7ffb26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:01.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.192+0000 7ffb26ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:01.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.416+0000 7ffb3cd2c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:48:01.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.416+0000 7ffb3cd2c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:48:01.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.416+0000 7ffb3cd2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:01.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.416+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ffb1c005b90 con 0x7ffb3809e230 2024-09-16T05:48:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb26ffd640 1 -- 172.21.15.6:0/1810525991 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7ffb2c018e40 con 0x7ffb3809e230 2024-09-16T05:48:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:48:01.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb26ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:01.422 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.422 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=11,logm=49} crc {auth=3514327616,config=555582061,health=765734218,kv=2361287224,logm=2922597891}) 2024-09-16T05:48:01.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=735135287,config=3541143443,health=765734218,kv=2361287224,logm=417983933}) 2024-09-16T05:48:01.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=11,logm=49} crc {auth=3514327616,config=555582061,health=765734218,kv=2361287224,logm=2922597891}) 2024-09-16T05:48:01.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=11,logm=47} crc {auth=2123588479,config=3541143443,health=1535110898,kv=2361287224,logm=769799317}) 2024-09-16T05:48:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 >> v1:172.21.15.63:6812/3643207211 conn(0x7ffb18036c00 legacy=0x7ffb180390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb3cd2c640 10 monclient: shutdown 2024-09-16T05:48:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb3cd2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 >> v1:172.21.15.6:6790/0 conn(0x7ffb3809e230 legacy=0x7ffb3813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 shutdown_connections 2024-09-16T05:48:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 >> 172.21.15.6:0/1810525991 conn(0x7ffb380a1650 msgr2=0x7ffb38071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:01.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 shutdown_connections 2024-09-16T05:48:01.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:01.420+0000 7ffb3cd2c640 1 -- 172.21.15.6:0/1810525991 wait complete. 2024-09-16T05:48:01.425 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.425 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=647439362}) 2024-09-16T05:48:01.426 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3230732361}) 2024-09-16T05:48:01.426 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.426 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=647439362}) 2024-09-16T05:48:01.426 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.420+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3496379743}) 2024-09-16T05:48:01.426 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853158579}) 2024-09-16T05:48:01.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3734485927}) 2024-09-16T05:48:01.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853158579}) 2024-09-16T05:48:01.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=583565414}) 2024-09-16T05:48:01.428 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=204449135}) 2024-09-16T05:48:01.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399772233}) 2024-09-16T05:48:01.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=204449135}) 2024-09-16T05:48:01.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.424+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3454378065}) 2024-09-16T05:48:01.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385580697}) 2024-09-16T05:48:01.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4017890089}) 2024-09-16T05:48:01.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385580697}) 2024-09-16T05:48:01.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1802918789}) 2024-09-16T05:48:01.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170970505}) 2024-09-16T05:48:01.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108145013}) 2024-09-16T05:48:01.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170970505}) 2024-09-16T05:48:01.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.428+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966225073}) 2024-09-16T05:48:01.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645121329}) 2024-09-16T05:48:01.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=233850341}) 2024-09-16T05:48:01.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645121329}) 2024-09-16T05:48:01.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=173720421}) 2024-09-16T05:48:01.436 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193292195}) 2024-09-16T05:48:01.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2338053471}) 2024-09-16T05:48:01.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193292195}) 2024-09-16T05:48:01.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.432+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1461559895}) 2024-09-16T05:48:01.438 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.438 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2231993197}) 2024-09-16T05:48:01.438 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399837969}) 2024-09-16T05:48:01.438 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.438 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2231993197}) 2024-09-16T05:48:01.438 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1710978012}) 2024-09-16T05:48:01.439 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:48:01.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642261117}) 2024-09-16T05:48:01.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748106750}) 2024-09-16T05:48:01.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642261117}) 2024-09-16T05:48:01.441 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.436+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284859958}) 2024-09-16T05:48:01.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709268462}) 2024-09-16T05:48:01.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3652595316}) 2024-09-16T05:48:01.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709268462}) 2024-09-16T05:48:01.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3611111158}) 2024-09-16T05:48:01.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168390019}) 2024-09-16T05:48:01.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3836366410}) 2024-09-16T05:48:01.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168390019}) 2024-09-16T05:48:01.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.440+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1930692279}) 2024-09-16T05:48:01.445 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.446 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399867814}) 2024-09-16T05:48:01.446 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1839438949}) 2024-09-16T05:48:01.446 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.446 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399867814}) 2024-09-16T05:48:01.446 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3347203700}) 2024-09-16T05:48:01.447 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.447 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612605817}) 2024-09-16T05:48:01.448 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1910335825}) 2024-09-16T05:48:01.448 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.448 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612605817}) 2024-09-16T05:48:01.448 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.444+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=615671508}) 2024-09-16T05:48:01.449 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.449 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717752553}) 2024-09-16T05:48:01.449 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991567304}) 2024-09-16T05:48:01.449 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.449 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717752553}) 2024-09-16T05:48:01.450 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1241287483}) 2024-09-16T05:48:01.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235643057}) 2024-09-16T05:48:01.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2056984294}) 2024-09-16T05:48:01.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235643057}) 2024-09-16T05:48:01.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.448+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3935306924}) 2024-09-16T05:48:01.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855198489}) 2024-09-16T05:48:01.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772655766}) 2024-09-16T05:48:01.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855198489}) 2024-09-16T05:48:01.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216478006}) 2024-09-16T05:48:01.455 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.455 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918613467}) 2024-09-16T05:48:01.455 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=438663847}) 2024-09-16T05:48:01.455 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.455 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918613467}) 2024-09-16T05:48:01.455 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.452+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172833747}) 2024-09-16T05:48:01.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437198155}) 2024-09-16T05:48:01.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3985621546}) 2024-09-16T05:48:01.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=437198155}) 2024-09-16T05:48:01.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2668864074}) 2024-09-16T05:48:01.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=992504698}) 2024-09-16T05:48:01.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2199311807}) 2024-09-16T05:48:01.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=992504698}) 2024-09-16T05:48:01.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4098736922}) 2024-09-16T05:48:01.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30766875}) 2024-09-16T05:48:01.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=515063745}) 2024-09-16T05:48:01.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30766875}) 2024-09-16T05:48:01.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.456+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278614015}) 2024-09-16T05:48:01.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743108568}) 2024-09-16T05:48:01.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968515070}) 2024-09-16T05:48:01.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743108568}) 2024-09-16T05:48:01.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618676705}) 2024-09-16T05:48:01.464 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977633382}) 2024-09-16T05:48:01.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2907241225}) 2024-09-16T05:48:01.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977633382}) 2024-09-16T05:48:01.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.460+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280651383}) 2024-09-16T05:48:01.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976434617}) 2024-09-16T05:48:01.467 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2678692691}) 2024-09-16T05:48:01.467 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.467 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976434617}) 2024-09-16T05:48:01.467 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4103054501}) 2024-09-16T05:48:01.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334958645}) 2024-09-16T05:48:01.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4085387522}) 2024-09-16T05:48:01.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334958645}) 2024-09-16T05:48:01.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.464+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=539158536}) 2024-09-16T05:48:01.470 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.470 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534555058}) 2024-09-16T05:48:01.470 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1353372711}) 2024-09-16T05:48:01.470 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.470 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534555058}) 2024-09-16T05:48:01.470 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784465097}) 2024-09-16T05:48:01.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577969605}) 2024-09-16T05:48:01.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=344332783}) 2024-09-16T05:48:01.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577969605}) 2024-09-16T05:48:01.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.468+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983893627}) 2024-09-16T05:48:01.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560350608}) 2024-09-16T05:48:01.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244527056}) 2024-09-16T05:48:01.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560350608}) 2024-09-16T05:48:01.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4247714680}) 2024-09-16T05:48:01.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120459647}) 2024-09-16T05:48:01.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649949695}) 2024-09-16T05:48:01.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120459647}) 2024-09-16T05:48:01.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.472+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513222741}) 2024-09-16T05:48:01.477 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.477 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807741574}) 2024-09-16T05:48:01.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1280034813}) 2024-09-16T05:48:01.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807741574}) 2024-09-16T05:48:01.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560501178}) 2024-09-16T05:48:01.479 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.479 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959845618}) 2024-09-16T05:48:01.479 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4165881466}) 2024-09-16T05:48:01.479 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959845618}) 2024-09-16T05:48:01.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.476+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040518354}) 2024-09-16T05:48:01.481 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.481 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453352276}) 2024-09-16T05:48:01.481 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=486952577}) 2024-09-16T05:48:01.481 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.481 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453352276}) 2024-09-16T05:48:01.481 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236924183}) 2024-09-16T05:48:01.483 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.483 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693912699}) 2024-09-16T05:48:01.483 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=383211717}) 2024-09-16T05:48:01.483 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.483 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693912699}) 2024-09-16T05:48:01.483 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.480+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2932016964}) 2024-09-16T05:48:01.485 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.485 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821195031}) 2024-09-16T05:48:01.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909860218}) 2024-09-16T05:48:01.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821195031}) 2024-09-16T05:48:01.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273385951}) 2024-09-16T05:48:01.487 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.487 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476910005}) 2024-09-16T05:48:01.487 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4175347870}) 2024-09-16T05:48:01.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476910005}) 2024-09-16T05:48:01.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.484+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2041809079}) 2024-09-16T05:48:01.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026670379}) 2024-09-16T05:48:01.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586898582}) 2024-09-16T05:48:01.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026670379}) 2024-09-16T05:48:01.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375481310}) 2024-09-16T05:48:01.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=535404541}) 2024-09-16T05:48:01.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=91480116}) 2024-09-16T05:48:01.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=535404541}) 2024-09-16T05:48:01.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.488+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2306102167}) 2024-09-16T05:48:01.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693843700}) 2024-09-16T05:48:01.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1557025267}) 2024-09-16T05:48:01.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693843700}) 2024-09-16T05:48:01.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1929249174}) 2024-09-16T05:48:01.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497413274}) 2024-09-16T05:48:01.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3721398541}) 2024-09-16T05:48:01.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497413274}) 2024-09-16T05:48:01.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.492+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1994179397}) 2024-09-16T05:48:01.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752921882}) 2024-09-16T05:48:01.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3862289138}) 2024-09-16T05:48:01.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752921882}) 2024-09-16T05:48:01.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3175770776}) 2024-09-16T05:48:01.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959805377}) 2024-09-16T05:48:01.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3583907627}) 2024-09-16T05:48:01.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959805377}) 2024-09-16T05:48:01.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1718685883}) 2024-09-16T05:48:01.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601707762}) 2024-09-16T05:48:01.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.496+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2707205798}) 2024-09-16T05:48:01.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601707762}) 2024-09-16T05:48:01.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616776009}) 2024-09-16T05:48:01.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303880406}) 2024-09-16T05:48:01.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2014752780}) 2024-09-16T05:48:01.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303880406}) 2024-09-16T05:48:01.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=336263153}) 2024-09-16T05:48:01.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1905820543}) 2024-09-16T05:48:01.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042468262}) 2024-09-16T05:48:01.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1905820543}) 2024-09-16T05:48:01.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.500+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173500252}) 2024-09-16T05:48:01.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849951140}) 2024-09-16T05:48:01.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457241645}) 2024-09-16T05:48:01.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849951140}) 2024-09-16T05:48:01.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235410763}) 2024-09-16T05:48:01.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770337553}) 2024-09-16T05:48:01.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1521851609}) 2024-09-16T05:48:01.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770337553}) 2024-09-16T05:48:01.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.504+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072735274}) 2024-09-16T05:48:01.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170140375}) 2024-09-16T05:48:01.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3247689319}) 2024-09-16T05:48:01.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170140375}) 2024-09-16T05:48:01.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=697643537}) 2024-09-16T05:48:01.512 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.512 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747068413}) 2024-09-16T05:48:01.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=437688831}) 2024-09-16T05:48:01.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747068413}) 2024-09-16T05:48:01.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.508+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732137136}) 2024-09-16T05:48:01.514 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.514 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856670333}) 2024-09-16T05:48:01.514 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422343539}) 2024-09-16T05:48:01.514 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856670333}) 2024-09-16T05:48:01.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1024378359}) 2024-09-16T05:48:01.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748850863}) 2024-09-16T05:48:01.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546451894}) 2024-09-16T05:48:01.517 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.517 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748850863}) 2024-09-16T05:48:01.517 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.512+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2996188287}) 2024-09-16T05:48:01.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.516+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.516+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=54,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=5} crc {logm=10331936,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3176912719,mon_config_key=782542404,monmap=3625042558,osd_metadata=678159955}) 2024-09-16T05:48:01.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.516+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=3} crc {logm=3278798864,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=1531317948,mon_config_key=782542404,monmap=3625042558,osd_metadata=304885223}) 2024-09-16T05:48:01.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.516+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.516+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=54,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=5} crc {logm=10331936,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3176912719,mon_config_key=782542404,monmap=3625042558,osd_metadata=678159955}) 2024-09-16T05:48:01.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.516+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=51,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=2632792399,mdsmap=604045574,mgr=1552851386,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3176912719,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393}) 2024-09-16T05:48:01.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.520+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.520+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=2,osdmap=96} crc {osd_metadata=610114474,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1011808690}) 2024-09-16T05:48:01.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.520+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=2,osdmap=96} crc {osd_metadata=610114475,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=3129514813}) 2024-09-16T05:48:01.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.520+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.523 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.520+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=2,osdmap=96} crc {osd_metadata=610114474,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1011808690}) 2024-09-16T05:48:01.523 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.520+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=2,osdmap=96} crc {osd_metadata=610114474,osd_pg_creating=1754493877,osd_snap=723727393,osdmap=3762411929}) 2024-09-16T05:48:01.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.524+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.524+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1759570463}) 2024-09-16T05:48:01.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.524+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1978272538}) 2024-09-16T05:48:01.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.524+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.524+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1759570463}) 2024-09-16T05:48:01.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.524+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4093587869}) 2024-09-16T05:48:01.530 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.528+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.530 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.528+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1643960481}) 2024-09-16T05:48:01.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.528+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=562693834}) 2024-09-16T05:48:01.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.528+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.528+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1643960481}) 2024-09-16T05:48:01.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.528+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=454895270}) 2024-09-16T05:48:01.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.540+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.540+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=69} crc {osdmap=1551171425}) 2024-09-16T05:48:01.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.540+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=67} crc {osdmap=3578551861}) 2024-09-16T05:48:01.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.540+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:48:01.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.540+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=69} crc {osdmap=1551171425}) 2024-09-16T05:48:01.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:48:01.540+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=69} crc {osdmap=1242913564}) 2024-09-16T05:48:01.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:48:01.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:48:01.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:48:01.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ c=''\''allow command service with prefix=list, allow command quorum_status'\''' 2024-09-16T05:48:01.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 0 2024-09-16T05:48:01.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 2024-09-16T05:48:01.873 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=0 2024-09-16T05:48:01.873 INFO:tasks.workunit.client.0.smithi063.stdout:ceph auth get-or-create client.bar mon 'allow command service with prefix=list, allow command quorum_status' > /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T05:48:01.874 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T05:48:01.874 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T05:48:02.425 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=0 2024-09-16T05:48:02.425 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T05:48:02.426 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:48:02.426 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 0 2024-09-16T05:48:02.426 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T05:48:02.426 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-16T05:48:02.426 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=0 2024-09-16T05:48:02.426 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T05:48:02.426 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T05:48:02.440 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:48:02.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:48:02.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient: get_monmap_and_config 2024-09-16T05:48:02.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient: build_initial_monmap 2024-09-16T05:48:02.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient: monmap: 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:02.572363+0000 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:02.572363+0000 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:02.571 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 1 Processor -- start 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 1 -- start start 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient: init 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient(hunting): picked mon.noname-c con 0x7fde4409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient(hunting): picked mon.noname-a con 0x7fde4409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient(hunting): picked mon.noname-b con 0x7fde440a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde440a1370 con 0x7fde4409e230 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde4409ead0 con 0x7fde4409aa30 2024-09-16T05:48:02.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde4409edb0 con 0x7fde440a0da0 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 10 monclient(hunting): authenticate will time out at 1744.378994s 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde427fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fde4409e230 0x7fde441399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58830/0 (socket says 172.21.15.6:58830) 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde427fc640 1 -- 172.21.15.6:0/287168097 learned_addr learned my addr 172.21.15.6:0/287168097 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1470042289 0 0) 0x7fde440a1370 con 0x7fde4409e230 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): my global_id is 144264 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde30003530 con 0x7fde4409e230 2024-09-16T05:48:02.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934963149 0 0) 0x7fde4409ead0 con 0x7fde4409aa30 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): my global_id is 154159 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde440a1370 con 0x7fde4409aa30 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815386508 0 0) 0x7fde4409edb0 con 0x7fde440a0da0 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient(hunting): my global_id is 7004 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde4409ead0 con 0x7fde440a0da0 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2076995520 0 0) 0x7fde30003530 con 0x7fde4409e230 2024-09-16T05:48:02.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 >> v1:172.21.15.6:6790/0 conn(0x7fde440a0da0 legacy=0x7fde4413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 >> v1:172.21.15.63:6789/0 conn(0x7fde4409aa30 legacy=0x7fde4409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 monclient: found mon.noname-a 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4409dd50 con 0x7fde4409e230 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient: _finish_auth 0 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 5 monclient: authenticate success, global_id 144264 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde34001620 con 0x7fde4409e230 2024-09-16T05:48:02.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde41ffb640 10 monclient: dump: 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.568+0000 7fde492d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde340018d0 con 0x7fde4409e230 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde41ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde41ffb640 1 -- 172.21.15.6:0/287168097 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde34001ca0 con 0x7fde4409e230 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 4 set_mon_vals no callback set 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde41ffb640 10 monclient: dump: 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.579 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: shutdown 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- 172.21.15.6:0/287168097 >> v1:172.21.15.6:6789/0 conn(0x7fde4409e230 legacy=0x7fde441399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- 172.21.15.6:0/287168097 shutdown_connections 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- 172.21.15.6:0/287168097 >> 172.21.15.6:0/287168097 conn(0x7fde440a1650 msgr2=0x7fde440a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- 172.21.15.6:0/287168097 shutdown_connections 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- 172.21.15.6:0/287168097 wait complete. 2024-09-16T05:48:02.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: build_initial_monmap 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: monmap: 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:02.581 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 Processor -- start 2024-09-16T05:48:02.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- start start 2024-09-16T05:48:02.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: init 2024-09-16T05:48:02.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:02.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:48:02.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient(hunting): picked mon.noname-b con 0x7fde4409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:02.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient(hunting): picked mon.noname-c con 0x7fde4409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:02.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient(hunting): picked mon.noname-a con 0x7fde440a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde4413cd90 con 0x7fde440a0da0 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde4413d070 con 0x7fde4409aa30 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde4413dac0 con 0x7fde4409e230 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient(hunting): authenticate will time out at 1744.382159s 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde427fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fde4409e230 0x7fde441d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39602/0 (socket says 172.21.15.6:39602) 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde427fc640 1 -- 172.21.15.6:0/2032827162 learned_addr learned my addr 172.21.15.6:0/2032827162 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2600534213 0 0) 0x7fde4413dac0 con 0x7fde4409e230 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): my global_id is 7007 2024-09-16T05:48:02.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde24003530 con 0x7fde4409e230 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515956615 0 0) 0x7fde4413cd90 con 0x7fde440a0da0 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): my global_id is 144267 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde4413dac0 con 0x7fde440a0da0 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713213443 0 0) 0x7fde4413d070 con 0x7fde4409aa30 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): my global_id is 154162 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde4413cd90 con 0x7fde4409aa30 2024-09-16T05:48:02.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3995809989 0 0) 0x7fde24003530 con 0x7fde4409e230 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde4413d070 con 0x7fde4409e230 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde34001c20 con 0x7fde4409e230 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): dump: 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.588 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 37227464 0 0) 0x7fde4413dac0 con 0x7fde440a0da0 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde24003530 con 0x7fde440a0da0 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde3c001bc0 con 0x7fde440a0da0 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient(hunting): dump: 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.589 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 620149318 0 0) 0x7fde4413d070 con 0x7fde4409e230 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 >> v1:172.21.15.63:6789/0 conn(0x7fde4409aa30 legacy=0x7fde441cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 >> v1:172.21.15.6:6789/0 conn(0x7fde440a0da0 legacy=0x7fde441d4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 monclient: found mon.c 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:02.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4406ec60 con 0x7fde4409e230 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 5 monclient: authenticate success, global_id 7007 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: _renew_subs 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fde4413cab0 con 0x7fde4409e230 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: _renew_subs 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fde4409cd50 con 0x7fde4409e230 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde34004c50 con 0x7fde4409e230 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fde34004fa0 con 0x7fde4409e230 2024-09-16T05:48:02.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde417fa640 4 set_mon_vals no callback set 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde3b7fe640 10 monclient: dump: 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.592 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.576+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fde3400fba0 con 0x7fde4409e230 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.572+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde28005090 con 0x7fde4409e230 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.576+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fde34044e00 con 0x7fde4409e230 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.576+0000 7fde42ffd640 10 monclient: get_auth_request con 0x7fde24036970 auth_method 0 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.580+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fde34074030 con 0x7fde4409e230 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.580+0000 7fde3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:02.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.580+0000 7fde3b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:02.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.800+0000 7fde492d5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:48:02.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.800+0000 7fde492d5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:02.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.800+0000 7fde492d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:02.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.800+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde28005380 con 0x7fde4409e230 2024-09-16T05:48:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde3b7fe640 1 -- 172.21.15.6:0/2032827162 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 577525703) 0x7fde34015490 con 0x7fde4409e230 2024-09-16T05:48:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:02.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde3b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:02.806 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:48:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 >> v1:172.21.15.63:6812/3643207211 conn(0x7fde24036970 legacy=0x7fde24038e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde492d5640 10 monclient: shutdown 2024-09-16T05:48:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde492d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 >> v1:172.21.15.6:6790/0 conn(0x7fde4409e230 legacy=0x7fde441d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 shutdown_connections 2024-09-16T05:48:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 >> 172.21.15.6:0/2032827162 conn(0x7fde440a1650 msgr2=0x7fde4413e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 shutdown_connections 2024-09-16T05:48:02.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.804+0000 7fde492d5640 1 -- 172.21.15.6:0/2032827162 wait complete. 2024-09-16T05:48:02.824 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:48:02.824 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:48:02.824 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:48:02.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:48:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient: get_monmap_and_config 2024-09-16T05:48:02.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient: build_initial_monmap 2024-09-16T05:48:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient: monmap: 2024-09-16T05:48:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:02.952499+0000 2024-09-16T05:48:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:02.952499+0000 2024-09-16T05:48:02.951 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:02.952 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:02.952 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 1 Processor -- start 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 1 -- start start 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient: init 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient(hunting): picked mon.noname-c con 0x7f0ef409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ef409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient(hunting): picked mon.noname-b con 0x7f0ef40a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef40a1370 con 0x7f0ef409e230 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef409ead0 con 0x7f0ef409aa30 2024-09-16T05:48:02.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef409edb0 con 0x7f0ef40a0da0 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 10 monclient(hunting): authenticate will time out at 1744.759140s 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef97a8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0ef40a0da0 0x7f0ef413d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39606/0 (socket says 172.21.15.6:39606) 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0ef409e230 0x7f0ef41399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58840/0 (socket says 172.21.15.6:58840) 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef97a8640 1 -- 172.21.15.6:0/3289432801 learned_addr learned my addr 172.21.15.6:0/3289432801 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3600915064 0 0) 0x7f0ef40a1370 con 0x7f0ef409e230 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient(hunting): my global_id is 144270 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ee0003530 con 0x7f0ef409e230 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020618516 0 0) 0x7f0ef409edb0 con 0x7f0ef40a0da0 2024-09-16T05:48:02.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient(hunting): my global_id is 7010 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ef40a1370 con 0x7f0ef40a0da0 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3932166335 0 0) 0x7f0ee0003530 con 0x7f0ef409e230 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 >> v1:172.21.15.6:6790/0 conn(0x7f0ef40a0da0 legacy=0x7f0ef413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 >> v1:172.21.15.63:6789/0 conn(0x7f0ef409aa30 legacy=0x7f0ef409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 monclient: found mon.noname-a 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:02.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef409dd50 con 0x7f0ef409e230 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient: _finish_auth 0 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ee4001550 con 0x7f0ef409e230 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 5 monclient: authenticate success, global_id 144270 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0efb20b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.948+0000 7f0ef37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef37fe640 10 monclient: dump: 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.962 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ee40017b0 con 0x7f0ef409e230 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef37fe640 1 -- 172.21.15.6:0/3289432801 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ee4001b40 con 0x7f0ef409e230 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef37fe640 10 monclient: dump: 2024-09-16T05:48:02.963 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 4 set_mon_vals no callback set 2024-09-16T05:48:02.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: shutdown 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- 172.21.15.6:0/3289432801 >> v1:172.21.15.6:6789/0 conn(0x7f0ef409e230 legacy=0x7f0ef41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- 172.21.15.6:0/3289432801 shutdown_connections 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- 172.21.15.6:0/3289432801 >> 172.21.15.6:0/3289432801 conn(0x7f0ef40a1650 msgr2=0x7f0ef40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- 172.21.15.6:0/3289432801 shutdown_connections 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- 172.21.15.6:0/3289432801 wait complete. 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: build_initial_monmap 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: monmap: 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:02.965 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 Processor -- start 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- start start 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: init 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient(hunting): picked mon.noname-b con 0x7f0ef409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient(hunting): picked mon.noname-c con 0x7f0ef409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ef40a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:02.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef411c8f0 con 0x7f0ef40a0da0 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef413d9f0 con 0x7f0ef409aa30 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef406ea80 con 0x7f0ef409e230 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient(hunting): authenticate will time out at 1744.762177s 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0ef409e230 0x7f0ef413b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39610/0 (socket says 172.21.15.6:39610) 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef3fff640 1 -- 172.21.15.6:0/3994407870 learned_addr learned my addr 172.21.15.6:0/3994407870 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef97a8640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0ef40a0da0 0x7f0ef413c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58842/0 (socket says 172.21.15.6:58842) 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 765820197 0 0) 0x7f0ef406ea80 con 0x7f0ef409e230 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): my global_id is 7013 2024-09-16T05:48:02.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ed4003530 con 0x7f0ef409e230 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3329607011 0 0) 0x7f0ef411c8f0 con 0x7f0ef40a0da0 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): my global_id is 144273 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ef406ea80 con 0x7f0ef40a0da0 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946972796 0 0) 0x7f0ef413d9f0 con 0x7f0ef409aa30 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): my global_id is 154168 2024-09-16T05:48:02.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ef411c8f0 con 0x7f0ef409aa30 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 345958848 0 0) 0x7f0ed4003530 con 0x7f0ef409e230 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ef413d9f0 con 0x7f0ef409e230 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ee4001490 con 0x7f0ef409e230 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): dump: 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.969 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3505917829 0 0) 0x7f0ef406ea80 con 0x7f0ef40a0da0 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ed4003530 con 0x7f0ef40a0da0 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0eec0018c0 con 0x7f0ef40a0da0 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient(hunting): dump: 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.970 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3777331045 0 0) 0x7f0ef413d9f0 con 0x7f0ef409e230 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 >> v1:172.21.15.63:6789/0 conn(0x7f0ef409aa30 legacy=0x7f0ef413b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 >> v1:172.21.15.6:6789/0 conn(0x7f0ef40a0da0 legacy=0x7f0ef413c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient: _finish_hunting 0 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 monclient: found mon.c 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:02.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef40740c0 con 0x7f0ef409e230 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient: _finish_auth 0 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient: _check_auth_tickets 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 5 monclient: authenticate success, global_id 7013 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: _renew_subs 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef411c610 con 0x7f0ef409e230 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: _renew_subs 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ef411b1a0 con 0x7f0ef409e230 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ee40017b0 con 0x7f0ef409e230 2024-09-16T05:48:02.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ee4004aa0 con 0x7f0ef409e230 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef17fa640 10 monclient: dump: 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.973 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.952+0000 7f0ef2ffd640 4 set_mon_vals no callback set 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0efb20b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ed80065d0 con 0x7f0ed8002870 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0ee4004aa0 con 0x7f0ef409e230 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f0ee4044bf0 con 0x7f0ef409e230 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595821254 0 0) 0x7f0ed80065d0 con 0x7f0ed8002870 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient(hunting): my global_id is 144276 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ef413d9f0 con 0x7f0ed8002870 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef97a8640 10 monclient: get_auth_request con 0x7f0ed4036960 auth_method 0 2024-09-16T05:48:02.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3961299531 0 0) 0x7f0ef413d9f0 con 0x7f0ed8002870 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ed80065d0 con 0x7f0ed8002870 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0edc003b20 con 0x7f0ed8002870 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient: dump: 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.975 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.976 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 292040257 0 0) 0x7f0ed80065d0 con 0x7f0ed8002870 2024-09-16T05:48:02.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ed800a160 con 0x7f0ed8002870 2024-09-16T05:48:02.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0edc005100 con 0x7f0ed8002870 2024-09-16T05:48:02.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:02.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:02.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 >> v1:172.21.15.6:6789/0 conn(0x7f0ed8002870 legacy=0x7f0ed8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.956+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 >> v1:172.21.15.6:6789/0 conn(0x7f0ed8002870 legacy=0x7f0ed8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:02.977 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=0 2024-09-16T05:48:02.978 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T05:48:02.978 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:48:02.978 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 13 2024-09-16T05:48:02.978 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 2024-09-16T05:48:02.978 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:48:02.978 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T05:48:02.978 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T05:48:02.979 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T05:48:02.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.960+0000 7f0efb20b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:48:02.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.960+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ed8009e50 con 0x7f0ed8009110 2024-09-16T05:48:02.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.960+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4015033561 0 0) 0x7f0ed8009e50 con 0x7f0ed8009110 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.960+0000 7f0ef17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.960+0000 7f0ef17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.960+0000 7f0ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.960+0000 7f0ef17fa640 10 monclient(hunting): my global_id is 144279 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.960+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ed80065d0 con 0x7f0ed8009110 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1694609509 0 0) 0x7f0ed80065d0 con 0x7f0ed8009110 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ed8009e50 con 0x7f0ed8009110 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0edc005750 con 0x7f0ed8009110 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:02.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 10 monclient: dump: 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4157091614 0 0) 0x7f0ed8009e50 con 0x7f0ed8009110 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ed800c3e0 con 0x7f0ed8009110 2024-09-16T05:48:02.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3964 (unknown 0 0 3657369278) 0x7f0edc0055b0 con 0x7f0ed8009110 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 >> v1:172.21.15.6:6789/0 conn(0x7f0ed8009110 legacy=0x7f0ed80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef17fa640 1 -- 172.21.15.6:0/3994407870 >> v1:172.21.15.6:6789/0 conn(0x7f0ed8009110 legacy=0x7f0ed80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 58, 2024-09-16T05:48:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 131578, 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:48:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:48:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:48:02.994 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:48:02.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0ed4036960 legacy=0x7f0ed4038e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0efb20b640 10 monclient: shutdown 2024-09-16T05:48:02.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0efb20b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 >> v1:172.21.15.6:6790/0 conn(0x7f0ef409e230 legacy=0x7f0ef413b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef97a8640 1 -- 172.21.15.6:0/3994407870 reap_dead start 2024-09-16T05:48:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0ef97a8640 1 -- 172.21.15.6:0/3994407870 reap_dead start 2024-09-16T05:48:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 shutdown_connections 2024-09-16T05:48:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 >> 172.21.15.6:0/3994407870 conn(0x7f0ef40a1650 msgr2=0x7f0ef4071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 shutdown_connections 2024-09-16T05:48:02.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:02.964+0000 7f0efb20b640 1 -- 172.21.15.6:0/3994407870 wait complete. 2024-09-16T05:48:02.996 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:48:02.996 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:48:02.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:48:03.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.108+0000 7f240bb1a640 10 monclient: get_monmap_and_config 2024-09-16T05:48:03.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.108+0000 7f240bb1a640 10 monclient: build_initial_monmap 2024-09-16T05:48:03.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.108+0000 7f240bb1a640 10 monclient: monmap: 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:03.114424+0000 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:03.114424+0000 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:03.113 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 Processor -- start 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- start start 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient: init 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient(hunting): picked mon.noname-b con 0x7f240406dd90 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient(hunting): picked mon.noname-a con 0x7f24040a3d60 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient(hunting): picked mon.noname-c con 0x7f240413d110 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:03.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24040a41b0 con 0x7f24040a3d60 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240409ff10 con 0x7f240413d110 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24040a4c80 con 0x7f240406dd90 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient(hunting): authenticate will time out at 1744.921157s 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24090b5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f24040a3d60 0x7f240413b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58852/0 (socket says 172.21.15.6:58852) 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24098b6640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f240406dd90 0x7f24040a2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39616/0 (socket says 172.21.15.6:39616) 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24090b5640 1 -- 172.21.15.6:0/3445591695 learned_addr learned my addr 172.21.15.6:0/3445591695 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2675519211 0 0) 0x7f24040a4c80 con 0x7f240406dd90 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): my global_id is 7022 2024-09-16T05:48:03.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23f0003530 con 0x7f240406dd90 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2836762573 0 0) 0x7f24040a41b0 con 0x7f24040a3d60 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): my global_id is 144288 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24040a4c80 con 0x7f24040a3d60 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214924598 0 0) 0x7f240409ff10 con 0x7f240413d110 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient(hunting): my global_id is 154177 2024-09-16T05:48:03.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24040a41b0 con 0x7f240413d110 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 969717621 0 0) 0x7f24040a4c80 con 0x7f24040a3d60 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 >> v1:172.21.15.6:6790/0 conn(0x7f240406dd90 legacy=0x7f24040a2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 >> v1:172.21.15.63:6789/0 conn(0x7f240413d110 legacy=0x7f240413f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: _finish_hunting 0 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 monclient: found mon.noname-a 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240406e950 con 0x7f24040a3d60 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: _finish_auth 0 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: _check_auth_tickets 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23f4001600 con 0x7f24040a3d60 2024-09-16T05:48:03.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: dump: 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 5 monclient: authenticate success, global_id 144288 2024-09-16T05:48:03.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23f4001860 con 0x7f24040a3d60 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 1 -- 172.21.15.6:0/3445591695 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23f4001cb0 con 0x7f24040a3d60 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f24088b4640 10 monclient: dump: 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.129 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 4 set_mon_vals no callback set 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient: shutdown 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- 172.21.15.6:0/3445591695 >> v1:172.21.15.6:6789/0 conn(0x7f24040a3d60 legacy=0x7f240413b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- 172.21.15.6:0/3445591695 shutdown_connections 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- 172.21.15.6:0/3445591695 >> 172.21.15.6:0/3445591695 conn(0x7f24040a59a0 msgr2=0x7f24040a5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- 172.21.15.6:0/3445591695 shutdown_connections 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- 172.21.15.6:0/3445591695 wait complete. 2024-09-16T05:48:03.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient: build_initial_monmap 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient: monmap: 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 Processor -- start 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 1 -- start start 2024-09-16T05:48:03.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.112+0000 7f240bb1a640 10 monclient: init 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient(hunting): picked mon.noname-a con 0x7f240406dd90 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient(hunting): picked mon.noname-c con 0x7f24040a3d60 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient(hunting): picked mon.noname-b con 0x7f240413d110 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24040a8240 con 0x7f240406dd90 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24040a0100 con 0x7f240413d110 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240409e820 con 0x7f24040a3d60 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient(hunting): authenticate will time out at 1744.924245s 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f24090b5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f24040a3d60 0x7f24041d04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39622/0 (socket says 172.21.15.6:39622) 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f24090b5640 1 -- 172.21.15.6:0/3834851310 learned_addr learned my addr 172.21.15.6:0/3834851310 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:03.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 7912267 0 0) 0x7f240409e820 con 0x7f24040a3d60 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): my global_id is 7025 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23e8003530 con 0x7f24040a3d60 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3067033360 0 0) 0x7f24040a8240 con 0x7f240406dd90 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): my global_id is 144291 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f240409e820 con 0x7f240406dd90 2024-09-16T05:48:03.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1210526356 0 0) 0x7f240409e820 con 0x7f240406dd90 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24040a8240 con 0x7f240406dd90 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2400001520 con 0x7f240406dd90 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): dump: 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.134 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3065056759 0 0) 0x7f23e8003530 con 0x7f24040a3d60 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f240409e820 con 0x7f24040a3d60 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3829083044 0 0) 0x7f24040a0100 con 0x7f240413d110 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): my global_id is 154180 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23e8003530 con 0x7f240413d110 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23f4001c30 con 0x7f24040a3d60 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): dump: 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.136 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1726986047 0 0) 0x7f24040a8240 con 0x7f240406dd90 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6790/0 conn(0x7f24040a3d60 legacy=0x7f24041d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.63:6789/0 conn(0x7f240413d110 legacy=0x7f24041d3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 monclient: found mon.a 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2404120c20 con 0x7f240406dd90 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _finish_auth 0 2024-09-16T05:48:03.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 5 monclient: authenticate success, global_id 144291 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient: _renew_subs 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2404136600 con 0x7f240406dd90 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f24098b6640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6789/0 conn(0x7f240406dd90 legacy=0x7f2404135510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f24098b6640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6789/0 conn(0x7f240406dd90 legacy=0x7f2404135510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f24098b6640 1 --1- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6789/0 conn(0x7f240406dd90 0x7f2404135510 :-1 s=OPENED pgs=996 cs=1 l=1).handle_message read tag failed 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f24098b6640 1 --1- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6789/0 conn(0x7f240406dd90 0x7f2404135510 :-1 s=OPENED pgs=996 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.6:6789/0 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6789/0 conn(0x7f240406dd90 legacy=0x7f2404135510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:48:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): picked mon.b con 0x7f240413d110 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): picked mon.a con 0x7f23e80089c0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): picked mon.c con 0x7f24040a3d60 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f24040a8240 con 0x7f23e80089c0 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f24040a0100 con 0x7f240413d110 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f23e8012280 con 0x7f24040a3d60 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1252606751 0 0) 0x7f24040a8240 con 0x7f23e80089c0 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:48:03.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f23e8012450 con 0x7f23e80089c0 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3792639814 0 0) 0x7f23e8012280 con 0x7f24040a3d60 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f24040a8240 con 0x7f24040a3d60 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 938204851 0 0) 0x7f24040a8240 con 0x7f24040a3d60 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.63:6789/0 conn(0x7f240413d110 legacy=0x7f23e8003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6789/0 conn(0x7f23e80089c0 legacy=0x7f23e800adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 monclient: found mon.c 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:03.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7f23e80135a0 con 0x7f24040a3d60 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2404135510 con 0x7f24040a3d60 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _finish_auth 0 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f23fc001690 con 0x7f24040a3d60 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: dump: 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2404069210 con 0x7f240409cbd0 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f23fc0018f0 con 0x7f24040a3d60 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fbfff640 4 set_mon_vals no callback set 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f23fc001b20 con 0x7f24040a3d60 2024-09-16T05:48:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 6 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f23fc043160 con 0x7f24040a3d60 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253454394 0 0) 0x7f2404069210 con 0x7f240409cbd0 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient(hunting): my global_id is 7028 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24040a8240 con 0x7f240409cbd0 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f24090b5640 10 monclient: get_auth_request con 0x7f240406dd90 auth_method 0 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3419630014 0 0) 0x7f24040a8240 con 0x7f240409cbd0 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2404069210 con 0x7f240409cbd0 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2400004f80 con 0x7f240409cbd0 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:03.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 10 monclient: dump: 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3613293668 0 0) 0x7f2404069210 con 0x7f240409cbd0 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.116+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24040008e0 con 0x7f240409cbd0 2024-09-16T05:48:03.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.120+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2400007360 con 0x7f240409cbd0 2024-09-16T05:48:03.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.120+0000 7f23fa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:03.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.120+0000 7f23fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:03.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.120+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6790/0 conn(0x7f240409cbd0 legacy=0x7f240409cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.120+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6790/0 conn(0x7f240409cbd0 legacy=0x7f240409cfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:03.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:48:03.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f240402da70 con 0x7f240402d660 2024-09-16T05:48:03.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 431769929 0 0) 0x7f240402da70 con 0x7f240402d660 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient(hunting): my global_id is 7031 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2404069210 con 0x7f240402d660 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2799453121 0 0) 0x7f2404069210 con 0x7f240402d660 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f240402da70 con 0x7f240402d660 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2400005570 con 0x7f240402d660 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient: dump: 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.147 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 215446641 0 0) 0x7f240402da70 con 0x7f240402d660 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f240409ddd0 con 0x7f240402d660 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 3881894476) 0x7f2400007590 con 0x7f240402d660 2024-09-16T05:48:03.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6790/0 conn(0x7f240402d660 legacy=0x7f240409d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f23fa7fc640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6790/0 conn(0x7f240402d660 legacy=0x7f240409d6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 58, 2024-09-16T05:48:03.149 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 812180, 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:03.150 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:03.151 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:48:03.152 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:48:03.153 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:03.154 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:48:03.155 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:03.156 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:48:03.157 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:03.158 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:48:03.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:48:03.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 7 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.162 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:48:03.162 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:48:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.63:6812/3643207211 conn(0x7f240406dd90 legacy=0x7f23e8044190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 10 monclient: shutdown 2024-09-16T05:48:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 >> v1:172.21.15.6:6790/0 conn(0x7f24040a3d60 legacy=0x7f23e800e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240a0b7640 1 -- 172.21.15.6:0/3834851310 reap_dead start 2024-09-16T05:48:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240a0b7640 1 -- 172.21.15.6:0/3834851310 reap_dead start 2024-09-16T05:48:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 shutdown_connections 2024-09-16T05:48:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 >> 172.21.15.6:0/3834851310 conn(0x7f24040a59a0 msgr2=0x7f24041392d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 shutdown_connections 2024-09-16T05:48:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.124+0000 7f240bb1a640 1 -- 172.21.15.6:0/3834851310 wait complete. 2024-09-16T05:48:03.164 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:48:03.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:48:03.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:48:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient: get_monmap_and_config 2024-09-16T05:48:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient: build_initial_monmap 2024-09-16T05:48:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient: monmap: 2024-09-16T05:48:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:03.280099+0000 2024-09-16T05:48:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:03.280099+0000 2024-09-16T05:48:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:03.279 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 1 Processor -- start 2024-09-16T05:48:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 1 -- start start 2024-09-16T05:48:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient: init 2024-09-16T05:48:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:48:03.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient(hunting): picked mon.noname-a con 0x7fe55c133700 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient(hunting): picked mon.noname-b con 0x7fe55c0733d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient(hunting): picked mon.noname-c con 0x7fe55c0737e0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe55c0721c0 con 0x7fe55c133700 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe55c072430 con 0x7fe55c0737e0 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe55c072710 con 0x7fe55c0733d0 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 10 monclient(hunting): authenticate will time out at 1745.086866s 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe561e8f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe55c133700 0x7fe55c133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58882/0 (socket says 172.21.15.6:58882) 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe561e8f640 1 -- 172.21.15.6:0/1222415907 learned_addr learned my addr 172.21.15.6:0/1222415907 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056550248 0 0) 0x7fe55c0721c0 con 0x7fe55c133700 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): my global_id is 144294 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe548003530 con 0x7fe55c133700 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543897978 0 0) 0x7fe55c072710 con 0x7fe55c0733d0 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): my global_id is 7034 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe55c0721c0 con 0x7fe55c0733d0 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 70986231 0 0) 0x7fe55c072430 con 0x7fe55c0737e0 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient(hunting): my global_id is 154183 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe55c072710 con 0x7fe55c0737e0 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3814201524 0 0) 0x7fe55c0721c0 con 0x7fe55c0733d0 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 >> v1:172.21.15.63:6789/0 conn(0x7fe55c0737e0 legacy=0x7fe55c13cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 >> v1:172.21.15.6:6789/0 conn(0x7fe55c133700 legacy=0x7fe55c133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient: _finish_hunting 0 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 monclient: found mon.noname-b 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55c0729f0 con 0x7fe55c0733d0 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient: _finish_auth 0 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient: _check_auth_tickets 2024-09-16T05:48:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe558001980 con 0x7fe55c0733d0 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe560e8d640 10 monclient: dump: 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.290 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 5 monclient: authenticate success, global_id 7034 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.276+0000 7fe5640f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe558001bf0 con 0x7fe55c0733d0 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe560e8d640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 4 set_mon_vals no callback set 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe560e8d640 1 -- 172.21.15.6:0/1222415907 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe558002000 con 0x7fe55c0733d0 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe560e8d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe560e8d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe560e8d640 10 monclient: dump: 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: shutdown 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- 172.21.15.6:0/1222415907 >> v1:172.21.15.6:6790/0 conn(0x7fe55c0733d0 legacy=0x7fe55c139650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- 172.21.15.6:0/1222415907 shutdown_connections 2024-09-16T05:48:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- 172.21.15.6:0/1222415907 >> 172.21.15.6:0/1222415907 conn(0x7fe55c0a2580 msgr2=0x7fe55c0a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- 172.21.15.6:0/1222415907 shutdown_connections 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- 172.21.15.6:0/1222415907 wait complete. 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: build_initial_monmap 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: monmap: 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.293 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 Processor -- start 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- start start 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: init 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient(hunting): picked mon.noname-a con 0x7fe55c0733d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient(hunting): picked mon.noname-b con 0x7fe55c0737e0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient(hunting): picked mon.noname-c con 0x7fe55c133700 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe55c074010 con 0x7fe55c0733d0 2024-09-16T05:48:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe55c12c0b0 con 0x7fe55c0737e0 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe55c12c390 con 0x7fe55c133700 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient(hunting): authenticate will time out at 1745.090192s 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe561e8f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe55c0733d0 0x7fe55c09cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58892/0 (socket says 172.21.15.6:58892) 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe561e8f640 1 -- 172.21.15.6:0/2143731580 learned_addr learned my addr 172.21.15.6:0/2143731580 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3782377572 0 0) 0x7fe55c12c390 con 0x7fe55c133700 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): my global_id is 7037 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe540003530 con 0x7fe55c133700 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 996845637 0 0) 0x7fe54c001920 con 0x7fe55c0733d0 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): my global_id is 144297 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe55c12c390 con 0x7fe55c0733d0 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2595800163 0 0) 0x7fe540003530 con 0x7fe55c133700 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe54c001920 con 0x7fe55c133700 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe554001820 con 0x7fe55c133700 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:48:03.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): dump: 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 280561167 0 0) 0x7fe55c12c390 con 0x7fe55c0733d0 2024-09-16T05:48:03.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe540003530 con 0x7fe55c0733d0 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035574098 0 0) 0x7fe55c12c0b0 con 0x7fe55c0737e0 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): my global_id is 154186 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe55c12c390 con 0x7fe55c0737e0 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe54c001c10 con 0x7fe55c0733d0 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient(hunting): dump: 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3986303432 0 0) 0x7fe54c001920 con 0x7fe55c133700 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 >> v1:172.21.15.63:6789/0 conn(0x7fe55c0737e0 legacy=0x7fe55c09d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 >> v1:172.21.15.6:6789/0 conn(0x7fe55c0733d0 legacy=0x7fe55c09cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient: _finish_hunting 0 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 monclient: found mon.c 2024-09-16T05:48:03.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55c09d9a0 con 0x7fe55c133700 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient: _finish_auth 0 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient: _check_auth_tickets 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 5 monclient: authenticate success, global_id 7037 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: _renew_subs 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe55c12b820 con 0x7fe55c133700 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: _renew_subs 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe55c1316b0 con 0x7fe55c133700 2024-09-16T05:48:03.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe554003620 con 0x7fe55c133700 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe554003970 con 0x7fe55c133700 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5527fc640 10 monclient: dump: 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.301 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe553fff640 4 set_mon_vals no callback set 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.280+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe5440065d0 con 0x7fe544002870 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe55400e710 con 0x7fe55c133700 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fe554043bd0 con 0x7fe55c133700 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe56168e640 10 monclient: get_auth_request con 0x7fe540036b10 auth_method 0 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1828577738 0 0) 0x7fe5440065d0 con 0x7fe544002870 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient(hunting): my global_id is 154189 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe54c001920 con 0x7fe544002870 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1436991196 0 0) 0x7fe54c001920 con 0x7fe544002870 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5440065d0 con 0x7fe544002870 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe54c004b80 con 0x7fe544002870 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient: dump: 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.303 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.304 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.304 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.304 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1380996092 0 0) 0x7fe5440065d0 con 0x7fe544002870 2024-09-16T05:48:03.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe54400a160 con 0x7fe544002870 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe54c006060 con 0x7fe544002870 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 >> v1:172.21.15.63:6789/0 conn(0x7fe544002870 legacy=0x7fe544004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.284+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 >> v1:172.21.15.63:6789/0 conn(0x7fe544002870 legacy=0x7fe544004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe544009e50 con 0x7fe544009110 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197287980 0 0) 0x7fe544009e50 con 0x7fe544009110 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient(hunting): my global_id is 154192 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5440065d0 con 0x7fe544009110 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3722154933 0 0) 0x7fe5440065d0 con 0x7fe544009110 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe544009e50 con 0x7fe544009110 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe54c0066a0 con 0x7fe544009110 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient: dump: 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.307 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 644340608 0 0) 0x7fe544009e50 con 0x7fe544009110 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe54400c3e0 con 0x7fe544009110 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3961 (unknown 0 0 3399373910) 0x7fe54c0064c0 con 0x7fe544009110 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 >> v1:172.21.15.63:6789/0 conn(0x7fe544009110 legacy=0x7fe5440095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5527fc640 1 -- 172.21.15.6:0/2143731580 >> v1:172.21.15.63:6789/0 conn(0x7fe544009110 legacy=0x7fe5440095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 58, 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 18031, 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.309 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 3, 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:03.310 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:03.311 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:48:03.312 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:03.313 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:03.314 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:03.315 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.316 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.317 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:03.319 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:03.320 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:03.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:03.320 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:48:03.320 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:48:03.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe540036b10 legacy=0x7fe540038fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 10 monclient: shutdown 2024-09-16T05:48:03.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:03.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 >> v1:172.21.15.6:6790/0 conn(0x7fe55c133700 legacy=0x7fe55c1d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe562690640 1 -- 172.21.15.6:0/2143731580 reap_dead start 2024-09-16T05:48:03.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe562690640 1 -- 172.21.15.6:0/2143731580 reap_dead start 2024-09-16T05:48:03.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 shutdown_connections 2024-09-16T05:48:03.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 >> 172.21.15.6:0/2143731580 conn(0x7fe55c0a2580 msgr2=0x7fe55c06bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:03.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 shutdown_connections 2024-09-16T05:48:03.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.292+0000 7fe5640f3640 1 -- 172.21.15.6:0/2143731580 wait complete. 2024-09-16T05:48:03.322 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:48:03.322 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T05:48:03.323 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:48:03.323 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T05:48:03.323 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:48:03.323 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T05:48:03.323 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:48:03.323 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:48:03.323 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:48:03.323 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:48:03.324 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 13 2024-09-16T05:48:03.324 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 2024-09-16T05:48:03.324 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:48:03.324 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T05:48:03.324 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T05:48:03.324 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T05:48:03.339 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T05:48:03.339 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:48:03.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:48:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient: get_monmap_and_config 2024-09-16T05:48:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient: build_initial_monmap 2024-09-16T05:48:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient: monmap: 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:03.500267+0000 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:03.500267+0000 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:03.500 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 1 Processor -- start 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 1 -- start start 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient: init 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6a340a27e0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6a34136a10 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6a3413cf70 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a340a0f80 con 0x7f6a3413cf70 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a340a4c40 con 0x7f6a34136a10 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a340a06e0 con 0x7f6a340a27e0 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 10 monclient(hunting): authenticate will time out at 1745.307270s 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a337fe640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7f6a340a27e0 legacy=0x7f6a340a2bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a33fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a3413cf70 0x7f6a3413f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58926/0 (socket says 172.21.15.6:58926) 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a33fff640 1 -- 172.21.15.6:0/1437275205 learned_addr learned my addr 172.21.15.6:0/1437275205 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824064814 0 0) 0x7f6a340a0f80 con 0x7f6a3413cf70 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient(hunting): my global_id is 144306 2024-09-16T05:48:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a20003530 con 0x7f6a3413cf70 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 5519053 0 0) 0x7f6a20003530 con 0x7f6a3413cf70 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 >> v1:172.21.15.6:6790/0 conn(0x7f6a340a27e0 legacy=0x7f6a340a2bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 >> v1:172.21.15.63:6789/0 conn(0x7f6a34136a10 legacy=0x7f6a3413b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 1 monclient: found mon.noname-a 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3406dff0 con 0x7f6a3413cf70 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient: _finish_auth 0 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:03.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a2c0019a0 con 0x7f6a3413cf70 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a32ffd640 10 monclient: dump: 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.509 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.496+0000 7f6a3a0f1640 5 monclient: authenticate success, global_id 144306 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a2c001d00 con 0x7f6a3413cf70 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a32ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a32ffd640 1 -- 172.21.15.6:0/1437275205 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a2c0020d0 con 0x7f6a3413cf70 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a32ffd640 10 monclient: dump: 2024-09-16T05:48:03.510 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 4 set_mon_vals no callback set 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: shutdown 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/1437275205 >> v1:172.21.15.6:6789/0 conn(0x7f6a3413cf70 legacy=0x7f6a3413f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/1437275205 shutdown_connections 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/1437275205 >> 172.21.15.6:0/1437275205 conn(0x7f6a340a9880 msgr2=0x7f6a340a5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/1437275205 shutdown_connections 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/1437275205 wait complete. 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: build_initial_monmap 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: monmap: 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 Processor -- start 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- start start 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: init 2024-09-16T05:48:03.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6a340a27e0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6a34136a10 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6a3413cf70 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3413ab10 con 0x7f6a340a27e0 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3413c080 con 0x7f6a3413cf70 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a34139640 con 0x7f6a34136a10 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient(hunting): authenticate will time out at 1745.310245s 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a2affd640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7f6a34136a10 legacy=0x7f6a34138f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a337fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a340a27e0 0x7f6a34138820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58940/0 (socket says 172.21.15.6:58940) 2024-09-16T05:48:03.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a337fe640 1 -- 172.21.15.6:0/3819262993 learned_addr learned my addr 172.21.15.6:0/3819262993 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1113145491 0 0) 0x7f6a3413ab10 con 0x7f6a340a27e0 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): my global_id is 144309 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a14003530 con 0x7f6a340a27e0 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1835271789 0 0) 0x7f6a14003530 con 0x7f6a340a27e0 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a3413ab10 con 0x7f6a340a27e0 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a24004160 con 0x7f6a340a27e0 2024-09-16T05:48:03.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): dump: 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.516 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1732766101 0 0) 0x7f6a3413c080 con 0x7f6a3413cf70 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient(hunting): my global_id is 154204 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a14003530 con 0x7f6a3413cf70 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 37635143 0 0) 0x7f6a3413ab10 con 0x7f6a340a27e0 2024-09-16T05:48:03.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 >> v1:172.21.15.6:6790/0 conn(0x7f6a34136a10 legacy=0x7f6a34138f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 >> v1:172.21.15.63:6789/0 conn(0x7f6a3413cf70 legacy=0x7f6a341d3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 monclient: found mon.a 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3409aa30 con 0x7f6a340a27e0 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient: _finish_auth 0 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 5 monclient: authenticate success, global_id 144309 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: _renew_subs 2024-09-16T05:48:03.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3413c970 con 0x7f6a340a27e0 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: _renew_subs 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a34140850 con 0x7f6a340a27e0 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a240044c0 con 0x7f6a340a27e0 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6a24004890 con 0x7f6a340a27e0 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a327fc640 4 set_mon_vals no callback set 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.500+0000 7f6a30ff9640 10 monclient: dump: 2024-09-16T05:48:03.519 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.504+0000 7f6a3a0f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.504+0000 7f6a3a0f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:03.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.504+0000 7f6a3a0f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.504+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a18005090 con 0x7f6a340a27e0 2024-09-16T05:48:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.504+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6a24004a30 con 0x7f6a340a27e0 2024-09-16T05:48:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.504+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f6a24044f00 con 0x7f6a340a27e0 2024-09-16T05:48:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.504+0000 7f6a2affd640 10 monclient: get_auth_request con 0x7f6a14036ad0 auth_method 0 2024-09-16T05:48:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.508+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6a24015590 con 0x7f6a340a27e0 2024-09-16T05:48:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.508+0000 7f6a30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:03.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.508+0000 7f6a30ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:48:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:48:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:48:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 13 2024-09-16T05:48:03.637 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 2024-09-16T05:48:03.638 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T05:48:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:48:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T05:48:03.638 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T05:48:03.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:48:03.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6a18005380 con 0x7f6a340a27e0 2024-09-16T05:48:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a30ff9640 1 -- 172.21.15.6:0/3819262993 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 567747052) 0x7f6a24014e90 con 0x7f6a340a27e0 2024-09-16T05:48:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a30ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:03.742 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:48:03.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6a14036ad0 legacy=0x7f6a14038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 10 monclient: shutdown 2024-09-16T05:48:03.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:03.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 >> v1:172.21.15.6:6789/0 conn(0x7f6a340a27e0 legacy=0x7f6a34138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:03.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 shutdown_connections 2024-09-16T05:48:03.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 >> 172.21.15.6:0/3819262993 conn(0x7f6a340a9880 msgr2=0x7f6a340a1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:03.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 shutdown_connections 2024-09-16T05:48:03.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:03.740+0000 7f6a3a0f1640 1 -- 172.21.15.6:0/3819262993 wait complete. 2024-09-16T05:48:03.972 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:48:03.972 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:48:03.973 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:48:03.973 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 13 2024-09-16T05:48:03.973 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 2024-09-16T05:48:03.973 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T05:48:03.973 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:48:03.973 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T05:48:03.974 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T05:48:04.254 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:48:04.254 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:48:04.255 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:48:04.255 INFO:tasks.workunit.client.0.smithi063.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T05:48:04.255 INFO:tasks.workunit.client.0.smithi063.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T05:48:04.255 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 110 2024-09-16T05:48:04.255 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 2024-09-16T05:48:04.255 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=110 2024-09-16T05:48:04.255 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T05:48:04.255 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T05:48:04.256 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T05:48:06.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.892+0000 7f1e83763640 10 monclient: get_monmap_and_config 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.892+0000 7f1e83763640 10 monclient: build_initial_monmap 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.892+0000 7f1e83763640 10 monclient: monmap: 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:06.896884+0000 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:06.896884+0000 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:06.895 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:06.896 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:06.896 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:06.896 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:06.896 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:06.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.892+0000 7f1e83763640 1 Processor -- start 2024-09-16T05:48:06.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.892+0000 7f1e83763640 1 -- start start 2024-09-16T05:48:06.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.892+0000 7f1e83763640 10 monclient: init 2024-09-16T05:48:06.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.892+0000 7f1e83763640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:06.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.892+0000 7f1e83763640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): picked mon.noname-b con 0x7f1e7c0a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): picked mon.noname-c con 0x7f1e7c0a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): picked mon.noname-a con 0x7f1e7c0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7c06e520 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7c0a0780 con 0x7f1e7c0a4290 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7c0a4c70 con 0x7f1e7c0a37a0 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): authenticate will time out at 1748.704112s 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e814ff640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7f1e7c0a37a0 legacy=0x7f1e7c0a3b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e81d00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1e7c0a1a00 0x7f1e7c13f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58956/0 (socket says 172.21.15.6:58956) 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e81d00640 1 -- 172.21.15.6:0/3351922130 learned_addr learned my addr 172.21.15.6:0/3351922130 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:06.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571802372 0 0) 0x7f1e7c06e520 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient(hunting): _negotiate 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient(hunting): my global_id is 144330 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e68003530 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1479016047 0 0) 0x7f1e68003530 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 >> v1:172.21.15.6:6790/0 conn(0x7f1e7c0a37a0 legacy=0x7f1e7c0a3b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 >> v1:172.21.15.63:6789/0 conn(0x7f1e7c0a4290 legacy=0x7f1e7c13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:06.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: _finish_hunting 0 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 monclient: found mon.noname-a 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7c0a65f0 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: _finish_auth 0 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: _check_auth_tickets 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e740019a0 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: dump: 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 5 monclient: authenticate success, global_id 144330 2024-09-16T05:48:06.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1e74001d00 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 4 set_mon_vals no callback set 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 1 -- 172.21.15.6:0/3351922130 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e74004c30 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e73fff640 10 monclient: dump: 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:06.903 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:06.904 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:06.904 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:06.904 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:06.904 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:06.904 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:06.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient: shutdown 2024-09-16T05:48:06.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:06.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- 172.21.15.6:0/3351922130 >> v1:172.21.15.6:6789/0 conn(0x7f1e7c0a1a00 legacy=0x7f1e7c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:06.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- 172.21.15.6:0/3351922130 shutdown_connections 2024-09-16T05:48:06.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- 172.21.15.6:0/3351922130 >> 172.21.15.6:0/3351922130 conn(0x7f1e7c0a58a0 msgr2=0x7f1e7c0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- 172.21.15.6:0/3351922130 shutdown_connections 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- 172.21.15.6:0/3351922130 wait complete. 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient: build_initial_monmap 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient: monmap: 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:06.908 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 Processor -- start 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- start start 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient: init 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): picked mon.noname-b con 0x7f1e7c0a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): picked mon.noname-c con 0x7f1e7c0a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): picked mon.noname-a con 0x7f1e7c0a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7c1357c0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7c135aa0 con 0x7f1e7c0a1a00 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e7c132ab0 con 0x7f1e7c0a37a0 2024-09-16T05:48:06.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e80cfe640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7f1e7c0a37a0 legacy=0x7f1e7c1323a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e83763640 10 monclient(hunting): authenticate will time out at 1748.707585s 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e81d00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1e7c0a4290 0x7f1e7c1d3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58964/0 (socket says 172.21.15.6:58964) 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e81d00640 1 -- 172.21.15.6:0/1979178696 learned_addr learned my addr 172.21.15.6:0/1979178696 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141351036 0 0) 0x7f1e7c1357c0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e71ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e71ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e71ffb640 10 monclient(hunting): my global_id is 144333 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.896+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e60003530 con 0x7f1e7c0a4290 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3261253336 0 0) 0x7f1e60003530 con 0x7f1e7c0a4290 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e7c1357c0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e740017c0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient(hunting): dump: 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:06.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2193976426 0 0) 0x7f1e7c1357c0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 >> v1:172.21.15.6:6790/0 conn(0x7f1e7c0a37a0 legacy=0x7f1e7c1323a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 >> v1:172.21.15.63:6789/0 conn(0x7f1e7c0a1a00 legacy=0x7f1e7c131c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 monclient: found mon.a 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7c133740 con 0x7f1e7c0a4290 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient: _finish_auth 0 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 5 monclient: authenticate success, global_id 144333 2024-09-16T05:48:06.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 10 monclient: _renew_subs 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1e7c1344c0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 10 monclient: _renew_subs 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1e7c137220 con 0x7f1e7c0a4290 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1e74001d00 con 0x7f1e7c0a4290 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1e74001a20 con 0x7f1e7c0a4290 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 10 monclient: dump: 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:06.913 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e737fe640 4 set_mon_vals no callback set 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e5c005090 con 0x7f1e7c0a4290 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1e7400fdf0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f1e740452c0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.900+0000 7f1e814ff640 10 monclient: get_auth_request con 0x7f1e600369f0 auth_method 0 2024-09-16T05:48:06.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.904+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1e740159b0 con 0x7f1e7c0a4290 2024-09-16T05:48:06.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.904+0000 7f1e71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:06.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:06.904+0000 7f1e71ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:07.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.128+0000 7f1e83763640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:48:07.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.128+0000 7f1e83763640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:07.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.128+0000 7f1e83763640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:07.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.128+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1e5c005380 con 0x7f1e7c0a4290 2024-09-16T05:48:07.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.128+0000 7f1e71ffb640 1 -- 172.21.15.6:0/1979178696 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 746688363) 0x7f1e740152b0 con 0x7f1e7c0a4290 2024-09-16T05:48:07.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.128+0000 7f1e71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:07.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.128+0000 7f1e71ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:07.132 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:48:07.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.132+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1e600369f0 legacy=0x7f1e60038eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:07.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.132+0000 7f1e83763640 10 monclient: shutdown 2024-09-16T05:48:07.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.132+0000 7f1e83763640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:07.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.132+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 >> v1:172.21.15.6:6789/0 conn(0x7f1e7c0a4290 legacy=0x7f1e7c1d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:07.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.132+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 shutdown_connections 2024-09-16T05:48:07.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.132+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 >> 172.21.15.6:0/1979178696 conn(0x7f1e7c0a58a0 msgr2=0x7f1e7c0a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:07.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.132+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 shutdown_connections 2024-09-16T05:48:07.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:07.132+0000 7f1e83763640 1 -- 172.21.15.6:0/1979178696 wait complete. 2024-09-16T05:48:10.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:48:10.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient: get_monmap_and_config 2024-09-16T05:48:10.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient: build_initial_monmap 2024-09-16T05:48:10.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient: monmap: 2024-09-16T05:48:10.282 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:10.284128+0000 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:10.284128+0000 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:10.283 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:10.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 1 Processor -- start 2024-09-16T05:48:10.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 1 -- start start 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient: init 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient(hunting): picked mon.noname-a con 0x7fad5409bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient(hunting): picked mon.noname-c con 0x7fad5409bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient(hunting): picked mon.noname-b con 0x7fad541397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad540a1350 con 0x7fad5409bb40 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad540a10d0 con 0x7fad5409bf20 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad5409d940 con 0x7fad541397f0 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad59618640 10 monclient(hunting): authenticate will time out at 1752.091317s 2024-09-16T05:48:10.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad537fe640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad5413bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:10.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad52ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fad5409bb40 0x7fad5409e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41874/0 (socket says 172.21.15.6:41874) 2024-09-16T05:48:10.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.280+0000 7fad52ffd640 1 -- 172.21.15.6:0/908885535 learned_addr learned my addr 172.21.15.6:0/908885535 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:10.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.484+0000 7fad537fe640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad5413bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:10.888+0000 7fad537fe640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad5413bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:11.692+0000 7fad537fe640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad5413bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 10 monclient(hunting): tick 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 1 monclient(hunting): continuing hunt 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad5413bbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.63:6789/0 conn(0x7fad5409bf20 legacy=0x7fad5409c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6789/0 conn(0x7fad5409bb40 legacy=0x7fad5409e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:48:13.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 10 monclient(hunting): picked mon.noname-a con 0x7fad340058e0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 10 monclient(hunting): picked mon.noname-b con 0x7fad541397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 10 monclient(hunting): picked mon.noname-c con 0x7fad3400cdd0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad5409dc20 con 0x7fad340058e0 2024-09-16T05:48:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad5409d940 con 0x7fad3400cdd0 2024-09-16T05:48:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad34012ac0 con 0x7fad541397f0 2024-09-16T05:48:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad517fa640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:13.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.284+0000 7fad4a7fc640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.488+0000 7fad4a7fc640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:13.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:13.892+0000 7fad4a7fc640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:14.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:14.696+0000 7fad4a7fc640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 10 monclient(hunting): tick 2024-09-16T05:48:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:48:16.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 1 monclient(hunting): continuing hunt 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.63:6789/0 conn(0x7fad3400cdd0 legacy=0x7fad3400f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6789/0 conn(0x7fad340058e0 legacy=0x7fad34007d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 10 monclient(hunting): picked mon.noname-c con 0x7fad34012dc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 10 monclient(hunting): picked mon.noname-a con 0x7fad3400fa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 10 monclient(hunting): picked mon.noname-b con 0x7fad541397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad34013db0 con 0x7fad3400fa30 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad34012ac0 con 0x7fad34012dc0 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 1 -- 172.21.15.6:0/908885535 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad3400fe30 con 0x7fad541397f0 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad517fa640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:16.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.284+0000 7fad537fe640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:16.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.488+0000 7fad537fe640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:16.892+0000 7fad537fe640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:17.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:17.696+0000 7fad537fe640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:18.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133647877 0 0) 0x7fad34012ac0 con 0x7fad34012dc0 2024-09-16T05:48:18.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient(hunting): my global_id is 154219 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad40003530 con 0x7fad34012dc0 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2483997772 0 0) 0x7fad40003530 con 0x7fad34012dc0 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad3400b600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.6:6789/0 conn(0x7fad3400fa30 legacy=0x7fad3400f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: _finish_hunting 0 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 monclient: found mon.noname-c 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad34007d40 con 0x7fad34012dc0 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: _finish_auth 0 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: _check_auth_tickets 2024-09-16T05:48:18.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad3c001520 con 0x7fad34012dc0 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: dump: 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.371 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 5 monclient: authenticate success, global_id 154219 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad3c001780 con 0x7fad34012dc0 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 4 set_mon_vals no callback set 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 1 -- 172.21.15.6:0/908885535 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad3c001bd0 con 0x7fad34012dc0 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:48:18.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad527fc640 10 monclient: dump: 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 10 monclient: shutdown 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:18.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 1 -- 172.21.15.6:0/908885535 >> v1:172.21.15.63:6789/0 conn(0x7fad34012dc0 legacy=0x7fad340131a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 1 -- 172.21.15.6:0/908885535 shutdown_connections 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 1 -- 172.21.15.6:0/908885535 >> 172.21.15.6:0/908885535 conn(0x7fad540a1650 msgr2=0x7fad540a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 1 -- 172.21.15.6:0/908885535 shutdown_connections 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.364+0000 7fad59618640 1 -- 172.21.15.6:0/908885535 wait complete. 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient: build_initial_monmap 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient: monmap: 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.374 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:18.375 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:18.375 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 1 Processor -- start 2024-09-16T05:48:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 1 -- start start 2024-09-16T05:48:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient: init 2024-09-16T05:48:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:48:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient(hunting): picked mon.noname-a con 0x7fad3400fa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient(hunting): picked mon.noname-b con 0x7fad34012dc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:18.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient(hunting): picked mon.noname-c con 0x7fad541397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad54120400 con 0x7fad3400fa30 2024-09-16T05:48:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad541206e0 con 0x7fad34012dc0 2024-09-16T05:48:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad54072460 con 0x7fad541397f0 2024-09-16T05:48:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad59618640 10 monclient(hunting): authenticate will time out at 1760.177906s 2024-09-16T05:48:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad537fe640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad541d4320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad52ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fad3400fa30 0x7fad541cbc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41908/0 (socket says 172.21.15.6:41908) 2024-09-16T05:48:18.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.368+0000 7fad52ffd640 1 -- 172.21.15.6:0/1528218631 learned_addr learned my addr 172.21.15.6:0/1528218631 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1884287908 0 0) 0x7fad54120400 con 0x7fad3400fa30 2024-09-16T05:48:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): my global_id is 144336 2024-09-16T05:48:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad34008050 con 0x7fad3400fa30 2024-09-16T05:48:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452049969 0 0) 0x7fad541206e0 con 0x7fad34012dc0 2024-09-16T05:48:18.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): my global_id is 154222 2024-09-16T05:48:18.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad54120400 con 0x7fad34012dc0 2024-09-16T05:48:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1037121742 0 0) 0x7fad34008050 con 0x7fad3400fa30 2024-09-16T05:48:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad541206e0 con 0x7fad3400fa30 2024-09-16T05:48:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad3c001b90 con 0x7fad3400fa30 2024-09-16T05:48:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:18.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): dump: 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:18.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3664935716 0 0) 0x7fad54120400 con 0x7fad34012dc0 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad34008050 con 0x7fad34012dc0 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad44001730 con 0x7fad34012dc0 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient(hunting): dump: 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.384 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 53373110 0 0) 0x7fad541206e0 con 0x7fad3400fa30 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 >> v1:172.21.15.6:6790/0 conn(0x7fad541397f0 legacy=0x7fad541d4320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 >> v1:172.21.15.63:6789/0 conn(0x7fad34012dc0 legacy=0x7fad541d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient: _finish_hunting 0 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 monclient: found mon.a 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad540a33d0 con 0x7fad3400fa30 2024-09-16T05:48:18.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient: _finish_auth 0 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient: _check_auth_tickets 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad59618640 5 monclient: authenticate success, global_id 144336 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad59618640 10 monclient: _renew_subs 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad59618640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fad541209c0 con 0x7fad3400fa30 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad59618640 10 monclient: _renew_subs 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad59618640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fad5411dae0 con 0x7fad3400fa30 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad3c001780 con 0x7fad3400fa30 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad3c004c40 con 0x7fad3400fa30 2024-09-16T05:48:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad51ffb640 4 set_mon_vals no callback set 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.376+0000 7fad4bfff640 10 monclient: dump: 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.387 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.380+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fad3c00f9e0 con 0x7fad3400fa30 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.380+0000 7fad59618640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.380+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fad3c044c30 con 0x7fad3400fa30 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.380+0000 7fad59618640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.380+0000 7fad59618640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.380+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad38005090 con 0x7fad3400fa30 2024-09-16T05:48:18.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.380+0000 7fad4a7fc640 10 monclient: get_auth_request con 0x7fad34038480 auth_method 0 2024-09-16T05:48:18.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.384+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fad3c015380 con 0x7fad3400fa30 2024-09-16T05:48:18.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.384+0000 7fad4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:18.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.384+0000 7fad4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:18.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.608+0000 7fad59618640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:48:18.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.608+0000 7fad59618640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:18.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.608+0000 7fad59618640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.608+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fad38005380 con 0x7fad3400fa30 2024-09-16T05:48:18.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.608+0000 7fad4bfff640 1 -- 172.21.15.6:0/1528218631 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 3608425426) 0x7fad3c014c80 con 0x7fad3400fa30 2024-09-16T05:48:18.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.608+0000 7fad4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:48:18.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.608+0000 7fad4bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:18.611 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":62,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:48:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.612+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 >> v1:172.21.15.63:6812/3643207211 conn(0x7fad34038480 legacy=0x7fad3403a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.612+0000 7fad59618640 10 monclient: shutdown 2024-09-16T05:48:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.612+0000 7fad59618640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.612+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 >> v1:172.21.15.6:6789/0 conn(0x7fad3400fa30 legacy=0x7fad541cbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.612+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 shutdown_connections 2024-09-16T05:48:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.612+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 >> 172.21.15.6:0/1528218631 conn(0x7fad540a1650 msgr2=0x7fad54133a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.612+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 shutdown_connections 2024-09-16T05:48:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.612+0000 7fad59618640 1 -- 172.21.15.6:0/1528218631 wait complete. 2024-09-16T05:48:18.628 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:48:18.628 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:48:18.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:48:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient: get_monmap_and_config 2024-09-16T05:48:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient: build_initial_monmap 2024-09-16T05:48:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient: monmap: 2024-09-16T05:48:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:18.755165+0000 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:18.755165+0000 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:18.754 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 Processor -- start 2024-09-16T05:48:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- start start 2024-09-16T05:48:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient: init 2024-09-16T05:48:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:48:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient(hunting): picked mon.noname-c con 0x7f78f809aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient(hunting): picked mon.noname-b con 0x7f78f809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient(hunting): picked mon.noname-a con 0x7f78f80a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f80a1370 con 0x7f78f80a0da0 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f809ead0 con 0x7f78f809aa30 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f5d9b640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7f78f809e230 legacy=0x7f78f81399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f809edb0 con 0x7f78f809e230 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient(hunting): authenticate will time out at 1760.561974s 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f6d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f78f80a0da0 0x7f78f813d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41920/0 (socket says 172.21.15.6:41920) 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f6d9d640 1 -- 172.21.15.6:0/1968011966 learned_addr learned my addr 172.21.15.6:0/1968011966 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1019095743 0 0) 0x7f78f80a1370 con 0x7f78f80a0da0 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient(hunting): my global_id is 144339 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78e0003530 con 0x7f78f80a0da0 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2361972482 0 0) 0x7f78e0003530 con 0x7f78f80a0da0 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 >> v1:172.21.15.6:6790/0 conn(0x7f78f809e230 legacy=0x7f78f81399d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 >> v1:172.21.15.63:6789/0 conn(0x7f78f809aa30 legacy=0x7f78f809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: _finish_hunting 0 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 monclient: found mon.noname-a 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f809dd50 con 0x7f78f80a0da0 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: _finish_auth 0 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: _check_auth_tickets 2024-09-16T05:48:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78f00019a0 con 0x7f78f80a0da0 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 5 monclient: authenticate success, global_id 144339 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: dump: 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78f0001c90 con 0x7f78f80a0da0 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 1 -- 172.21.15.6:0/1968011966 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78f0002060 con 0x7f78f80a0da0 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 4 set_mon_vals no callback set 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78f559a640 10 monclient: dump: 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 10 monclient: shutdown 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- 172.21.15.6:0/1968011966 >> v1:172.21.15.6:6789/0 conn(0x7f78f80a0da0 legacy=0x7f78f813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- 172.21.15.6:0/1968011966 shutdown_connections 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- 172.21.15.6:0/1968011966 >> 172.21.15.6:0/1968011966 conn(0x7f78f80a1650 msgr2=0x7f78f80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- 172.21.15.6:0/1968011966 shutdown_connections 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.752+0000 7f78fca86640 1 -- 172.21.15.6:0/1968011966 wait complete. 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: build_initial_monmap 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: monmap: 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 1 Processor -- start 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 1 -- start start 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: init 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient(hunting): picked mon.noname-b con 0x7f78f809aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient(hunting): picked mon.noname-a con 0x7f78f809e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient(hunting): picked mon.noname-c con 0x7f78f80a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f813ca30 con 0x7f78f809e230 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f809c990 con 0x7f78f809aa30 2024-09-16T05:48:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f813bd80 con 0x7f78f80a0da0 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient(hunting): authenticate will time out at 1760.564900s 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78f6d9d640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7f78f80a0da0 legacy=0x7f78f81d3720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78f5d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f78f809e230 0x7f78f809c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41922/0 (socket says 172.21.15.6:41922) 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78f5d9b640 1 -- 172.21.15.6:0/685113737 learned_addr learned my addr 172.21.15.6:0/685113737 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097882308 0 0) 0x7f78f813ca30 con 0x7f78f809e230 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): my global_id is 144342 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d8003530 con 0x7f78f809e230 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 149468585 0 0) 0x7f78d8003530 con 0x7f78f809e230 2024-09-16T05:48:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78f813ca30 con 0x7f78f809e230 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78ec004160 con 0x7f78f809e230 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): dump: 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030418687 0 0) 0x7f78f809c990 con 0x7f78f809aa30 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): my global_id is 154228 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d8003530 con 0x7f78f809aa30 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3309514081 0 0) 0x7f78f813ca30 con 0x7f78f809e230 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 >> v1:172.21.15.6:6790/0 conn(0x7f78f80a0da0 legacy=0x7f78f81d3720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 >> v1:172.21.15.63:6789/0 conn(0x7f78f809aa30 legacy=0x7f78f809bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:48:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 monclient: found mon.a 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f809d810 con 0x7f78f809e230 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: _finish_auth 0 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 5 monclient: authenticate success, global_id 144342 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: _renew_subs 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78f8135160 con 0x7f78f809e230 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: _renew_subs 2024-09-16T05:48:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78f813d660 con 0x7f78f809e230 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78ec0043c0 con 0x7f78f809e230 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78ec004790 con 0x7f78f809e230 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: dump: 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78f4d99640 4 set_mon_vals no callback set 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78dc0065d0 con 0x7f78dc002870 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f78ec0049b0 con 0x7f78f809e230 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7f78ec044ed0 con 0x7f78f809e230 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771298257 0 0) 0x7f78dc0065d0 con 0x7f78dc002870 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient(hunting): my global_id is 144345 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78f813ca30 con 0x7f78dc002870 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3442326534 0 0) 0x7f78f813ca30 con 0x7f78dc002870 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78f6d9d640 10 monclient: get_auth_request con 0x7f78d8036a80 auth_method 0 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78dc0065d0 con 0x7f78dc002870 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78e8001720 con 0x7f78dc002870 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: dump: 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1226313487 0 0) 0x7f78dc0065d0 con 0x7f78dc002870 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78dc00a160 con 0x7f78dc002870 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f78e8001b50 con 0x7f78dc002870 2024-09-16T05:48:18.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:18.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:18.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 >> v1:172.21.15.6:6789/0 conn(0x7f78dc002870 legacy=0x7f78dc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.756+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 >> v1:172.21.15.6:6789/0 conn(0x7f78dc002870 legacy=0x7f78dc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:48:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78dc009e50 con 0x7f78dc009110 2024-09-16T05:48:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289441010 0 0) 0x7f78dc009e50 con 0x7f78dc009110 2024-09-16T05:48:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient(hunting): my global_id is 144348 2024-09-16T05:48:18.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78dc0065d0 con 0x7f78dc009110 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2165913573 0 0) 0x7f78dc0065d0 con 0x7f78dc009110 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78dc009e50 con 0x7f78dc009110 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f78e80037a0 con 0x7f78dc009110 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient: dump: 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1422078343 0 0) 0x7f78dc009e50 con 0x7f78dc009110 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f78dc00c3e0 con 0x7f78dc009110 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4287 (unknown 0 0 1935512088) 0x7f78e8003990 con 0x7f78dc009110 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 >> v1:172.21.15.6:6789/0 conn(0x7f78dc009110 legacy=0x7f78dc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78e6ffd640 1 -- 172.21.15.6:0/685113737 >> v1:172.21.15.6:6789/0 conn(0x7f78dc009110 legacy=0x7f78dc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 62, 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 147378, 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: 1 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.781 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:18.782 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:18.783 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:18.784 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:18.785 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:18.786 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:18.787 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.788 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:18.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:18.791 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.792 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.793 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:48:18.793 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:48:18.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 >> v1:172.21.15.63:6812/3643207211 conn(0x7f78d8036a80 legacy=0x7f78d8038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 10 monclient: shutdown 2024-09-16T05:48:18.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:18.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 >> v1:172.21.15.6:6789/0 conn(0x7f78f809e230 legacy=0x7f78f809c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78f6d9d640 1 -- 172.21.15.6:0/685113737 reap_dead start 2024-09-16T05:48:18.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78f6d9d640 1 -- 172.21.15.6:0/685113737 reap_dead start 2024-09-16T05:48:18.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 shutdown_connections 2024-09-16T05:48:18.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 >> 172.21.15.6:0/685113737 conn(0x7f78f80a1650 msgr2=0x7f78f8071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:18.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 shutdown_connections 2024-09-16T05:48:18.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.764+0000 7f78fca86640 1 -- 172.21.15.6:0/685113737 wait complete. 2024-09-16T05:48:18.795 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:48:18.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:48:18.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:48:18.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient: get_monmap_and_config 2024-09-16T05:48:18.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient: build_initial_monmap 2024-09-16T05:48:18.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient: monmap: 2024-09-16T05:48:18.910 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:18.910 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.911 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:48:18.912288+0000 2024-09-16T05:48:18.911 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:48:18.912288+0000 2024-09-16T05:48:18.911 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:18.911 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.911 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:18.911 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:48:18.911 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:48:18.911 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 1 Processor -- start 2024-09-16T05:48:18.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 1 -- start start 2024-09-16T05:48:18.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient: init 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient(hunting): picked mon.noname-a con 0x7fa61009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient(hunting): picked mon.noname-b con 0x7fa61009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient(hunting): picked mon.noname-c con 0x7fa6100a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6100a1370 con 0x7fa61009aa30 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61009ead0 con 0x7fa6100a0da0 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61009edb0 con 0x7fa61009e230 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient(hunting): authenticate will time out at 1760.718881s 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa615888640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa61009aa30 0x7fa61009ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41950/0 (socket says 172.21.15.6:41950) 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa615888640 1 -- 172.21.15.6:0/4040422748 learned_addr learned my addr 172.21.15.6:0/4040422748 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:18.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa615087640 1 -- 172.21.15.6:0/4040422748 >> v1:172.21.15.6:6790/0 conn(0x7fa61009e230 legacy=0x7fa6101399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4283619495 0 0) 0x7fa6100a1370 con 0x7fa61009aa30 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient(hunting): my global_id is 144351 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5fc003530 con 0x7fa61009aa30 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4276381312 0 0) 0x7fa5fc003530 con 0x7fa61009aa30 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 >> v1:172.21.15.6:6790/0 conn(0x7fa61009e230 legacy=0x7fa6101399d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 >> v1:172.21.15.63:6789/0 conn(0x7fa6100a0da0 legacy=0x7fa61013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: _finish_hunting 0 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 monclient: found mon.noname-a 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:18.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61009dd50 con 0x7fa61009aa30 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: _finish_auth 0 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: _check_auth_tickets 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa60c001620 con 0x7fa61009aa30 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 5 monclient: authenticate success, global_id 144351 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: dump: 2024-09-16T05:48:18.921 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa60c001910 con 0x7fa61009aa30 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:18.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 1 -- 172.21.15.6:0/4040422748 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa60c001d20 con 0x7fa61009aa30 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 10 monclient: get_monmap_and_config success 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa617aec640 4 set_mon_vals no callback set 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.908+0000 7fa614886640 10 monclient: dump: 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.923 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: shutdown 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- 172.21.15.6:0/4040422748 >> v1:172.21.15.6:6789/0 conn(0x7fa61009aa30 legacy=0x7fa61009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- 172.21.15.6:0/4040422748 shutdown_connections 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- 172.21.15.6:0/4040422748 >> 172.21.15.6:0/4040422748 conn(0x7fa6100a1650 msgr2=0x7fa6100a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- 172.21.15.6:0/4040422748 shutdown_connections 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- 172.21.15.6:0/4040422748 wait complete. 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: build_initial_monmap 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: monmap: 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:48:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 Processor -- start 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- start start 2024-09-16T05:48:18.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: init 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: _reopen_session rank -1 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient(hunting): picked mon.noname-b con 0x7fa61009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient(hunting): picked mon.noname-c con 0x7fa61009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient(hunting): picked mon.noname-a con 0x7fa6100a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61013cc80 con 0x7fa6100a0da0 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61013dac0 con 0x7fa61009aa30 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61006eb60 con 0x7fa61009e230 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient(hunting): _renew_subs 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient(hunting): authenticate will time out at 1760.721769s 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa615087640 1 -- >> v1:172.21.15.6:6790/0 conn(0x7fa61009e230 legacy=0x7fa6101d0910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6790/0 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa616089640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6100a0da0 0x7fa6101d4040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41960/0 (socket says 172.21.15.6:41960) 2024-09-16T05:48:18.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa616089640 1 -- 172.21.15.6:0/2661629196 learned_addr learned my addr 172.21.15.6:0/2661629196 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669473339 0 0) 0x7fa61013cc80 con 0x7fa6100a0da0 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): my global_id is 144354 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5f4003530 con 0x7fa6100a0da0 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 997329226 0 0) 0x7fa5f4003530 con 0x7fa6100a0da0 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa61013cc80 con 0x7fa6100a0da0 2024-09-16T05:48:18.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa608001c40 con 0x7fa6100a0da0 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): dump: 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 520940497 0 0) 0x7fa61013dac0 con 0x7fa61009aa30 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient(hunting): my global_id is 154234 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5f4003530 con 0x7fa61009aa30 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 747312638 0 0) 0x7fa61013cc80 con 0x7fa6100a0da0 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 >> v1:172.21.15.6:6790/0 conn(0x7fa61009e230 legacy=0x7fa6101d0910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 >> v1:172.21.15.63:6789/0 conn(0x7fa61009aa30 legacy=0x7fa6101cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient: _finish_hunting 0 2024-09-16T05:48:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 monclient: found mon.a 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61013c9a0 con 0x7fa6100a0da0 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient: _finish_auth 0 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient: _check_auth_tickets 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 5 monclient: authenticate success, global_id 144354 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: _renew_subs 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa61013d660 con 0x7fa6100a0da0 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: _renew_subs 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:48:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa61009cd50 con 0x7fa6100a0da0 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6080039e0 con 0x7fa6100a0da0 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa603fff640 4 set_mon_vals no callback set 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa608003d30 con 0x7fa6100a0da0 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 10 monclient: dump: 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5f80065d0 con 0x7fa5f8002870 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa608003f50 con 0x7fa6100a0da0 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.912+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 20..259) ==== 6817+0+0 (unknown 1936949120 0 0) 0x7fa608045240 con 0x7fa6100a0da0 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa615087640 10 monclient: get_auth_request con 0x7fa5f4036ad0 auth_method 0 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 996657216 0 0) 0x7fa5f80065d0 con 0x7fa5f8002870 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient(hunting): my global_id is 154237 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61013cc80 con 0x7fa5f8002870 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3585362246 0 0) 0x7fa61013cc80 con 0x7fa5f8002870 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5f80065d0 con 0x7fa5f8002870 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa60c001620 con 0x7fa5f8002870 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient: dump: 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.934 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2089321985 0 0) 0x7fa5f80065d0 con 0x7fa5f8002870 2024-09-16T05:48:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5f800a160 con 0x7fa5f8002870 2024-09-16T05:48:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa60c001990 con 0x7fa5f8002870 2024-09-16T05:48:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:48:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:48:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 >> v1:172.21.15.63:6789/0 conn(0x7fa5f8002870 legacy=0x7fa5f8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.916+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 >> v1:172.21.15.63:6789/0 conn(0x7fa5f8002870 legacy=0x7fa5f8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.920+0000 7fa617aec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:48:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.920+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5f8009e50 con 0x7fa5f8009110 2024-09-16T05:48:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2420543481 0 0) 0x7fa5f8009e50 con 0x7fa5f8009110 2024-09-16T05:48:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:48:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:48:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:48:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient(hunting): my global_id is 154240 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5f80065d0 con 0x7fa5f8009110 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2029321871 0 0) 0x7fa5f80065d0 con 0x7fa5f8009110 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5f8009e50 con 0x7fa5f8009110 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa60c004a20 con 0x7fa5f8009110 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient: dump: 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:48:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:48:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:48:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:48:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:48:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:48:18.938 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:48:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 541928086 0 0) 0x7fa5f8009e50 con 0x7fa5f8009110 2024-09-16T05:48:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa5f800c3e0 con 0x7fa5f8009110 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4117 (unknown 0 0 754975712) 0x7fa60c006020 con 0x7fa5f8009110 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 >> v1:172.21.15.63:6789/0 conn(0x7fa5f8009110 legacy=0x7fa5f80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa6027fc640 1 -- 172.21.15.6:0/2661629196 >> v1:172.21.15.63:6789/0 conn(0x7fa5f8009110 legacy=0x7fa5f80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:18.939 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 62, 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 33663, 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: 1 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:18.940 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:18.941 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:48:18.942 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:48:18.943 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:48:18.944 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:48:18.945 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:48:18.946 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:18.947 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.948 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:18.949 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:48:18.950 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:48:18.951 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:48:18.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:48:18.951 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:48:18.951 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:48:18.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa5f4036ad0 legacy=0x7fa5f4038f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa617aec640 10 monclient: shutdown 2024-09-16T05:48:18.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa617aec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:48:18.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 >> v1:172.21.15.6:6789/0 conn(0x7fa6100a0da0 legacy=0x7fa6101d4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:48:18.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa616089640 1 -- 172.21.15.6:0/2661629196 reap_dead start 2024-09-16T05:48:18.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa616089640 1 -- 172.21.15.6:0/2661629196 reap_dead start 2024-09-16T05:48:18.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 shutdown_connections 2024-09-16T05:48:18.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 >> 172.21.15.6:0/2661629196 conn(0x7fa6100a1650 msgr2=0x7fa6101359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:48:18.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 shutdown_connections 2024-09-16T05:48:18.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:48:18.924+0000 7fa617aec640 1 -- 172.21.15.6:0/2661629196 wait complete. 2024-09-16T05:48:18.953 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:48:18.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:49:48.954 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T05:49:48.954 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T05:49:48.955 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T05:49:48.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T05:49:48.959 INFO:tasks.ceph.mon.c:Started 2024-09-16T05:49:48.959 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:49:48.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:49:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient: get_monmap_and_config 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient: build_initial_monmap 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient: monmap: 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:49.087584+0000 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:49.087584+0000 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 1 Processor -- start 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 1 -- start start 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient: init 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient(hunting): picked mon.noname-a con 0x7fed500a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient(hunting): picked mon.noname-b con 0x7fed500a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient(hunting): picked mon.noname-c con 0x7fed500a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed5006e520 con 0x7fed500a37a0 2024-09-16T05:49:49.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed500a0780 con 0x7fed500a1a00 2024-09-16T05:49:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed500a4c70 con 0x7fed500a4290 2024-09-16T05:49:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed578f3640 10 monclient(hunting): authenticate will time out at 1850.893838s 2024-09-16T05:49:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed5568f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fed500a37a0 0x7fed500a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52416/0 (socket says 172.21.15.6:52416) 2024-09-16T05:49:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed54e8e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fed500a4290 0x7fed5013bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35962/0 (socket says 172.21.15.6:35962) 2024-09-16T05:49:49.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.083+0000 7fed5568f640 1 -- 172.21.15.6:0/1102958028 learned_addr learned my addr 172.21.15.6:0/1102958028 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:49.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3775865854 0 0) 0x7fed500a0780 con 0x7fed500a1a00 2024-09-16T05:49:49.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient(hunting): my global_id is 154243 2024-09-16T05:49:49.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed38003530 con 0x7fed500a1a00 2024-09-16T05:49:49.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3387507354 0 0) 0x7fed38003530 con 0x7fed500a1a00 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 >> v1:172.21.15.6:6790/0 conn(0x7fed500a4290 legacy=0x7fed5013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 >> v1:172.21.15.6:6789/0 conn(0x7fed500a37a0 legacy=0x7fed500a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: _finish_hunting 0 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 monclient: found mon.noname-c 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed500a65f0 con 0x7fed500a1a00 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: _finish_auth 0 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: _check_auth_tickets 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:49.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fed40001620 con 0x7fed500a1a00 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: dump: 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.105 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 5 monclient: authenticate success, global_id 154243 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed40001980 con 0x7fed500a1a00 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 1 -- 172.21.15.6:0/1102958028 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fed40001d50 con 0x7fed500a1a00 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed47fff640 10 monclient: dump: 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.106 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 4 set_mon_vals no callback set 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient: shutdown 2024-09-16T05:49:49.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- 172.21.15.6:0/1102958028 >> v1:172.21.15.63:6789/0 conn(0x7fed500a1a00 legacy=0x7fed5013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- 172.21.15.6:0/1102958028 shutdown_connections 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- 172.21.15.6:0/1102958028 >> 172.21.15.6:0/1102958028 conn(0x7fed500a58a0 msgr2=0x7fed500a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- 172.21.15.6:0/1102958028 shutdown_connections 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- 172.21.15.6:0/1102958028 wait complete. 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient: build_initial_monmap 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient: monmap: 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:49.108 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:49.109 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.109 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:49.109 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:49.109 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:49.109 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 Processor -- start 2024-09-16T05:49:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- start start 2024-09-16T05:49:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient: init 2024-09-16T05:49:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:49:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient(hunting): picked mon.noname-c con 0x7fed500a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient(hunting): picked mon.noname-a con 0x7fed500a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient(hunting): picked mon.noname-b con 0x7fed500a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed500a0100 con 0x7fed500a37a0 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed5013f030 con 0x7fed500a4290 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed5013dda0 con 0x7fed500a1a00 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed578f3640 10 monclient(hunting): authenticate will time out at 1850.911857s 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed54e8e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fed500a37a0 0x7fed5013bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52428/0 (socket says 172.21.15.6:52428) 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed54e8e640 1 -- 172.21.15.6:0/1772320886 learned_addr learned my addr 172.21.15.6:0/1772320886 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:49.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.099+0000 7fed5568f640 1 --1- 172.21.15.6:0/1772320886 >> v1:172.21.15.6:6790/0 conn(0x7fed500a1a00 0x7fed5013b5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35970/0 (socket says 172.21.15.6:35970) 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460531539 0 0) 0x7fed500a4c70 con 0x7fed500a37a0 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient(hunting): my global_id is 144357 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed3c003530 con 0x7fed500a37a0 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3137136667 0 0) 0x7fed3c003530 con 0x7fed500a37a0 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed500a4c70 con 0x7fed500a37a0 2024-09-16T05:49:49.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fed4c001bf0 con 0x7fed500a37a0 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient(hunting): dump: 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2807224133 0 0) 0x7fed500a4c70 con 0x7fed500a37a0 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 >> v1:172.21.15.6:6790/0 conn(0x7fed500a1a00 legacy=0x7fed5013b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 >> v1:172.21.15.63:6789/0 conn(0x7fed500a4290 legacy=0x7fed501d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 monclient: found mon.a 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed500a82f0 con 0x7fed500a37a0 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient: _finish_auth 0 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:49:49.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 5 monclient: authenticate success, global_id 144357 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 10 monclient: _renew_subs 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fed5013ed50 con 0x7fed500a37a0 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 10 monclient: _renew_subs 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fed5013c400 con 0x7fed500a37a0 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed4c002020 con 0x7fed500a37a0 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fed4c004fc0 con 0x7fed500a37a0 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed477fe640 4 set_mon_vals no callback set 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 10 monclient: dump: 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fed4c0051e0 con 0x7fed500a37a0 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed30005090 con 0x7fed500a37a0 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fed4c045310 con 0x7fed500a37a0 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.103+0000 7fed5568f640 10 monclient: get_auth_request con 0x7fed3c0369c0 auth_method 0 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.107+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fed500a0100 con 0x7fed500a37a0 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.107+0000 7fed45ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.107+0000 7fed45ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.331+0000 7fed578f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:49:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.331+0000 7fed578f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:49:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.331+0000 7fed578f3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.331+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fed30005380 con 0x7fed500a37a0 2024-09-16T05:49:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed45ffb640 1 -- 172.21.15.6:0/1772320886 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3087103344) 0x7fed4c015880 con 0x7fed500a37a0 2024-09-16T05:49:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed45ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:49:49.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed45ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:49.338 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:49:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 >> v1:172.21.15.63:6812/3643207211 conn(0x7fed3c0369c0 legacy=0x7fed3c038e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed578f3640 10 monclient: shutdown 2024-09-16T05:49:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed578f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 >> v1:172.21.15.6:6789/0 conn(0x7fed500a37a0 legacy=0x7fed5013bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 shutdown_connections 2024-09-16T05:49:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 >> 172.21.15.6:0/1772320886 conn(0x7fed500a58a0 msgr2=0x7fed501400d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 shutdown_connections 2024-09-16T05:49:49.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.335+0000 7fed578f3640 1 -- 172.21.15.6:0/1772320886 wait complete. 2024-09-16T05:49:49.357 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:49:49.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:49:49.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:49:49.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient: get_monmap_and_config 2024-09-16T05:49:49.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient: build_initial_monmap 2024-09-16T05:49:49.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient: monmap: 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:49.487098+0000 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:49.487098+0000 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:49.486 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 1 Processor -- start 2024-09-16T05:49:49.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 1 -- start start 2024-09-16T05:49:49.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient: init 2024-09-16T05:49:49.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:49.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:49:49.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c00a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:49.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c00a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c00a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3c00a33a0 con 0x7fc3c00a2980 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3c00a07a0 con 0x7fc3c00a2190 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3c00a2d90 con 0x7fc3c00a2570 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient(hunting): authenticate will time out at 1851.293704s 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3bf7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc3c00a2570 0x7fc3c013b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46378/0 (socket says 172.21.15.6:46378) 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3bf7fe640 1 -- 172.21.15.6:0/2264571165 learned_addr learned my addr 172.21.15.6:0/2264571165 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c4e6b640 1 --1- 172.21.15.6:0/2264571165 >> v1:172.21.15.6:6789/0 conn(0x7fc3c00a2980 0x7fc3c013f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45530/0 (socket says 172.21.15.6:45530) 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1002935229 0 0) 0x7fc3c00a33a0 con 0x7fc3c00a2980 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient(hunting): my global_id is 144360 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3a0003530 con 0x7fc3c00a2980 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446898382 0 0) 0x7fc3c00a07a0 con 0x7fc3c00a2190 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient(hunting): my global_id is 154246 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c00a33a0 con 0x7fc3c00a2190 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2082026103 0 0) 0x7fc3a0003530 con 0x7fc3c00a2980 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 >> v1:172.21.15.6:6790/0 conn(0x7fc3c00a2570 legacy=0x7fc3c013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 >> v1:172.21.15.63:6789/0 conn(0x7fc3c00a2190 legacy=0x7fc3c00a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: _finish_hunting 0 2024-09-16T05:49:49.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 monclient: found mon.noname-a 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c00a66e0 con 0x7fc3c00a2980 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: _finish_auth 0 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: _check_auth_tickets 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3b8001550 con 0x7fc3c00a2980 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 5 monclient: authenticate success, global_id 144360 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: dump: 2024-09-16T05:49:49.493 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.494 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.494 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.494 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.494 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.494 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3b8001870 con 0x7fc3c00a2980 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 1 -- 172.21.15.6:0/2264571165 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3b8001c40 con 0x7fc3c00a2980 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 4 set_mon_vals no callback set 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3beffd640 10 monclient: dump: 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 10 monclient: shutdown 2024-09-16T05:49:49.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2264571165 >> v1:172.21.15.6:6789/0 conn(0x7fc3c00a2980 legacy=0x7fc3c013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.483+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2264571165 shutdown_connections 2024-09-16T05:49:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2264571165 >> 172.21.15.6:0/2264571165 conn(0x7fc3c00a9850 msgr2=0x7fc3c00a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2264571165 shutdown_connections 2024-09-16T05:49:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2264571165 wait complete. 2024-09-16T05:49:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: build_initial_monmap 2024-09-16T05:49:49.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: monmap: 2024-09-16T05:49:49.498 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:49.498 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:49.498 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:49.498 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:49.498 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:49.499 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.499 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:49.499 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:49.499 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:49.499 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 Processor -- start 2024-09-16T05:49:49.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- start start 2024-09-16T05:49:49.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: init 2024-09-16T05:49:49.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:49.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:49:49.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c00a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:49.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c00a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c00a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3c0122960 con 0x7fc3c00a2570 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3c0122230 con 0x7fc3c00a2980 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3c012d6d0 con 0x7fc3c00a2190 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient(hunting): authenticate will time out at 1851.298268s 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc3c00a2190 0x7fc3c01207e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46394/0 (socket says 172.21.15.6:46394) 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bffff640 1 -- 172.21.15.6:0/2544622619 learned_addr learned my addr 172.21.15.6:0/2544622619 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1145028152 0 0) 0x7fc3c0122960 con 0x7fc3c00a2570 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): my global_id is 144363 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3a4003530 con 0x7fc3c00a2570 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019862325 0 0) 0x7fc3c012d6d0 con 0x7fc3c00a2190 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): my global_id is 164099 2024-09-16T05:49:49.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c0122960 con 0x7fc3c00a2190 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 669148726 0 0) 0x7fc3a4003530 con 0x7fc3c00a2570 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3c012d6d0 con 0x7fc3c00a2570 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3a80017f0 con 0x7fc3c00a2570 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): dump: 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.511 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2593127159 0 0) 0x7fc3c0122960 con 0x7fc3c00a2190 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3a4003530 con 0x7fc3c00a2190 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3b0001c60 con 0x7fc3c00a2190 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient(hunting): dump: 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.512 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1383210501 0 0) 0x7fc3c012d6d0 con 0x7fc3c00a2570 2024-09-16T05:49:49.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 >> v1:172.21.15.6:6790/0 conn(0x7fc3c00a2190 legacy=0x7fc3c01207e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 >> v1:172.21.15.63:6789/0 conn(0x7fc3c00a2980 legacy=0x7fc3c0121600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient: _finish_hunting 0 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 monclient: found mon.a 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c0124570 con 0x7fc3c00a2570 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient: _finish_auth 0 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient: _check_auth_tickets 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 5 monclient: authenticate success, global_id 144363 2024-09-16T05:49:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: _renew_subs 2024-09-16T05:49:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c0123cd0 con 0x7fc3c00a2570 2024-09-16T05:49:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: _renew_subs 2024-09-16T05:49:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3c01d4d20 con 0x7fc3c00a2570 2024-09-16T05:49:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3a8001d00 con 0x7fc3c00a2570 2024-09-16T05:49:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3a8004990 con 0x7fc3c00a2570 2024-09-16T05:49:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3be7fc640 4 set_mon_vals no callback set 2024-09-16T05:49:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.487+0000 7fc3bcff9640 10 monclient: dump: 2024-09-16T05:49:49.517 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.517 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.517 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.517 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.517 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.517 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.517 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3c68ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3980065d0 con 0x7fc398002870 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc3a8004bb0 con 0x7fc3c00a2570 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fc3a8044d70 con 0x7fc3c00a2570 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633373936 0 0) 0x7fc3980065d0 con 0x7fc398002870 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient(hunting): my global_id is 144366 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c012d6d0 con 0x7fc398002870 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3829149697 0 0) 0x7fc3c012d6d0 con 0x7fc398002870 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3980065d0 con 0x7fc398002870 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3b0003b40 con 0x7fc398002870 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient: dump: 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.520 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3c4e6b640 10 monclient: get_auth_request con 0x7fc3a4036ac0 auth_method 0 2024-09-16T05:49:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1692827536 0 0) 0x7fc3980065d0 con 0x7fc398002870 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc39800a160 con 0x7fc398002870 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc3b0005b50 con 0x7fc398002870 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 >> v1:172.21.15.6:6789/0 conn(0x7fc398002870 legacy=0x7fc398004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.491+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 >> v1:172.21.15.6:6789/0 conn(0x7fc398002870 legacy=0x7fc398004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.495+0000 7fc3c68ce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.495+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc398009e50 con 0x7fc398009110 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.495+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452373444 0 0) 0x7fc398009e50 con 0x7fc398009110 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.495+0000 7fc3bcff9640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.495+0000 7fc3bcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.495+0000 7fc3bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.495+0000 7fc3bcff9640 10 monclient(hunting): my global_id is 144369 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.495+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3980065d0 con 0x7fc398009110 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2382333989 0 0) 0x7fc3980065d0 con 0x7fc398009110 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc398009e50 con 0x7fc398009110 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc3b0004060 con 0x7fc398009110 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 10 monclient: dump: 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.522 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2629958850 0 0) 0x7fc398009e50 con 0x7fc398009110 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc39800c3e0 con 0x7fc398009110 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 1766531938) 0x7fc3b00040d0 con 0x7fc398009110 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 >> v1:172.21.15.6:6789/0 conn(0x7fc398009110 legacy=0x7fc3980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3bcff9640 1 -- 172.21.15.6:0/2544622619 >> v1:172.21.15.6:6789/0 conn(0x7fc398009110 legacy=0x7fc3980095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:49.524 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:49:49.524 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:49:49.524 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:49:49.524 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:49:49.524 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 64, 2024-09-16T05:49:49.524 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 238113, 2024-09-16T05:49:49.524 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:49:49.524 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.525 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.526 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:49:49.527 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:49:49.528 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:49:49.528 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:49:49.528 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:49:49.528 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:49:49.528 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:49.528 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:49.528 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:49:49.529 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:49:49.529 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:49:49.529 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:49:49.529 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:49:49.529 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:49:49.529 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:49:49.529 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.530 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:49:49.531 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:49:49.532 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.533 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.535 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:49:49.536 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.537 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.537 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:49:49.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.537 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.537 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.537 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:49:49.537 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.537 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.538 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:49:49.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.538 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.538 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.538 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:49.538 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.539 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.539 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:49:49.539 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:49:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc3a4036ac0 legacy=0x7fc3a4038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c68ce640 10 monclient: shutdown 2024-09-16T05:49:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c68ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 >> v1:172.21.15.6:6789/0 conn(0x7fc3c00a2570 legacy=0x7fc3c0120ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c4e6b640 1 -- 172.21.15.6:0/2544622619 reap_dead start 2024-09-16T05:49:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c4e6b640 1 -- 172.21.15.6:0/2544622619 reap_dead start 2024-09-16T05:49:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 shutdown_connections 2024-09-16T05:49:49.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 >> 172.21.15.6:0/2544622619 conn(0x7fc3c00a9850 msgr2=0x7fc3c012fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:49.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 shutdown_connections 2024-09-16T05:49:49.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.499+0000 7fc3c68ce640 1 -- 172.21.15.6:0/2544622619 wait complete. 2024-09-16T05:49:49.543 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:49:49.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:49:49.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient: get_monmap_and_config 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient: build_initial_monmap 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient: monmap: 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:49.660761+0000 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:49.660761+0000 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:49.659 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.660 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:49.660 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:49.660 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:49.660 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 1 Processor -- start 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 1 -- start start 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient: init 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient(hunting): picked mon.noname-b con 0x7f187409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient(hunting): picked mon.noname-a con 0x7f187409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient(hunting): picked mon.noname-c con 0x7f18740a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18740a1370 con 0x7f187409e230 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f187409ead0 con 0x7f18740a0da0 2024-09-16T05:49:49.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f187409edb0 con 0x7f187409aa30 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1879184640 10 monclient(hunting): authenticate will time out at 1851.467459s 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f187259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f187409e230 0x7f18741399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45580/0 (socket says 172.21.15.6:45580) 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f187259c640 1 -- 172.21.15.6:0/1430864572 learned_addr learned my addr 172.21.15.6:0/1430864572 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3876805071 0 0) 0x7f18740a1370 con 0x7f187409e230 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1871d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1871d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1871d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.655+0000 7f1871d9b640 10 monclient(hunting): my global_id is 144372 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1854003530 con 0x7f187409e230 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270991028 0 0) 0x7f187409edb0 con 0x7f187409aa30 2024-09-16T05:49:49.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient(hunting): my global_id is 164102 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18740a1370 con 0x7f187409aa30 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2723085808 0 0) 0x7f1854003530 con 0x7f187409e230 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 >> v1:172.21.15.6:6790/0 conn(0x7f187409aa30 legacy=0x7f187409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 >> v1:172.21.15.63:6789/0 conn(0x7f18740a0da0 legacy=0x7f187413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: _finish_hunting 0 2024-09-16T05:49:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 monclient: found mon.noname-a 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187409dd50 con 0x7f187409e230 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: _finish_auth 0 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: _check_auth_tickets 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 5 monclient: authenticate success, global_id 144372 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f185c001620 con 0x7f187409e230 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: dump: 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.666 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f185c0018d0 con 0x7f187409e230 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 1 -- 172.21.15.6:0/1430864572 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f185c001ce0 con 0x7f187409e230 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 4 set_mon_vals no callback set 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1871d9b640 10 monclient: dump: 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.668 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: shutdown 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- 172.21.15.6:0/1430864572 >> v1:172.21.15.6:6789/0 conn(0x7f187409e230 legacy=0x7f18741399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- 172.21.15.6:0/1430864572 shutdown_connections 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- 172.21.15.6:0/1430864572 >> 172.21.15.6:0/1430864572 conn(0x7f18740a1650 msgr2=0x7f18740a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- 172.21.15.6:0/1430864572 shutdown_connections 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- 172.21.15.6:0/1430864572 wait complete. 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: build_initial_monmap 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: monmap: 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:49.669 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 Processor -- start 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- start start 2024-09-16T05:49:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: init 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient(hunting): picked mon.noname-c con 0x7f187409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient(hunting): picked mon.noname-a con 0x7f187409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient(hunting): picked mon.noname-b con 0x7f18740a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f187413cda0 con 0x7f187409e230 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f187413db50 con 0x7f18740a0da0 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f187406ea80 con 0x7f187409aa30 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient(hunting): authenticate will time out at 1851.470601s 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f187259c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f187409e230 0x7f18741d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45590/0 (socket says 172.21.15.6:45590) 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f187259c640 1 -- 172.21.15.6:0/2878660521 learned_addr learned my addr 172.21.15.6:0/2878660521 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1872d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f187409aa30 0x7f18741cb930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46400/0 (socket says 172.21.15.6:46400) 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 532708486 0 0) 0x7f187413cda0 con 0x7f187409e230 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): my global_id is 144375 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1858003530 con 0x7f187409e230 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556686695 0 0) 0x7f187406ea80 con 0x7f187409aa30 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): my global_id is 164105 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f187413cda0 con 0x7f187409aa30 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1291044118 0 0) 0x7f187413cda0 con 0x7f187409aa30 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f187406ea80 con 0x7f187409aa30 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1864001a20 con 0x7f187409aa30 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): dump: 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2849986951 0 0) 0x7f1858003530 con 0x7f187409e230 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f187413cda0 con 0x7f187409e230 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f185c001c20 con 0x7f187409e230 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:49.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): dump: 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633414103 0 0) 0x7f187413db50 con 0x7f18740a0da0 2024-09-16T05:49:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient(hunting): my global_id is 154252 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1858003530 con 0x7f18740a0da0 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2514209237 0 0) 0x7f187406ea80 con 0x7f187409aa30 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 >> v1:172.21.15.63:6789/0 conn(0x7f18740a0da0 legacy=0x7f18741d4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 >> v1:172.21.15.6:6789/0 conn(0x7f187409e230 legacy=0x7f18741d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 monclient: found mon.c 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18740740c0 con 0x7f187409aa30 2024-09-16T05:49:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 5 monclient: authenticate success, global_id 164105 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: _renew_subs 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f187413cac0 con 0x7f187409aa30 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: _renew_subs 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f187409cb80 con 0x7f187409aa30 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1864001bc0 con 0x7f187409aa30 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1864003ad0 con 0x7f187409aa30 2024-09-16T05:49:49.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f1870d99640 4 set_mon_vals no callback set 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.659+0000 7f186b7fe640 10 monclient: dump: 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.683 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f1879184640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f184c0065d0 con 0x7f184c002870 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1864003c70 con 0x7f187409aa30 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f1864044df0 con 0x7f187409aa30 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813921389 0 0) 0x7f184c0065d0 con 0x7f184c002870 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient(hunting): my global_id is 164108 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f187406ea80 con 0x7f184c002870 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f187359e640 10 monclient: get_auth_request con 0x7f1858036b80 auth_method 0 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3232171980 0 0) 0x7f187406ea80 con 0x7f184c002870 2024-09-16T05:49:49.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f184c0065d0 con 0x7f184c002870 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f185c004ec0 con 0x7f184c002870 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient: dump: 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1675914152 0 0) 0x7f184c0065d0 con 0x7f184c002870 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f184c00a160 con 0x7f184c002870 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f185c007200 con 0x7f184c002870 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 >> v1:172.21.15.6:6790/0 conn(0x7f184c002870 legacy=0x7f184c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.663+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 >> v1:172.21.15.6:6790/0 conn(0x7f184c002870 legacy=0x7f184c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.667+0000 7f1879184640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.667+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f184c009e30 con 0x7f184c0090f0 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 854810149 0 0) 0x7f184c009e30 con 0x7f184c0090f0 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient(hunting): my global_id is 164111 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f184c0065d0 con 0x7f184c0090f0 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2064129801 0 0) 0x7f184c0065d0 con 0x7f184c0090f0 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f184c009e30 con 0x7f184c0090f0 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f185c0056b0 con 0x7f184c0090f0 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient: dump: 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.687 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2398539856 0 0) 0x7f184c009e30 con 0x7f184c0090f0 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f184c00c3a0 con 0x7f184c0090f0 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3959 (unknown 0 0 2337894418) 0x7f185c0073b0 con 0x7f184c0090f0 2024-09-16T05:49:49.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 >> v1:172.21.15.6:6790/0 conn(0x7f184c0090f0 legacy=0x7f184c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f186b7fe640 1 -- 172.21.15.6:0/2878660521 >> v1:172.21.15.6:6790/0 conn(0x7f184c0090f0 legacy=0x7f184c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 64, 2024-09-16T05:49:49.689 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 631, 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.690 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:49:49.692 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:49:49.693 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.694 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.695 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.697 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:49:49.698 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:49:49.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.700 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.701 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:49:49.701 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:49:49.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1858036b80 legacy=0x7f1858039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f1879184640 10 monclient: shutdown 2024-09-16T05:49:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f1879184640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 >> v1:172.21.15.6:6790/0 conn(0x7f187409aa30 legacy=0x7f18741cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f187359e640 1 -- 172.21.15.6:0/2878660521 reap_dead start 2024-09-16T05:49:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f187359e640 1 -- 172.21.15.6:0/2878660521 reap_dead start 2024-09-16T05:49:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 shutdown_connections 2024-09-16T05:49:49.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 >> 172.21.15.6:0/2878660521 conn(0x7f18740a1650 msgr2=0x7f18741359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 shutdown_connections 2024-09-16T05:49:49.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.671+0000 7f1879184640 1 -- 172.21.15.6:0/2878660521 wait complete. 2024-09-16T05:49:49.704 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:49:49.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:49:49.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: get_monmap_and_config 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: build_initial_monmap 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: monmap: 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:49.823086+0000 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:49.823086+0000 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:49.822 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:49.823 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 Processor -- start 2024-09-16T05:49:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- start start 2024-09-16T05:49:49.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: init 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient(hunting): picked mon.noname-b con 0x7f5d100a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient(hunting): picked mon.noname-c con 0x7f5d100a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient(hunting): picked mon.noname-a con 0x7f5d100a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d100a33a0 con 0x7f5d100a2980 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d100a07a0 con 0x7f5d100a2570 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d100a2d90 con 0x7f5d100a2190 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient(hunting): authenticate will time out at 1851.629702s 2024-09-16T05:49:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5d100a2980 0x7f5d1013f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45592/0 (socket says 172.21.15.6:45592) 2024-09-16T05:49:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5d100a2190 0x7f5d100a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46424/0 (socket says 172.21.15.6:46424) 2024-09-16T05:49:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0f7fe640 1 -- 172.21.15.6:0/1694484669 learned_addr learned my addr 172.21.15.6:0/1694484669 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724974423 0 0) 0x7f5d100a2d90 con 0x7f5d100a2190 2024-09-16T05:49:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient(hunting): my global_id is 164114 2024-09-16T05:49:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cfc003530 con 0x7f5d100a2190 2024-09-16T05:49:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 375566643 0 0) 0x7f5d100a33a0 con 0x7f5d100a2980 2024-09-16T05:49:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient(hunting): my global_id is 144378 2024-09-16T05:49:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d100a2d90 con 0x7f5d100a2980 2024-09-16T05:49:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3252193387 0 0) 0x7f5d100a2d90 con 0x7f5d100a2980 2024-09-16T05:49:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 >> v1:172.21.15.6:6790/0 conn(0x7f5d100a2190 legacy=0x7f5d100a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 >> v1:172.21.15.63:6789/0 conn(0x7f5d100a2570 legacy=0x7f5d1013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:49:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 monclient: found mon.noname-a 2024-09-16T05:49:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d100a66e0 con 0x7f5d100a2980 2024-09-16T05:49:49.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: _finish_auth 0 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 5 monclient: authenticate success, global_id 144378 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d08001920 con 0x7f5d100a2980 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: dump: 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.836 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d08001b80 con 0x7f5d100a2980 2024-09-16T05:49:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 4 set_mon_vals no callback set 2024-09-16T05:49:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 1 -- 172.21.15.6:0/1694484669 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d08001f50 con 0x7f5d100a2980 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d0dffb640 10 monclient: dump: 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.838 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.838 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.838 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: shutdown 2024-09-16T05:49:49.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:49.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- 172.21.15.6:0/1694484669 >> v1:172.21.15.6:6789/0 conn(0x7f5d100a2980 legacy=0x7f5d1013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- 172.21.15.6:0/1694484669 shutdown_connections 2024-09-16T05:49:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- 172.21.15.6:0/1694484669 >> 172.21.15.6:0/1694484669 conn(0x7f5d100a9850 msgr2=0x7f5d100a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- 172.21.15.6:0/1694484669 shutdown_connections 2024-09-16T05:49:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 1 -- 172.21.15.6:0/1694484669 wait complete. 2024-09-16T05:49:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: build_initial_monmap 2024-09-16T05:49:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.819+0000 7f5d1571b640 10 monclient: monmap: 2024-09-16T05:49:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:49.841 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:49.842 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:49.843 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 1 Processor -- start 2024-09-16T05:49:49.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 1 -- start start 2024-09-16T05:49:49.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient: init 2024-09-16T05:49:49.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:49.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:49:49.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient(hunting): picked mon.noname-c con 0x7f5d100a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:49.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient(hunting): picked mon.noname-b con 0x7f5d100a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:49.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient(hunting): picked mon.noname-a con 0x7f5d100a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:49.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d10122cc0 con 0x7f5d100a2980 2024-09-16T05:49:49.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d10124170 con 0x7f5d100a2570 2024-09-16T05:49:49.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d10121430 con 0x7f5d100a2190 2024-09-16T05:49:49.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:49.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient(hunting): authenticate will time out at 1851.632717s 2024-09-16T05:49:49.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d0effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5d100a2190 0x7f5d10120610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46440/0 (socket says 172.21.15.6:46440) 2024-09-16T05:49:49.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d0effd640 1 -- 172.21.15.6:0/3689883735 learned_addr learned my addr 172.21.15.6:0/3689883735 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:49.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1767463625 0 0) 0x7f5d10122cc0 con 0x7f5d100a2980 2024-09-16T05:49:49.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): my global_id is 144381 2024-09-16T05:49:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf0003530 con 0x7f5d100a2980 2024-09-16T05:49:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2033068647 0 0) 0x7f5d10121430 con 0x7f5d100a2190 2024-09-16T05:49:49.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): my global_id is 164117 2024-09-16T05:49:49.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d10122cc0 con 0x7f5d100a2190 2024-09-16T05:49:49.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1196803479 0 0) 0x7f5cf0003530 con 0x7f5d100a2980 2024-09-16T05:49:49.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d10121430 con 0x7f5d100a2980 2024-09-16T05:49:49.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d08001d30 con 0x7f5d100a2980 2024-09-16T05:49:49.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:49.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:49.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): dump: 2024-09-16T05:49:49.849 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.849 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.849 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.849 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.849 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.850 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.850 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.850 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.850 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.850 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:49.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635253156 0 0) 0x7f5d10124170 con 0x7f5d100a2570 2024-09-16T05:49:49.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): my global_id is 154258 2024-09-16T05:49:49.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf0003530 con 0x7f5d100a2570 2024-09-16T05:49:49.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2891946387 0 0) 0x7f5d10122cc0 con 0x7f5d100a2190 2024-09-16T05:49:49.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d10124170 con 0x7f5d100a2190 2024-09-16T05:49:49.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d00001a40 con 0x7f5d100a2190 2024-09-16T05:49:49.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:49.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:49.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): dump: 2024-09-16T05:49:49.853 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.853 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.853 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.853 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.853 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.853 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2858227922 0 0) 0x7f5d10121430 con 0x7f5d100a2980 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 >> v1:172.21.15.6:6790/0 conn(0x7f5d100a2190 legacy=0x7f5d10120610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 >> v1:172.21.15.63:6789/0 conn(0x7f5d100a2570 legacy=0x7f5d10120d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: _finish_hunting 0 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 monclient: found mon.a 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d10124450 con 0x7f5d100a2980 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: _finish_auth 0 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: _check_auth_tickets 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 5 monclient: authenticate success, global_id 144381 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient: _renew_subs 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d100a7f30 con 0x7f5d100a2980 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient: _renew_subs 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d101225a0 con 0x7f5d100a2980 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d08004f80 con 0x7f5d100a2980 2024-09-16T05:49:49.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d080052d0 con 0x7f5d100a2980 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: dump: 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.856 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d0d7fa640 4 set_mon_vals no callback set 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5d10075290 con 0x7f5d101401a0 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5d08010070 con 0x7f5d100a2980 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f5d08045140 con 0x7f5d100a2980 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d0e7fc640 10 monclient: get_auth_request con 0x7f5cf0036c40 auth_method 0 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1521763192 0 0) 0x7f5d10075290 con 0x7f5d101401a0 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient(hunting): my global_id is 154261 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d10121430 con 0x7f5d101401a0 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1331506281 0 0) 0x7f5d10121430 con 0x7f5d101401a0 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d10075290 con 0x7f5d101401a0 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d00004990 con 0x7f5d101401a0 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.823+0000 7f5d077fe640 10 monclient: dump: 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.858 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.827+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2588307109 0 0) 0x7f5d10075290 con 0x7f5d101401a0 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.827+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d10140580 con 0x7f5d101401a0 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.827+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5d00006bd0 con 0x7f5d101401a0 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.827+0000 7f5d077fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.827+0000 7f5d077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.827+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 >> v1:172.21.15.63:6789/0 conn(0x7f5d101401a0 legacy=0x7f5d1009c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.827+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 >> v1:172.21.15.63:6789/0 conn(0x7f5d101401a0 legacy=0x7f5d1009c590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.831+0000 7f5d1571b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:49:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.831+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5d10124870 con 0x7f5d1002d660 2024-09-16T05:49:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566576413 0 0) 0x7f5d10124870 con 0x7f5d1002d660 2024-09-16T05:49:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient(hunting): _negotiate 2024-09-16T05:49:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:49.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient(hunting): my global_id is 154264 2024-09-16T05:49:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d10075290 con 0x7f5d1002d660 2024-09-16T05:49:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1717567665 0 0) 0x7f5d10075290 con 0x7f5d1002d660 2024-09-16T05:49:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d10124870 con 0x7f5d1002d660 2024-09-16T05:49:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5d000062f0 con 0x7f5d1002d660 2024-09-16T05:49:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:49.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:49:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient: dump: 2024-09-16T05:49:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:49.863 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:49.864 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:49.864 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:49.864 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:49.864 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:49.864 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:49.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4076227068 0 0) 0x7f5d10124870 con 0x7f5d1002d660 2024-09-16T05:49:49.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d1002da40 con 0x7f5d1002d660 2024-09-16T05:49:49.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 198281703) 0x7f5d000060d0 con 0x7f5d1002d660 2024-09-16T05:49:49.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:49:49.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:49.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 >> v1:172.21.15.63:6789/0 conn(0x7f5d1002d660 legacy=0x7f5d1009cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d077fe640 1 -- 172.21.15.6:0/3689883735 >> v1:172.21.15.63:6789/0 conn(0x7f5d1002d660 legacy=0x7f5d1009cca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:49.866 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:49:49.866 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:49:49.866 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:49:49.866 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:49:49.866 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 64, 2024-09-16T05:49:49.866 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 124574, 2024-09-16T05:49:49.867 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:49:49.867 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:49:49.867 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:49:49.867 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:49:49.867 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.867 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:49:49.868 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:49.868 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:49:49.868 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:49:49.868 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.868 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.868 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.868 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.869 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.869 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.869 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.869 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.869 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.870 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.871 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.871 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.871 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.871 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.871 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.871 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.872 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:49:49.872 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:49:49.872 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:49:49.872 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:49:49.872 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:49:49.873 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:49:49.873 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:49.873 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:49.873 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:49:49.873 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:49:49.873 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:49:49.874 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:49:49.874 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:49:49.874 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:49:49.874 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:49:49.874 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:49.875 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:49:49.875 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:49.875 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:49.875 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:49.875 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:49.876 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:49.876 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:49.876 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:49.876 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:49.876 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:49.876 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:49.877 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:49.877 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.877 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:49:49.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.877 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:49:49.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.878 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:49:49.878 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:49:49.878 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.878 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.879 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.879 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:49:49.879 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.879 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.879 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.880 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:49:49.882 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:49:49.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.884 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.884 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.884 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:49.884 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.884 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:49.885 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:49:49.886 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:49.886 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:49.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:49.886 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:49:49.886 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:49:49.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5cf0036c40 legacy=0x7f5cf0039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d1571b640 10 monclient: shutdown 2024-09-16T05:49:49.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d1571b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:49.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 >> v1:172.21.15.6:6789/0 conn(0x7f5d100a2980 legacy=0x7f5d101d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:49.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d0f7fe640 1 -- 172.21.15.6:0/3689883735 reap_dead start 2024-09-16T05:49:49.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d0f7fe640 1 -- 172.21.15.6:0/3689883735 reap_dead start 2024-09-16T05:49:49.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 shutdown_connections 2024-09-16T05:49:49.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 >> 172.21.15.6:0/3689883735 conn(0x7f5d100a9850 msgr2=0x7f5d1006ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:49.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 shutdown_connections 2024-09-16T05:49:49.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.835+0000 7f5d1571b640 1 -- 172.21.15.6:0/3689883735 wait complete. 2024-09-16T05:49:49.888 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:49:49.888 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:49:49.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:49:50.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient: get_monmap_and_config 2024-09-16T05:49:50.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient: build_initial_monmap 2024-09-16T05:49:50.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient: monmap: 2024-09-16T05:49:50.002 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:50.002 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:50.003 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:50.004156+0000 2024-09-16T05:49:50.003 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:50.004156+0000 2024-09-16T05:49:50.003 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:50.003 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:50.003 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:50.003 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:50.003 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:50.003 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:50.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 1 Processor -- start 2024-09-16T05:49:50.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 1 -- start start 2024-09-16T05:49:50.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient: init 2024-09-16T05:49:50.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient(hunting): picked mon.noname-a con 0x7f2bf406bc10 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient(hunting): picked mon.noname-c con 0x7f2bf406bff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient(hunting): picked mon.noname-b con 0x7f2bf413a970 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bf4072190 con 0x7f2bf406bc10 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bf4072470 con 0x7f2bf406bff0 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bf40a16f0 con 0x7f2bf413a970 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bfb4df640 10 monclient(hunting): authenticate will time out at 1851.811519s 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bf927b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2bf406bc10 0x7f2bf4073460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45612/0 (socket says 172.21.15.6:45612) 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bf927b640 1 -- 172.21.15.6:0/726167050 learned_addr learned my addr 172.21.15.6:0/726167050 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:49.999+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 608471344 0 0) 0x7f2bf4072190 con 0x7f2bf406bc10 2024-09-16T05:49:50.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient(hunting): my global_id is 144384 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be0003530 con 0x7f2bf406bc10 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 546256151 0 0) 0x7f2bf40a16f0 con 0x7f2bf413a970 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient(hunting): _negotiate 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient(hunting): my global_id is 164120 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bf4072190 con 0x7f2bf413a970 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3450516108 0 0) 0x7f2bf4072190 con 0x7f2bf413a970 2024-09-16T05:49:50.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 >> v1:172.21.15.63:6789/0 conn(0x7f2bf406bff0 legacy=0x7f2bf406c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 >> v1:172.21.15.6:6789/0 conn(0x7f2bf406bc10 legacy=0x7f2bf4073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: _finish_hunting 0 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 monclient: found mon.noname-b 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf41414b0 con 0x7f2bf413a970 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: _finish_auth 0 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: _check_auth_tickets 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2bec001530 con 0x7f2bf413a970 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:49:50.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: dump: 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 5 monclient: authenticate success, global_id 164120 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2bec001890 con 0x7f2bf413a970 2024-09-16T05:49:50.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 1 -- 172.21.15.6:0/726167050 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2bec001ca0 con 0x7f2bf413a970 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bebfff640 10 monclient: dump: 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:50.011 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 4 set_mon_vals no callback set 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient: shutdown 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- 172.21.15.6:0/726167050 >> v1:172.21.15.6:6790/0 conn(0x7f2bf413a970 legacy=0x7f2bf413cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- 172.21.15.6:0/726167050 shutdown_connections 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- 172.21.15.6:0/726167050 >> 172.21.15.6:0/726167050 conn(0x7f2bf40a2580 msgr2=0x7f2bf40a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- 172.21.15.6:0/726167050 shutdown_connections 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- 172.21.15.6:0/726167050 wait complete. 2024-09-16T05:49:50.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient: build_initial_monmap 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient: monmap: 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:50.013 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:50.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 Processor -- start 2024-09-16T05:49:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- start start 2024-09-16T05:49:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient: init 2024-09-16T05:49:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:49:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient(hunting): picked mon.noname-a con 0x7f2bf406bc10 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient(hunting): picked mon.noname-c con 0x7f2bf406bff0 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient(hunting): picked mon.noname-b con 0x7f2bf413a970 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bf409d0d0 con 0x7f2bf406bc10 2024-09-16T05:49:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bf409d3b0 con 0x7f2bf413a970 2024-09-16T05:49:50.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bf409b850 con 0x7f2bf406bff0 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bfb4df640 10 monclient(hunting): authenticate will time out at 1851.815210s 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bf927b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2bf406bc10 0x7f2bf409aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45616/0 (socket says 172.21.15.6:45616) 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2bf927b640 1 -- 172.21.15.6:0/486648962 learned_addr learned my addr 172.21.15.6:0/486648962 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2503044487 0 0) 0x7f2bf409d0d0 con 0x7f2bf406bc10 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): my global_id is 144387 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bd8003530 con 0x7f2bf406bc10 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894978789 0 0) 0x7f2bf409b850 con 0x7f2bf406bff0 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:49:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): my global_id is 164123 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bf409d0d0 con 0x7f2bf406bff0 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1706017655 0 0) 0x7f2bd8003530 con 0x7f2bf406bc10 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bf409b850 con 0x7f2bf406bc10 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2bf0001dc0 con 0x7f2bf406bc10 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): dump: 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3879432180 0 0) 0x7f2bf409d0d0 con 0x7f2bf406bff0 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bd8003530 con 0x7f2bf406bff0 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2be4001860 con 0x7f2bf406bff0 2024-09-16T05:49:50.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): dump: 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:50.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100237393 0 0) 0x7f2bf409d3b0 con 0x7f2bf413a970 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 10 monclient(hunting): my global_id is 154270 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bf409d0d0 con 0x7f2bf413a970 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.003+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1211247554 0 0) 0x7f2bf409b850 con 0x7f2bf406bc10 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 >> v1:172.21.15.6:6790/0 conn(0x7f2bf406bff0 legacy=0x7f2bf409b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 >> v1:172.21.15.63:6789/0 conn(0x7f2bf413a970 legacy=0x7f2bf41d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 1 monclient: found mon.a 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:50.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf409c140 con 0x7f2bf406bc10 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 5 monclient: authenticate success, global_id 144387 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 10 monclient: _renew_subs 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf4074010 con 0x7f2bf406bc10 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 10 monclient: _renew_subs 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2bf409bf00 con 0x7f2bf406bc10 2024-09-16T05:49:50.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2bf0002020 con 0x7f2bf406bc10 2024-09-16T05:49:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2bf0004fc0 con 0x7f2bf406bc10 2024-09-16T05:49:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2beb7fe640 4 set_mon_vals no callback set 2024-09-16T05:49:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 10 monclient: dump: 2024-09-16T05:49:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:50.032 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bd4005090 con 0x7f2bf406bc10 2024-09-16T05:49:50.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2bf000fd60 con 0x7f2bf406bc10 2024-09-16T05:49:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f2bf00451c0 con 0x7f2bf406bc10 2024-09-16T05:49:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.007+0000 7f2bf8a7a640 10 monclient: get_auth_request con 0x7f2bd8036bd0 auth_method 0 2024-09-16T05:49:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.011+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2bf0015790 con 0x7f2bf406bc10 2024-09-16T05:49:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.011+0000 7f2be9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:50.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.011+0000 7f2be9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.231+0000 7f2bfb4df640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:49:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.231+0000 7f2bfb4df640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:49:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.231+0000 7f2bfb4df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.231+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f2bd4005b90 con 0x7f2bf406bc10 2024-09-16T05:49:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f2be9ffb640 1 -- 172.21.15.6:0/486648962 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f2bf0015090 con 0x7f2bf406bc10 2024-09-16T05:49:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f2be9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:49:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f2be9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:50.239 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.240 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=8,logm=51} crc {auth=3327125092,config=3541143443,health=1363563658,kv=2617142028,logm=598141370}) 2024-09-16T05:49:50.240 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=11,kv=11,logm=47} crc {auth=3269671989,config=3541143443,health=2313241988,kv=2361287224,logm=574633663}) 2024-09-16T05:49:50.240 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.240 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=8,logm=51} crc {auth=3327125092,config=3541143443,health=1363563658,kv=2617142028,logm=598141370}) 2024-09-16T05:49:50.240 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.235+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=10,logm=46} crc {auth=1097153771,config=3541143443,health=1089699483,kv=2000773687,logm=2870167737}) 2024-09-16T05:49:50.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2bd8036bd0 legacy=0x7f2bd8039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:50.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f2bfb4df640 10 monclient: shutdown 2024-09-16T05:49:50.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f2bfb4df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:50.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 >> v1:172.21.15.6:6789/0 conn(0x7f2bf406bc10 legacy=0x7f2bf409aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:50.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 shutdown_connections 2024-09-16T05:49:50.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 >> 172.21.15.6:0/486648962 conn(0x7f2bf40a2580 msgr2=0x7f2bf413a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:50.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 shutdown_connections 2024-09-16T05:49:50.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f2bfb4df640 1 -- 172.21.15.6:0/486648962 wait complete. 2024-09-16T05:49:50.243 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.243 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062923559}) 2024-09-16T05:49:50.243 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3171898073}) 2024-09-16T05:49:50.243 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.243 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062923559}) 2024-09-16T05:49:50.243 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.239+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=211766670}) 2024-09-16T05:49:50.245 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.245 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=461023781}) 2024-09-16T05:49:50.245 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2688451980}) 2024-09-16T05:49:50.245 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.245 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=461023781}) 2024-09-16T05:49:50.245 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3853294588}) 2024-09-16T05:49:50.247 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.247 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535777463}) 2024-09-16T05:49:50.247 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271401575}) 2024-09-16T05:49:50.247 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.248 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535777463}) 2024-09-16T05:49:50.248 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.243+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3854614009}) 2024-09-16T05:49:50.249 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.249 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249874385}) 2024-09-16T05:49:50.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803982932}) 2024-09-16T05:49:50.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249874385}) 2024-09-16T05:49:50.250 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702678423}) 2024-09-16T05:49:50.252 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.252 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114016149}) 2024-09-16T05:49:50.252 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1348762191}) 2024-09-16T05:49:50.252 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.252 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114016149}) 2024-09-16T05:49:50.252 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.247+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2971044347}) 2024-09-16T05:49:50.254 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.254 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5132864}) 2024-09-16T05:49:50.254 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1286642117}) 2024-09-16T05:49:50.255 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.255 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5132864}) 2024-09-16T05:49:50.255 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442540248}) 2024-09-16T05:49:50.257 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.257 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816060474}) 2024-09-16T05:49:50.257 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2871956698}) 2024-09-16T05:49:50.257 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.257 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816060474}) 2024-09-16T05:49:50.257 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.251+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354928143}) 2024-09-16T05:49:50.258 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:49:50.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.255+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.255+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258920006}) 2024-09-16T05:49:50.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.255+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=60487024}) 2024-09-16T05:49:50.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.255+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.255+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258920006}) 2024-09-16T05:49:50.259 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.255+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804977872}) 2024-09-16T05:49:50.261 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922041294}) 2024-09-16T05:49:50.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3199806033}) 2024-09-16T05:49:50.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922041294}) 2024-09-16T05:49:50.262 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3355732361}) 2024-09-16T05:49:50.264 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.264 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158311587}) 2024-09-16T05:49:50.264 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3369825450}) 2024-09-16T05:49:50.264 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.264 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158311587}) 2024-09-16T05:49:50.265 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.259+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1880283705}) 2024-09-16T05:49:50.266 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.266 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960379727}) 2024-09-16T05:49:50.267 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614635188}) 2024-09-16T05:49:50.267 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.267 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960379727}) 2024-09-16T05:49:50.267 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717981197}) 2024-09-16T05:49:50.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3915706576}) 2024-09-16T05:49:50.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1891247468}) 2024-09-16T05:49:50.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3915706576}) 2024-09-16T05:49:50.269 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.263+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831191667}) 2024-09-16T05:49:50.271 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.267+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.271 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.267+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110894833}) 2024-09-16T05:49:50.271 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.267+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060314227}) 2024-09-16T05:49:50.271 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.267+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.271 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.267+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110894833}) 2024-09-16T05:49:50.272 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.267+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886805370}) 2024-09-16T05:49:50.274 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.274 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521260954}) 2024-09-16T05:49:50.274 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958392106}) 2024-09-16T05:49:50.274 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.274 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521260954}) 2024-09-16T05:49:50.275 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1889153372}) 2024-09-16T05:49:50.276 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.276 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2837984218}) 2024-09-16T05:49:50.276 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3549987262}) 2024-09-16T05:49:50.276 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.277 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2837984218}) 2024-09-16T05:49:50.277 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.271+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656290636}) 2024-09-16T05:49:50.278 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.278 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205549474}) 2024-09-16T05:49:50.279 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=448894779}) 2024-09-16T05:49:50.279 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.279 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205549474}) 2024-09-16T05:49:50.279 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=861945678}) 2024-09-16T05:49:50.281 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.281 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145191951}) 2024-09-16T05:49:50.281 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2599515430}) 2024-09-16T05:49:50.282 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.282 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145191951}) 2024-09-16T05:49:50.282 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.275+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740341316}) 2024-09-16T05:49:50.283 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.279+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.283 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.279+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=633255671}) 2024-09-16T05:49:50.283 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.279+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1759272671}) 2024-09-16T05:49:50.284 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.279+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.284 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.279+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=633255671}) 2024-09-16T05:49:50.284 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.279+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1884702658}) 2024-09-16T05:49:50.285 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.279+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.285 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.279+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351422716}) 2024-09-16T05:49:50.286 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1373786135}) 2024-09-16T05:49:50.286 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.286 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351422716}) 2024-09-16T05:49:50.286 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=765828826}) 2024-09-16T05:49:50.287 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.288 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562906584}) 2024-09-16T05:49:50.288 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=380718610}) 2024-09-16T05:49:50.288 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.288 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562906584}) 2024-09-16T05:49:50.288 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.283+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3074190453}) 2024-09-16T05:49:50.289 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.290 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161570904}) 2024-09-16T05:49:50.290 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2165054237}) 2024-09-16T05:49:50.290 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.290 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161570904}) 2024-09-16T05:49:50.290 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610936800}) 2024-09-16T05:49:50.292 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.292 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=282528426}) 2024-09-16T05:49:50.292 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836203565}) 2024-09-16T05:49:50.292 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.292 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=282528426}) 2024-09-16T05:49:50.292 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.287+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=954273918}) 2024-09-16T05:49:50.294 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.294 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056667891}) 2024-09-16T05:49:50.294 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=347514957}) 2024-09-16T05:49:50.294 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.294 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056667891}) 2024-09-16T05:49:50.294 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011136374}) 2024-09-16T05:49:50.296 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.297 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982997384}) 2024-09-16T05:49:50.297 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2195957726}) 2024-09-16T05:49:50.297 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.297 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982997384}) 2024-09-16T05:49:50.297 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.291+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230970364}) 2024-09-16T05:49:50.299 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.299 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262666086}) 2024-09-16T05:49:50.299 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1194462546}) 2024-09-16T05:49:50.299 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.299 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262666086}) 2024-09-16T05:49:50.299 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=846892024}) 2024-09-16T05:49:50.301 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.301 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861178176}) 2024-09-16T05:49:50.301 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596906909}) 2024-09-16T05:49:50.301 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.301 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861178176}) 2024-09-16T05:49:50.301 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.295+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2262078740}) 2024-09-16T05:49:50.303 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.299+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.303 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.299+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271906004}) 2024-09-16T05:49:50.303 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.299+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3020080260}) 2024-09-16T05:49:50.303 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.299+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.303 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.299+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271906004}) 2024-09-16T05:49:50.304 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.299+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586510469}) 2024-09-16T05:49:50.305 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.305 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550200766}) 2024-09-16T05:49:50.305 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2316765271}) 2024-09-16T05:49:50.305 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.305 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550200766}) 2024-09-16T05:49:50.306 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2208666271}) 2024-09-16T05:49:50.307 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.307 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028660473}) 2024-09-16T05:49:50.307 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3530486779}) 2024-09-16T05:49:50.307 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.307 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028660473}) 2024-09-16T05:49:50.307 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.303+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3564184965}) 2024-09-16T05:49:50.309 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.309 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294240334}) 2024-09-16T05:49:50.309 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295169270}) 2024-09-16T05:49:50.310 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.310 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294240334}) 2024-09-16T05:49:50.310 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975643828}) 2024-09-16T05:49:50.312 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.312 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782994896}) 2024-09-16T05:49:50.312 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3549328392}) 2024-09-16T05:49:50.312 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.312 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782994896}) 2024-09-16T05:49:50.313 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.307+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197168864}) 2024-09-16T05:49:50.314 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.314 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2112374558}) 2024-09-16T05:49:50.314 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1029892837}) 2024-09-16T05:49:50.314 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.315 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2112374558}) 2024-09-16T05:49:50.315 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2487745925}) 2024-09-16T05:49:50.316 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.316 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648953253}) 2024-09-16T05:49:50.316 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868508991}) 2024-09-16T05:49:50.317 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.317 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648953253}) 2024-09-16T05:49:50.317 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.311+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1606650160}) 2024-09-16T05:49:50.319 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.319 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505702015}) 2024-09-16T05:49:50.319 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1333691644}) 2024-09-16T05:49:50.319 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.319 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505702015}) 2024-09-16T05:49:50.319 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3494037577}) 2024-09-16T05:49:50.321 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.321 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556597440}) 2024-09-16T05:49:50.321 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398734910}) 2024-09-16T05:49:50.321 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.315+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.321 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.319+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556597440}) 2024-09-16T05:49:50.322 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.319+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004027740}) 2024-09-16T05:49:50.323 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.319+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.324 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.319+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444364269}) 2024-09-16T05:49:50.324 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.319+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3707475976}) 2024-09-16T05:49:50.324 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.319+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.324 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.319+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444364269}) 2024-09-16T05:49:50.324 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.319+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3398997725}) 2024-09-16T05:49:50.327 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.327 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854363511}) 2024-09-16T05:49:50.327 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=210477834}) 2024-09-16T05:49:50.327 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.327 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854363511}) 2024-09-16T05:49:50.327 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763053008}) 2024-09-16T05:49:50.329 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.329 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723245169}) 2024-09-16T05:49:50.329 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2581735467}) 2024-09-16T05:49:50.329 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.329 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723245169}) 2024-09-16T05:49:50.329 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.323+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2250658360}) 2024-09-16T05:49:50.331 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.331 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655883648}) 2024-09-16T05:49:50.331 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078717150}) 2024-09-16T05:49:50.331 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.331 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655883648}) 2024-09-16T05:49:50.331 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2870922498}) 2024-09-16T05:49:50.333 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.333 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979042574}) 2024-09-16T05:49:50.333 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1311125523}) 2024-09-16T05:49:50.333 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.333 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979042574}) 2024-09-16T05:49:50.333 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.327+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=487079838}) 2024-09-16T05:49:50.335 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.331+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.335 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.331+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165609477}) 2024-09-16T05:49:50.335 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.331+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40401931}) 2024-09-16T05:49:50.335 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.331+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.335 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.331+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165609477}) 2024-09-16T05:49:50.336 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.331+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1878712172}) 2024-09-16T05:49:50.337 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.337 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961203646}) 2024-09-16T05:49:50.337 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2007088715}) 2024-09-16T05:49:50.338 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.338 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961203646}) 2024-09-16T05:49:50.338 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1880515586}) 2024-09-16T05:49:50.339 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.339 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=992944482}) 2024-09-16T05:49:50.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939650683}) 2024-09-16T05:49:50.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=992944482}) 2024-09-16T05:49:50.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.335+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402757090}) 2024-09-16T05:49:50.341 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.341 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698980518}) 2024-09-16T05:49:50.341 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495810177}) 2024-09-16T05:49:50.341 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.341 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698980518}) 2024-09-16T05:49:50.341 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1509693415}) 2024-09-16T05:49:50.343 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.343 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558911333}) 2024-09-16T05:49:50.343 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3221744411}) 2024-09-16T05:49:50.343 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.343 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558911333}) 2024-09-16T05:49:50.343 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.339+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610784000}) 2024-09-16T05:49:50.345 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.345 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635610146}) 2024-09-16T05:49:50.345 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1514382463}) 2024-09-16T05:49:50.345 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.345 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635610146}) 2024-09-16T05:49:50.345 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1895617008}) 2024-09-16T05:49:50.347 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.347 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018377401}) 2024-09-16T05:49:50.347 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4187475203}) 2024-09-16T05:49:50.347 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.347 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018377401}) 2024-09-16T05:49:50.347 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.343+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078061626}) 2024-09-16T05:49:50.349 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.349 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196222429}) 2024-09-16T05:49:50.349 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550561891}) 2024-09-16T05:49:50.349 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.349 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196222429}) 2024-09-16T05:49:50.349 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289089675}) 2024-09-16T05:49:50.351 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.351 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035218859}) 2024-09-16T05:49:50.351 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4140456807}) 2024-09-16T05:49:50.351 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.351 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035218859}) 2024-09-16T05:49:50.351 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.347+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842083455}) 2024-09-16T05:49:50.353 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.353 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2203760740}) 2024-09-16T05:49:50.353 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1651025383}) 2024-09-16T05:49:50.353 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.353 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2203760740}) 2024-09-16T05:49:50.353 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568402334}) 2024-09-16T05:49:50.355 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.355 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783919738}) 2024-09-16T05:49:50.355 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476000716}) 2024-09-16T05:49:50.355 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.356 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783919738}) 2024-09-16T05:49:50.356 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.351+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278131682}) 2024-09-16T05:49:50.357 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.357 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880705316}) 2024-09-16T05:49:50.357 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1773111262}) 2024-09-16T05:49:50.357 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.357 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880705316}) 2024-09-16T05:49:50.358 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574554756}) 2024-09-16T05:49:50.359 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.359 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=36774322}) 2024-09-16T05:49:50.359 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059231969}) 2024-09-16T05:49:50.359 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.359 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=36774322}) 2024-09-16T05:49:50.359 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.355+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171863937}) 2024-09-16T05:49:50.361 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.361 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790411141}) 2024-09-16T05:49:50.361 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=298107195}) 2024-09-16T05:49:50.361 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.361 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790411141}) 2024-09-16T05:49:50.361 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737552628}) 2024-09-16T05:49:50.364 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.364 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=24} crc {logm=2330623684,mdsmap=1832746017,mgr=1137420307,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3577039827,mon_config_key=4183301857,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=3168080625}) 2024-09-16T05:49:50.364 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=21} crc {logm=753095131,mdsmap=1832746017,mgr=1026383475,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3270707914,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1384745985}) 2024-09-16T05:49:50.364 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.365 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=24} crc {logm=2330623684,mdsmap=1832746017,mgr=1137420307,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3577039827,mon_config_key=4183301857,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=3168080625}) 2024-09-16T05:49:50.365 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.359+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=24} crc {logm=3255671699,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=518028644,mon_config_key=1533998164,monmap=3625042558,osd_metadata=1319166418,osd_pg_creating=1754493876,osd_snap=2941457380,osdmap=3866591864}) 2024-09-16T05:49:50.377 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.371+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.377 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.371+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=2206885730}) 2024-09-16T05:49:50.377 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.371+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=26} crc {osdmap=2047436541}) 2024-09-16T05:49:50.377 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.371+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:49:50.377 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.371+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=2206885730}) 2024-09-16T05:49:50.377 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:49:50.371+0000 7f36e479b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=1657804941}) 2024-09-16T05:49:51.259 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:49:51.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:49:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient: get_monmap_and_config 2024-09-16T05:49:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient: build_initial_monmap 2024-09-16T05:49:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient: monmap: 2024-09-16T05:49:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:51.392721+0000 2024-09-16T05:49:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:51.392721+0000 2024-09-16T05:49:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 1 Processor -- start 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 1 -- start start 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient: init 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient(hunting): picked mon.noname-c con 0x7fb97809aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:51.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient(hunting): picked mon.noname-a con 0x7fb97809e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient(hunting): picked mon.noname-b con 0x7fb9780a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9780a1370 con 0x7fb97809e230 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97809ead0 con 0x7fb97809aa30 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97809edb0 con 0x7fb9780a0da0 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb97d98f640 10 monclient(hunting): authenticate will time out at 1853.199335s 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb9777fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb9780a0da0 0x7fb97813d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46486/0 (socket says 172.21.15.6:46486) 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb9777fe640 1 -- 172.21.15.6:0/3273667221 learned_addr learned my addr 172.21.15.6:0/3273667221 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb9767fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb97809e230 0x7fb9781399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45618/0 (socket says 172.21.15.6:45618) 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2921021462 0 0) 0x7fb97809edb0 con 0x7fb9780a0da0 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 10 monclient(hunting): my global_id is 164126 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb964003530 con 0x7fb9780a0da0 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 106704382 0 0) 0x7fb9780a1370 con 0x7fb97809e230 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.387+0000 7fb975ffb640 10 monclient(hunting): my global_id is 144390 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb97809edb0 con 0x7fb97809e230 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 588107458 0 0) 0x7fb964003530 con 0x7fb9780a0da0 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 >> v1:172.21.15.63:6789/0 conn(0x7fb97809aa30 legacy=0x7fb97809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 >> v1:172.21.15.6:6789/0 conn(0x7fb97809e230 legacy=0x7fb9781399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:49:51.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 monclient: found mon.noname-b 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97809dd50 con 0x7fb9780a0da0 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: _finish_auth 0 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 5 monclient: authenticate success, global_id 164126 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb970001620 con 0x7fb9780a0da0 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: dump: 2024-09-16T05:49:51.404 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb970001880 con 0x7fb9780a0da0 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:51.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 4 set_mon_vals no callback set 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 1 -- 172.21.15.6:0/3273667221 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb970001bd0 con 0x7fb9780a0da0 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb975ffb640 10 monclient: dump: 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.406 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: shutdown 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- 172.21.15.6:0/3273667221 >> v1:172.21.15.6:6790/0 conn(0x7fb9780a0da0 legacy=0x7fb97813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- 172.21.15.6:0/3273667221 shutdown_connections 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- 172.21.15.6:0/3273667221 >> 172.21.15.6:0/3273667221 conn(0x7fb9780a1650 msgr2=0x7fb9780a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- 172.21.15.6:0/3273667221 shutdown_connections 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- 172.21.15.6:0/3273667221 wait complete. 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: build_initial_monmap 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: monmap: 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:51.407 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 Processor -- start 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- start start 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: init 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient(hunting): picked mon.noname-b con 0x7fb97809aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient(hunting): picked mon.noname-c con 0x7fb97809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient(hunting): picked mon.noname-a con 0x7fb9780a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97809bcd0 con 0x7fb9780a0da0 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97809d8f0 con 0x7fb97809aa30 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb978071910 con 0x7fb97809e230 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient(hunting): authenticate will time out at 1853.202696s 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb9767fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb97809e230 0x7fb97806f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46488/0 (socket says 172.21.15.6:46488) 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb9767fc640 1 -- 172.21.15.6:0/2707123810 learned_addr learned my addr 172.21.15.6:0/2707123810 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2624364930 0 0) 0x7fb97809bcd0 con 0x7fb9780a0da0 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): my global_id is 144393 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb958003530 con 0x7fb9780a0da0 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734497493 0 0) 0x7fb978071910 con 0x7fb97809e230 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): my global_id is 164129 2024-09-16T05:49:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb97809bcd0 con 0x7fb97809e230 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1324269207 0 0) 0x7fb97809bcd0 con 0x7fb97809e230 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb978071910 con 0x7fb97809e230 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb968001d90 con 0x7fb97809e230 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): dump: 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 72149731 0 0) 0x7fb958003530 con 0x7fb9780a0da0 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb97809bcd0 con 0x7fb9780a0da0 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb970001520 con 0x7fb9780a0da0 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): dump: 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2147582677 0 0) 0x7fb97809d8f0 con 0x7fb97809aa30 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient(hunting): my global_id is 154276 2024-09-16T05:49:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb958003530 con 0x7fb97809aa30 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3761263387 0 0) 0x7fb978071910 con 0x7fb97809e230 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 >> v1:172.21.15.63:6789/0 conn(0x7fb97809aa30 legacy=0x7fb97806ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 >> v1:172.21.15.6:6789/0 conn(0x7fb9780a0da0 legacy=0x7fb97806f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 monclient: found mon.c 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb978139e60 con 0x7fb97809e230 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient: _finish_auth 0 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 5 monclient: authenticate success, global_id 164129 2024-09-16T05:49:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: _renew_subs 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb9780722e0 con 0x7fb97809e230 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: _renew_subs 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb9781d4730 con 0x7fb97809e230 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb968002030 con 0x7fb97809e230 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb968004fc0 con 0x7fb97809e230 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb9757fa640 4 set_mon_vals no callback set 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.391+0000 7fb96ffff640 10 monclient: dump: 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.395+0000 7fb97d98f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.395+0000 7fb97d98f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.395+0000 7fb97d98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.395+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95c005090 con 0x7fb97809e230 2024-09-16T05:49:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.395+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb96800fd60 con 0x7fb97809e230 2024-09-16T05:49:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.395+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fb968045340 con 0x7fb97809e230 2024-09-16T05:49:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.395+0000 7fb976ffd640 10 monclient: get_auth_request con 0x7fb958036c20 auth_method 0 2024-09-16T05:49:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.399+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fb968015970 con 0x7fb97809e230 2024-09-16T05:49:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.399+0000 7fb96ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.399+0000 7fb96ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.623+0000 7fb97d98f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:49:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.623+0000 7fb97d98f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:49:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.623+0000 7fb97d98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.623+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb95c005380 con 0x7fb97809e230 2024-09-16T05:49:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.623+0000 7fb96ffff640 1 -- 172.21.15.6:0/2707123810 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3212286246) 0x7fb968015270 con 0x7fb97809e230 2024-09-16T05:49:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.623+0000 7fb96ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:49:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.623+0000 7fb96ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:51.628 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:49:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.627+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb958036c20 legacy=0x7fb9580390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.627+0000 7fb97d98f640 10 monclient: shutdown 2024-09-16T05:49:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.627+0000 7fb97d98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.627+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 >> v1:172.21.15.6:6790/0 conn(0x7fb97809e230 legacy=0x7fb97806f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.627+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 shutdown_connections 2024-09-16T05:49:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.627+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 >> 172.21.15.6:0/2707123810 conn(0x7fb9780a1650 msgr2=0x7fb97813de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.627+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 shutdown_connections 2024-09-16T05:49:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.627+0000 7fb97d98f640 1 -- 172.21.15.6:0/2707123810 wait complete. 2024-09-16T05:49:51.646 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:49:51.647 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:49:51.647 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:49:51.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:49:51.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient: get_monmap_and_config 2024-09-16T05:49:51.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient: build_initial_monmap 2024-09-16T05:49:51.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient: monmap: 2024-09-16T05:49:51.766 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:51.766 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.766 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:51.767259+0000 2024-09-16T05:49:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:51.767259+0000 2024-09-16T05:49:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:51.768 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 Processor -- start 2024-09-16T05:49:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- start start 2024-09-16T05:49:51.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient: init 2024-09-16T05:49:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:49:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient(hunting): picked mon.noname-b con 0x7fc05c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient(hunting): picked mon.noname-c con 0x7fc05c09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:51.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient(hunting): picked mon.noname-a con 0x7fc05c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc05c0a1370 con 0x7fc05c0a0da0 2024-09-16T05:49:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc05c09ead0 con 0x7fc05c09e230 2024-09-16T05:49:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc05c09edb0 con 0x7fc05c09aa30 2024-09-16T05:49:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient(hunting): authenticate will time out at 1853.573948s 2024-09-16T05:49:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06272c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc05c0a0da0 0x7fc05c13d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45636/0 (socket says 172.21.15.6:45636) 2024-09-16T05:49:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc061f2b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc05c09aa30 0x7fc05c09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46502/0 (socket says 172.21.15.6:46502) 2024-09-16T05:49:51.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06272c640 1 -- 172.21.15.6:0/2945114979 learned_addr learned my addr 172.21.15.6:0/2945114979 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395223830 0 0) 0x7fc05c09edb0 con 0x7fc05c09aa30 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient(hunting): my global_id is 164132 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc044003530 con 0x7fc05c09aa30 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3891191905 0 0) 0x7fc05c0a1370 con 0x7fc05c0a0da0 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient(hunting): my global_id is 144396 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc05c09edb0 con 0x7fc05c0a0da0 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2140255597 0 0) 0x7fc05c09edb0 con 0x7fc05c0a0da0 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 >> v1:172.21.15.6:6790/0 conn(0x7fc05c09aa30 legacy=0x7fc05c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 >> v1:172.21.15.63:6789/0 conn(0x7fc05c09e230 legacy=0x7fc05c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: _finish_hunting 0 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 monclient: found mon.noname-a 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05c09dd50 con 0x7fc05c0a0da0 2024-09-16T05:49:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: _finish_auth 0 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: _check_auth_tickets 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc04c001620 con 0x7fc05c0a0da0 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: dump: 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.789 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 5 monclient: authenticate success, global_id 144396 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc04c001810 con 0x7fc05c0a0da0 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 1 -- 172.21.15.6:0/2945114979 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc04c001c20 con 0x7fc05c0a0da0 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 4 set_mon_vals no callback set 2024-09-16T05:49:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc060f29640 10 monclient: dump: 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 10 monclient: shutdown 2024-09-16T05:49:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- 172.21.15.6:0/2945114979 >> v1:172.21.15.6:6789/0 conn(0x7fc05c0a0da0 legacy=0x7fc05c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- 172.21.15.6:0/2945114979 shutdown_connections 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- 172.21.15.6:0/2945114979 >> 172.21.15.6:0/2945114979 conn(0x7fc05c0a1650 msgr2=0x7fc05c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- 172.21.15.6:0/2945114979 shutdown_connections 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.763+0000 7fc06418f640 1 -- 172.21.15.6:0/2945114979 wait complete. 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: build_initial_monmap 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: monmap: 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:51.792 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 1 Processor -- start 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 1 -- start start 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: init 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient(hunting): picked mon.noname-c con 0x7fc05c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient(hunting): picked mon.noname-b con 0x7fc05c09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient(hunting): picked mon.noname-a con 0x7fc05c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc05c132710 con 0x7fc05c0a0da0 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc05c1340f0 con 0x7fc05c09e230 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc05c133e70 con 0x7fc05c09aa30 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient(hunting): authenticate will time out at 1853.577045s 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06272c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc05c0a0da0 0x7fc05c1d1700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45650/0 (socket says 172.21.15.6:45650) 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06272c640 1 -- 172.21.15.6:0/3310237933 learned_addr learned my addr 172.21.15.6:0/3310237933 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 14587374 0 0) 0x7fc05c132710 con 0x7fc05c0a0da0 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): my global_id is 144399 2024-09-16T05:49:51.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc048003530 con 0x7fc05c0a0da0 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2165289780 0 0) 0x7fc05c133e70 con 0x7fc05c09aa30 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): my global_id is 164135 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc05c132710 con 0x7fc05c09aa30 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2357674668 0 0) 0x7fc048003530 con 0x7fc05c0a0da0 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc05c133e70 con 0x7fc05c0a0da0 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc04c001c20 con 0x7fc05c0a0da0 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): dump: 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.795 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.796 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.796 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.796 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.796 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.796 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.796 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.800 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.800 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:51.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2136440274 0 0) 0x7fc05c132710 con 0x7fc05c09aa30 2024-09-16T05:49:51.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc048003530 con 0x7fc05c09aa30 2024-09-16T05:49:51.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc0540016e0 con 0x7fc05c09aa30 2024-09-16T05:49:51.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:51.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:51.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): dump: 2024-09-16T05:49:51.801 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.801 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.802 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.802 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.802 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.802 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.802 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.803 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.803 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.803 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4271165023 0 0) 0x7fc05c1340f0 con 0x7fc05c09e230 2024-09-16T05:49:51.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): my global_id is 154282 2024-09-16T05:49:51.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc05c132710 con 0x7fc05c09e230 2024-09-16T05:49:51.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3627623033 0 0) 0x7fc05c133e70 con 0x7fc05c0a0da0 2024-09-16T05:49:51.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 >> v1:172.21.15.6:6790/0 conn(0x7fc05c09aa30 legacy=0x7fc05c1318f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 >> v1:172.21.15.63:6789/0 conn(0x7fc05c09e230 legacy=0x7fc05c132000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:49:51.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 monclient: found mon.a 2024-09-16T05:49:51.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:51.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:51.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05c134fe0 con 0x7fc05c0a0da0 2024-09-16T05:49:51.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: _finish_auth 0 2024-09-16T05:49:51.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:49:51.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:51.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 5 monclient: authenticate success, global_id 144399 2024-09-16T05:49:51.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: _renew_subs 2024-09-16T05:49:51.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc05c135cc0 con 0x7fc05c0a0da0 2024-09-16T05:49:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: _renew_subs 2024-09-16T05:49:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:49:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc05c1352c0 con 0x7fc05c0a0da0 2024-09-16T05:49:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc04c004c80 con 0x7fc05c0a0da0 2024-09-16T05:49:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc04c005050 con 0x7fc05c0a0da0 2024-09-16T05:49:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:51.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05bfff640 4 set_mon_vals no callback set 2024-09-16T05:49:51.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: dump: 2024-09-16T05:49:51.809 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.809 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.809 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.809 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.809 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.810 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.810 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.810 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.810 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.810 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:51.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc03c0065d0 con 0x7fc03c002870 2024-09-16T05:49:51.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc04c005270 con 0x7fc05c0a0da0 2024-09-16T05:49:51.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fc04c044e20 con 0x7fc05c0a0da0 2024-09-16T05:49:51.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859058823 0 0) 0x7fc03c0065d0 con 0x7fc03c002870 2024-09-16T05:49:51.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient(hunting): my global_id is 144402 2024-09-16T05:49:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc05c133e70 con 0x7fc03c002870 2024-09-16T05:49:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 749997943 0 0) 0x7fc05c133e70 con 0x7fc03c002870 2024-09-16T05:49:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc03c0065d0 con 0x7fc03c002870 2024-09-16T05:49:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc054004d10 con 0x7fc03c002870 2024-09-16T05:49:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:51.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 10 monclient: dump: 2024-09-16T05:49:51.814 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.814 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.814 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.814 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.815 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.815 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.815 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.815 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.815 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.815 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc06172a640 10 monclient: get_auth_request con 0x7fc048036b40 auth_method 0 2024-09-16T05:49:51.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2307569948 0 0) 0x7fc03c0065d0 con 0x7fc03c002870 2024-09-16T05:49:51.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.767+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc03c00a160 con 0x7fc03c002870 2024-09-16T05:49:51.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.771+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc054004ef0 con 0x7fc03c002870 2024-09-16T05:49:51.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.771+0000 7fc05a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:51.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.771+0000 7fc05a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:51.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.771+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 >> v1:172.21.15.6:6789/0 conn(0x7fc03c002870 legacy=0x7fc03c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.771+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 >> v1:172.21.15.6:6789/0 conn(0x7fc03c002870 legacy=0x7fc03c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:51.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:49:51.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc03c009e50 con 0x7fc03c009110 2024-09-16T05:49:51.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3222238440 0 0) 0x7fc03c009e50 con 0x7fc03c009110 2024-09-16T05:49:51.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient(hunting): my global_id is 144405 2024-09-16T05:49:51.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc03c0065d0 con 0x7fc03c009110 2024-09-16T05:49:51.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1122286615 0 0) 0x7fc03c0065d0 con 0x7fc03c009110 2024-09-16T05:49:51.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc03c009e50 con 0x7fc03c009110 2024-09-16T05:49:51.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc0540054c0 con 0x7fc03c009110 2024-09-16T05:49:51.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:51.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient: dump: 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.820 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2439352189 0 0) 0x7fc03c009e50 con 0x7fc03c009110 2024-09-16T05:49:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc03c00c3e0 con 0x7fc03c009110 2024-09-16T05:49:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 3296290474) 0x7fc0540056f0 con 0x7fc03c009110 2024-09-16T05:49:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:49:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 >> v1:172.21.15.6:6789/0 conn(0x7fc03c009110 legacy=0x7fc03c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc05a7fc640 1 -- 172.21.15.6:0/3310237933 >> v1:172.21.15.6:6789/0 conn(0x7fc03c009110 legacy=0x7fc03c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 64, 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 240389, 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.822 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:51.823 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:51.824 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:51.824 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:51.824 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.824 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:49:51.824 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:49:51.824 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:51.824 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:51.825 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:49:51.826 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:51.827 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:51.828 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:51.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.829 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:51.829 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:49:51.829 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:51.829 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.829 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:51.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.832 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.833 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:49:51.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:51.835 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:51.836 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.836 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.836 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:49:51.836 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:49:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc048036b40 legacy=0x7fc048039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 10 monclient: shutdown 2024-09-16T05:49:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 >> v1:172.21.15.6:6789/0 conn(0x7fc05c0a0da0 legacy=0x7fc05c1d1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06272c640 1 -- 172.21.15.6:0/3310237933 reap_dead start 2024-09-16T05:49:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06272c640 1 -- 172.21.15.6:0/3310237933 reap_dead start 2024-09-16T05:49:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 shutdown_connections 2024-09-16T05:49:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 >> 172.21.15.6:0/3310237933 conn(0x7fc05c0a1650 msgr2=0x7fc05c13de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 shutdown_connections 2024-09-16T05:49:51.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.775+0000 7fc06418f640 1 -- 172.21.15.6:0/3310237933 wait complete. 2024-09-16T05:49:51.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:49:51.840 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:49:51.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:49:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient: get_monmap_and_config 2024-09-16T05:49:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient: build_initial_monmap 2024-09-16T05:49:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient: monmap: 2024-09-16T05:49:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:51.959884+0000 2024-09-16T05:49:51.958 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:51.959884+0000 2024-09-16T05:49:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:51.959 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:51.959 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 1 Processor -- start 2024-09-16T05:49:51.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 1 -- start start 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient: init 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient(hunting): picked mon.noname-b con 0x7f2edc0a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient(hunting): picked mon.noname-c con 0x7f2edc0a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient(hunting): picked mon.noname-a con 0x7f2edc0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc06e520 con 0x7f2edc0a1a00 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc0a0780 con 0x7f2edc0a4290 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc0a4c70 con 0x7f2edc0a37a0 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 10 monclient(hunting): authenticate will time out at 1853.766951s 2024-09-16T05:49:51.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee2114640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2edc0a37a0 0x7f2edc0a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46520/0 (socket says 172.21.15.6:46520) 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee2114640 1 -- 172.21.15.6:0/2287331367 learned_addr learned my addr 172.21.15.6:0/2287331367 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee2915640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2edc0a1a00 0x7f2edc13f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45688/0 (socket says 172.21.15.6:45688) 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1745642991 0 0) 0x7f2edc06e520 con 0x7f2edc0a1a00 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient(hunting): my global_id is 144408 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ec8003530 con 0x7f2edc0a1a00 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2935444931 0 0) 0x7f2edc0a4c70 con 0x7f2edc0a37a0 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient(hunting): my global_id is 164138 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2edc06e520 con 0x7f2edc0a37a0 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1093528430 0 0) 0x7f2ec8003530 con 0x7f2edc0a1a00 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 >> v1:172.21.15.6:6790/0 conn(0x7f2edc0a37a0 legacy=0x7f2edc0a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 >> v1:172.21.15.63:6789/0 conn(0x7f2edc0a4290 legacy=0x7f2edc13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient: _finish_hunting 0 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 monclient: found mon.noname-a 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc0a65f0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient: _finish_auth 0 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient: _check_auth_tickets 2024-09-16T05:49:51.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 5 monclient: authenticate success, global_id 144408 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ed40019a0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee1112640 10 monclient: dump: 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.966 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.955+0000 7f2ee4378640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ed4001cc0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee1112640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee1112640 1 -- 172.21.15.6:0/2287331367 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ed40020d0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 4 set_mon_vals no callback set 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee1112640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee1112640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee1112640 10 monclient: dump: 2024-09-16T05:49:51.967 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: shutdown 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- 172.21.15.6:0/2287331367 >> v1:172.21.15.6:6789/0 conn(0x7f2edc0a1a00 legacy=0x7f2edc13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- 172.21.15.6:0/2287331367 shutdown_connections 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- 172.21.15.6:0/2287331367 >> 172.21.15.6:0/2287331367 conn(0x7f2edc0a58a0 msgr2=0x7f2edc0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- 172.21.15.6:0/2287331367 shutdown_connections 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- 172.21.15.6:0/2287331367 wait complete. 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: build_initial_monmap 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: monmap: 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:51.969 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:51.970 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:51.970 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 Processor -- start 2024-09-16T05:49:51.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- start start 2024-09-16T05:49:51.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: init 2024-09-16T05:49:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:49:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient(hunting): picked mon.noname-c con 0x7f2edc0a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient(hunting): picked mon.noname-a con 0x7f2edc0a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient(hunting): picked mon.noname-b con 0x7f2edc0a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc0a0100 con 0x7f2edc0a37a0 2024-09-16T05:49:51.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc13cca0 con 0x7f2edc0a4290 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc13e6f0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient(hunting): authenticate will time out at 1853.770299s 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee2114640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2edc0a1a00 0x7f2edc136c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46528/0 (socket says 172.21.15.6:46528) 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee2114640 1 -- 172.21.15.6:0/969120734 learned_addr learned my addr 172.21.15.6:0/969120734 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4098591299 0 0) 0x7f2edc13e6f0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): my global_id is 164141 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ec0003560 con 0x7f2edc0a1a00 2024-09-16T05:49:51.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2533423562 0 0) 0x7f2edc0a0100 con 0x7f2edc0a37a0 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): my global_id is 144411 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2edc13e6f0 con 0x7f2edc0a37a0 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2322904324 0 0) 0x7f2ec0003560 con 0x7f2edc0a1a00 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2edc0a0100 con 0x7f2edc0a1a00 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ed80018c0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): dump: 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.977 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2329149899 0 0) 0x7f2edc13e6f0 con 0x7f2edc0a37a0 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ec0003560 con 0x7f2edc0a37a0 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ecc0017e0 con 0x7f2edc0a37a0 2024-09-16T05:49:51.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): dump: 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 503237353 0 0) 0x7f2edc13cca0 con 0x7f2edc0a4290 2024-09-16T05:49:51.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient(hunting): my global_id is 154288 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2edc13e6f0 con 0x7f2edc0a4290 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2518461887 0 0) 0x7f2edc0a0100 con 0x7f2edc0a1a00 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 >> v1:172.21.15.63:6789/0 conn(0x7f2edc0a4290 legacy=0x7f2edc1d3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 >> v1:172.21.15.6:6789/0 conn(0x7f2edc0a37a0 legacy=0x7f2edc1d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 monclient: found mon.c 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc13ef90 con 0x7f2edc0a1a00 2024-09-16T05:49:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 5 monclient: authenticate success, global_id 164141 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: _renew_subs 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2edc13da20 con 0x7f2edc0a1a00 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: _renew_subs 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2edc137530 con 0x7f2edc0a1a00 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ed80047f0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ed8004b40 con 0x7f2edc0a1a00 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed3fff640 4 set_mon_vals no callback set 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ed2ffd640 10 monclient: dump: 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.959+0000 7f2ee4378640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ec40065d0 con 0x7f2ec4002870 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2ed800f8e0 con 0x7f2edc0a1a00 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f2ed8044e70 con 0x7f2edc0a1a00 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814598021 0 0) 0x7f2ec40065d0 con 0x7f2ec4002870 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient(hunting): my global_id is 164144 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2edc0a0100 con 0x7f2ec4002870 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ee2915640 10 monclient: get_auth_request con 0x7f2ec0036c70 auth_method 0 2024-09-16T05:49:51.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3617460799 0 0) 0x7f2edc0a0100 con 0x7f2ec4002870 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ec40065d0 con 0x7f2ec4002870 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ecc0037c0 con 0x7f2ec4002870 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient: dump: 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.984 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2598872157 0 0) 0x7f2ec40065d0 con 0x7f2ec4002870 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ec400a160 con 0x7f2ec4002870 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2ecc003750 con 0x7f2ec4002870 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 >> v1:172.21.15.6:6790/0 conn(0x7f2ec4002870 legacy=0x7f2ec4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.963+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 >> v1:172.21.15.6:6790/0 conn(0x7f2ec4002870 legacy=0x7f2ec4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.967+0000 7f2ee4378640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.967+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ec4009e50 con 0x7f2ec4009110 2024-09-16T05:49:51.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.967+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3467779219 0 0) 0x7f2ec4009e50 con 0x7f2ec4009110 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient(hunting): my global_id is 164147 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ec40065d0 con 0x7f2ec4009110 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3310898111 0 0) 0x7f2ec40065d0 con 0x7f2ec4009110 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ec4009e50 con 0x7f2ec4009110 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ecc003d90 con 0x7f2ec4009110 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient: dump: 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 391566460 0 0) 0x7f2ec4009e50 con 0x7f2ec4009110 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ec400c3e0 con 0x7f2ec4009110 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3960 (unknown 0 0 2707996346) 0x7f2ecc004f80 con 0x7f2ec4009110 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 >> v1:172.21.15.6:6790/0 conn(0x7f2ec4009110 legacy=0x7f2ec40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:51.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ed2ffd640 1 -- 172.21.15.6:0/969120734 >> v1:172.21.15.6:6790/0 conn(0x7f2ec4009110 legacy=0x7f2ec40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:51.988 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:49:51.988 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:49:51.988 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:49:51.988 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:49:51.988 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 64, 2024-09-16T05:49:51.988 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 2931, 2024-09-16T05:49:51.988 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:51.989 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:49:51.990 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:51.991 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:49:51.992 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:51.993 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:51.994 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:51.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:49:51.996 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:51.997 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:51.998 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:51.999 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:49:52.000 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:49:52.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2ec0036c70 legacy=0x7f2ec0039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee4378640 10 monclient: shutdown 2024-09-16T05:49:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee4378640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 >> v1:172.21.15.6:6790/0 conn(0x7f2edc0a1a00 legacy=0x7f2edc136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee2915640 1 -- 172.21.15.6:0/969120734 reap_dead start 2024-09-16T05:49:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee2915640 1 -- 172.21.15.6:0/969120734 reap_dead start 2024-09-16T05:49:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 shutdown_connections 2024-09-16T05:49:52.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 >> 172.21.15.6:0/969120734 conn(0x7f2edc0a58a0 msgr2=0x7f2edc13ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:52.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 shutdown_connections 2024-09-16T05:49:52.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:51.971+0000 7f2ee4378640 1 -- 172.21.15.6:0/969120734 wait complete. 2024-09-16T05:49:52.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:49:52.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:49:52.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:49:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient: get_monmap_and_config 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient: build_initial_monmap 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient: monmap: 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:52.123503+0000 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:52.123503+0000 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:52.122 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 1 Processor -- start 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 1 -- start start 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient: init 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient(hunting): picked mon.noname-c con 0x7f054009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient(hunting): picked mon.noname-b con 0x7f054009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient(hunting): picked mon.noname-a con 0x7f05400a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05400a1370 con 0x7f05400a0da0 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054009ead0 con 0x7f054009aa30 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054009edb0 con 0x7f054009e230 2024-09-16T05:49:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient(hunting): authenticate will time out at 1853.930580s 2024-09-16T05:49:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0544ebd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f05400a0da0 0x7f054013d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45714/0 (socket says 172.21.15.6:45714) 2024-09-16T05:49:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0544ebd640 1 -- 172.21.15.6:0/3347506913 learned_addr learned my addr 172.21.15.6:0/3347506913 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4065744763 0 0) 0x7f05400a1370 con 0x7f05400a0da0 2024-09-16T05:49:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient(hunting): my global_id is 144414 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0528003530 con 0x7f05400a0da0 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3753974117 0 0) 0x7f054009edb0 con 0x7f054009e230 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient(hunting): my global_id is 164150 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05400a1370 con 0x7f054009e230 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 102154084 0 0) 0x7f0528003530 con 0x7f05400a0da0 2024-09-16T05:49:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 >> v1:172.21.15.6:6790/0 conn(0x7f054009e230 legacy=0x7f05401399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 >> v1:172.21.15.63:6789/0 conn(0x7f054009aa30 legacy=0x7f054009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: _finish_hunting 0 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 monclient: found mon.noname-a 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054009dd50 con 0x7f05400a0da0 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: _finish_auth 0 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: _check_auth_tickets 2024-09-16T05:49:52.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:52.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0538001620 con 0x7f05400a0da0 2024-09-16T05:49:52.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 5 monclient: authenticate success, global_id 144414 2024-09-16T05:49:52.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:52.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:49:52.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: dump: 2024-09-16T05:49:52.130 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.130 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f05380018d0 con 0x7f05400a0da0 2024-09-16T05:49:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 1 -- 172.21.15.6:0/3347506913 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0538001ce0 con 0x7f05400a0da0 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f053effd640 10 monclient: dump: 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.132 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.119+0000 7f0546920640 4 set_mon_vals no callback set 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: shutdown 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- 172.21.15.6:0/3347506913 >> v1:172.21.15.6:6789/0 conn(0x7f05400a0da0 legacy=0x7f054013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- 172.21.15.6:0/3347506913 shutdown_connections 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- 172.21.15.6:0/3347506913 >> 172.21.15.6:0/3347506913 conn(0x7f05400a1650 msgr2=0x7f05400a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- 172.21.15.6:0/3347506913 shutdown_connections 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- 172.21.15.6:0/3347506913 wait complete. 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: build_initial_monmap 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: monmap: 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:52.133 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:52.134 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:52.134 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:52.134 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:52.134 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.134 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:52.134 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:52.134 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:52.134 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 Processor -- start 2024-09-16T05:49:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- start start 2024-09-16T05:49:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: init 2024-09-16T05:49:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:52.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient(hunting): picked mon.noname-c con 0x7f054009aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient(hunting): picked mon.noname-b con 0x7f054009e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient(hunting): picked mon.noname-a con 0x7f05400a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05401379a0 con 0x7f05400a0da0 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0540136730 con 0x7f054009e230 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0540134d50 con 0x7f054009aa30 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient(hunting): authenticate will time out at 1853.933915s 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f054009aa30 0x7f054013d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46548/0 (socket says 172.21.15.6:46548) 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053ffff640 1 -- 172.21.15.6:0/2605727088 learned_addr learned my addr 172.21.15.6:0/2605727088 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1262407164 0 0) 0x7f0540134d50 con 0x7f054009aa30 2024-09-16T05:49:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): my global_id is 164153 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f052c003530 con 0x7f054009aa30 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513870904 0 0) 0x7f05401379a0 con 0x7f05400a0da0 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): my global_id is 144417 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0540134d50 con 0x7f05400a0da0 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3780349022 0 0) 0x7f052c003530 con 0x7f054009aa30 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05401379a0 con 0x7f054009aa30 2024-09-16T05:49:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0530001ce0 con 0x7f054009aa30 2024-09-16T05:49:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:49:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): dump: 2024-09-16T05:49:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4258900645 0 0) 0x7f0540134d50 con 0x7f05400a0da0 2024-09-16T05:49:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f052c003530 con 0x7f05400a0da0 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0538001c20 con 0x7f05400a0da0 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient(hunting): dump: 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3955200574 0 0) 0x7f05401379a0 con 0x7f054009aa30 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 >> v1:172.21.15.63:6789/0 conn(0x7f054009e230 legacy=0x7f05401cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 >> v1:172.21.15.6:6789/0 conn(0x7f05400a0da0 legacy=0x7f05401d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 monclient: found mon.c 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054006e900 con 0x7f054009aa30 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient: _finish_auth 0 2024-09-16T05:49:52.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 5 monclient: authenticate success, global_id 164153 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: _renew_subs 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f054006e540 con 0x7f054009aa30 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: _renew_subs 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0540136270 con 0x7f054009aa30 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f05300039a0 con 0x7f054009aa30 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0530003d70 con 0x7f054009aa30 2024-09-16T05:49:52.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 10 monclient: dump: 2024-09-16T05:49:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.148 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053e7fc640 4 set_mon_vals no callback set 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0530003f10 con 0x7f054009aa30 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f05200065d0 con 0x7f0520002870 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.123+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f0530045140 con 0x7f054009aa30 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f0544ebd640 10 monclient: get_auth_request con 0x7f052c036a10 auth_method 0 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174156826 0 0) 0x7f05200065d0 con 0x7f0520002870 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient(hunting): my global_id is 154297 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05401379a0 con 0x7f0520002870 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2812732940 0 0) 0x7f05401379a0 con 0x7f0520002870 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05200065d0 con 0x7f0520002870 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0534001040 con 0x7f0520002870 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient: dump: 2024-09-16T05:49:52.150 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 336084836 0 0) 0x7f05200065d0 con 0x7f0520002870 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f052000a160 con 0x7f0520002870 2024-09-16T05:49:52.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f05340012a0 con 0x7f0520002870 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 >> v1:172.21.15.63:6789/0 conn(0x7f0520002870 legacy=0x7f0520004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.127+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 >> v1:172.21.15.63:6789/0 conn(0x7f0520002870 legacy=0x7f0520004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0520009e50 con 0x7f0520009110 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972612806 0 0) 0x7f0520009e50 con 0x7f0520009110 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient(hunting): my global_id is 154300 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05200065d0 con 0x7f0520009110 2024-09-16T05:49:52.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2973224306 0 0) 0x7f05200065d0 con 0x7f0520009110 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0520009e50 con 0x7f0520009110 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0534003600 con 0x7f0520009110 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient: dump: 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.153 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.154 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.154 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3474992686 0 0) 0x7f0520009e50 con 0x7f0520009110 2024-09-16T05:49:52.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f052000c3e0 con 0x7f0520009110 2024-09-16T05:49:52.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 1823294182) 0x7f0534003860 con 0x7f0520009110 2024-09-16T05:49:52.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 >> v1:172.21.15.63:6789/0 conn(0x7f0520009110 legacy=0x7f05200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f053cff9640 1 -- 172.21.15.6:0/2605727088 >> v1:172.21.15.63:6789/0 conn(0x7f0520009110 legacy=0x7f05200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 64, 2024-09-16T05:49:52.155 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 126878, 2024-09-16T05:49:52.156 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:49:52.156 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:49:52.156 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:49:52.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:49:52.156 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:52.157 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 3, 2024-09-16T05:49:52.157 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:52.157 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:49:52.157 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:49:52.157 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:52.157 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:52.158 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:52.158 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:52.158 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:52.158 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:52.159 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:52.159 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:52.159 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:52.159 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:52.159 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:52.159 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:52.159 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:49:52.159 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:52.160 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:49:52.161 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:49:52.162 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.163 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:49:52.164 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:49:52.164 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:52.164 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:52.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.164 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:52.164 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:49:52.164 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:52.164 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:52.165 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:52.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.165 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:49:52.165 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:49:52.165 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:52.165 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:52.165 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:52.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.166 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:49:52.166 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:49:52.166 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:52.166 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:52.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.166 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:52.166 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:49:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:49:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:49:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:52.167 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:52.168 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:52.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.168 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:49:52.168 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:49:52.168 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:49:52.168 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:49:52.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.169 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:49:52.169 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:49:52.169 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:49:52.169 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:52.169 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:52.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.170 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:49:52.170 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:49:52.170 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:49:52.170 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:49:52.170 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:49:52.170 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:52.170 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:52.171 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:52.172 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:49:52.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.172 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:52.172 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:52.172 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:49:52.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:52.172 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:49:52.172 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:49:52.173 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 >> v1:172.21.15.63:6812/3643207211 conn(0x7f052c036a10 legacy=0x7f052c038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 10 monclient: shutdown 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 >> v1:172.21.15.6:6790/0 conn(0x7f054009aa30 legacy=0x7f054013d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0544ebd640 1 -- 172.21.15.6:0/2605727088 reap_dead start 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0544ebd640 1 -- 172.21.15.6:0/2605727088 reap_dead start 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 shutdown_connections 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 >> 172.21.15.6:0/2605727088 conn(0x7f05400a1650 msgr2=0x7f05400a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 shutdown_connections 2024-09-16T05:49:52.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.135+0000 7f0546920640 1 -- 172.21.15.6:0/2605727088 wait complete. 2024-09-16T05:49:52.176 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:49:52.176 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T05:49:52.176 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:49:52.176 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T05:49:52.176 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-16T05:49:52.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-16T05:49:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient: get_monmap_and_config 2024-09-16T05:49:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient: build_initial_monmap 2024-09-16T05:49:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient: monmap: 2024-09-16T05:49:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:52.304595+0000 2024-09-16T05:49:52.303 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:52.304595+0000 2024-09-16T05:49:52.304 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:52.304 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.304 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:52.304 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:52.304 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:52.304 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 1 Processor -- start 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 1 -- start start 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient: init 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ecc072300 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient(hunting): picked mon.noname-c con 0x7f3ecc072df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ecc13d320 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ecc09b330 con 0x7f3ecc072300 2024-09-16T05:49:52.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ecc09b050 con 0x7f3ecc072df0 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ecc09b610 con 0x7f3ecc13d320 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ed092d640 10 monclient(hunting): authenticate will time out at 1854.111305s 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3eca59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3ecc072300 0x7f3ecc0726e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45742/0 (socket says 172.21.15.6:45742) 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3eca59c640 1 -- 172.21.15.6:0/2733048622 learned_addr learned my addr 172.21.15.6:0/2733048622 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1068883525 0 0) 0x7f3ecc09b610 con 0x7f3ecc13d320 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 10 monclient(hunting): my global_id is 164156 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3eb8003530 con 0x7f3ecc13d320 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467006840 0 0) 0x7f3eb4001120 con 0x7f3ecc072300 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.299+0000 7f3ec959a640 10 monclient(hunting): my global_id is 144420 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ecc09b610 con 0x7f3ecc072300 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3512293006 0 0) 0x7f3eb8003530 con 0x7f3ecc13d320 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 >> v1:172.21.15.63:6789/0 conn(0x7f3ecc072df0 legacy=0x7f3ecc13bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 >> v1:172.21.15.6:6789/0 conn(0x7f3ecc072300 legacy=0x7f3ecc0726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: _finish_hunting 0 2024-09-16T05:49:52.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 monclient: found mon.noname-b 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ecc09c070 con 0x7f3ecc13d320 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: _finish_auth 0 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: _check_auth_tickets 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3ec4001660 con 0x7f3ecc13d320 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: dump: 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.310 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 5 monclient: authenticate success, global_id 164156 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ec40018b0 con 0x7f3ecc13d320 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:52.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 4 set_mon_vals no callback set 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 1 -- 172.21.15.6:0/2733048622 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3ec4001cc0 con 0x7f3ecc13d320 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec959a640 10 monclient: dump: 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.312 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: shutdown 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- 172.21.15.6:0/2733048622 >> v1:172.21.15.6:6790/0 conn(0x7f3ecc13d320 legacy=0x7f3ecc13f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- 172.21.15.6:0/2733048622 shutdown_connections 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- 172.21.15.6:0/2733048622 >> 172.21.15.6:0/2733048622 conn(0x7f3ecc0a1650 msgr2=0x7f3ecc0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- 172.21.15.6:0/2733048622 shutdown_connections 2024-09-16T05:49:52.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- 172.21.15.6:0/2733048622 wait complete. 2024-09-16T05:49:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: build_initial_monmap 2024-09-16T05:49:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: monmap: 2024-09-16T05:49:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:52.314 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.315 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:52.315 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:52.315 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:52.315 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 Processor -- start 2024-09-16T05:49:52.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- start start 2024-09-16T05:49:52.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: init 2024-09-16T05:49:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:49:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient(hunting): picked mon.noname-a con 0x7f3ecc072300 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient(hunting): picked mon.noname-c con 0x7f3ecc072df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient(hunting): picked mon.noname-b con 0x7f3ecc13d320 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ecc137780 con 0x7f3ecc072300 2024-09-16T05:49:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ecc11e9d0 con 0x7f3ecc13d320 2024-09-16T05:49:52.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ecc140a50 con 0x7f3ecc072df0 2024-09-16T05:49:52.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:52.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient(hunting): authenticate will time out at 1854.114620s 2024-09-16T05:49:52.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec9d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3ecc072df0 0x7f3ecc1d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46568/0 (socket says 172.21.15.6:46568) 2024-09-16T05:49:52.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec9d9b640 1 -- 172.21.15.6:0/96525871 learned_addr learned my addr 172.21.15.6:0/96525871 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:52.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360391267 0 0) 0x7f3ecc140a50 con 0x7f3ecc072df0 2024-09-16T05:49:52.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): my global_id is 164159 2024-09-16T05:49:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3eac003530 con 0x7f3ecc072df0 2024-09-16T05:49:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470406092 0 0) 0x7f3ecc137780 con 0x7f3ecc072300 2024-09-16T05:49:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): my global_id is 144423 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ecc140a50 con 0x7f3ecc072300 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 207068138 0 0) 0x7f3eac003530 con 0x7f3ecc072df0 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ecc137780 con 0x7f3ecc072df0 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 164360500 0 0) 0x7f3ecc140a50 con 0x7f3ecc072300 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3eac003530 con 0x7f3ecc072300 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3ebc001ca0 con 0x7f3ecc072df0 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): dump: 2024-09-16T05:49:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333245964 0 0) 0x7f3ecc11e9d0 con 0x7f3ecc13d320 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): my global_id is 154306 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ecc140a50 con 0x7f3ecc13d320 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3eb4001a10 con 0x7f3ecc072300 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient(hunting): dump: 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3664058693 0 0) 0x7f3ecc137780 con 0x7f3ecc072df0 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 >> v1:172.21.15.63:6789/0 conn(0x7f3ecc13d320 legacy=0x7f3ecc1d3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 >> v1:172.21.15.6:6789/0 conn(0x7f3ecc072300 legacy=0x7f3ecc137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 monclient: found mon.c 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ecc0740c0 con 0x7f3ecc072df0 2024-09-16T05:49:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient: _finish_auth 0 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 5 monclient: authenticate success, global_id 164159 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: _renew_subs 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ecc11c6e0 con 0x7f3ecc072df0 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: _renew_subs 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ecc11d460 con 0x7f3ecc072df0 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ebc002080 con 0x7f3ecc072df0 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3ebc003df0 con 0x7f3ecc072df0 2024-09-16T05:49:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec8d99640 4 set_mon_vals no callback set 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.303+0000 7f3ec2ffd640 10 monclient: dump: 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ed092d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3eb00065d0 con 0x7f3eb0002870 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3ebc003f90 con 0x7f3ecc072df0 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f3ebc045070 con 0x7f3ecc072df0 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622886214 0 0) 0x7f3eb00065d0 con 0x7f3eb0002870 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient(hunting): my global_id is 144426 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ecc137780 con 0x7f3eb0002870 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1547849956 0 0) 0x7f3ecc137780 con 0x7f3eb0002870 2024-09-16T05:49:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3eb00065d0 con 0x7f3eb0002870 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3eb4004c40 con 0x7f3eb0002870 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient: dump: 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ecad9d640 10 monclient: get_auth_request con 0x7f3eac036bd0 auth_method 0 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 690441238 0 0) 0x7f3eb00065d0 con 0x7f3eb0002870 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3eb000a160 con 0x7f3eb0002870 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3eb40060c0 con 0x7f3eb0002870 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 >> v1:172.21.15.6:6789/0 conn(0x7f3eb0002870 legacy=0x7f3eb0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.307+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 >> v1:172.21.15.6:6789/0 conn(0x7f3eb0002870 legacy=0x7f3eb0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3eb0009e60 con 0x7f3eb0009120 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3561074408 0 0) 0x7f3eb0009e60 con 0x7f3eb0009120 2024-09-16T05:49:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient(hunting): my global_id is 144429 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3eb00065d0 con 0x7f3eb0009120 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2050492716 0 0) 0x7f3eb00065d0 con 0x7f3eb0009120 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3eb0009e60 con 0x7f3eb0009120 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3eb40066a0 con 0x7f3eb0009120 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient: dump: 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4237760247 0 0) 0x7f3eb0009e60 con 0x7f3eb0009120 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f3eb000c3e0 con 0x7f3eb0009120 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (unknown 0 0 2946029456) 0x7f3eb4006500 con 0x7f3eb0009120 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 >> v1:172.21.15.6:6789/0 conn(0x7f3eb0009120 legacy=0x7f3eb00095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ec2ffd640 1 -- 172.21.15.6:0/96525871 >> v1:172.21.15.6:6789/0 conn(0x7f3eb0009120 legacy=0x7f3eb00095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:49:52.338 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:49:52.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ret": 0, 2024-09-16T05:49:52.338 INFO:teuthology.orchestra.run.smithi006.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T05:49:52.338 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:49:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3eac036bd0 legacy=0x7f3eac039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 10 monclient: shutdown 2024-09-16T05:49:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 >> v1:172.21.15.6:6790/0 conn(0x7f3ecc072df0 legacy=0x7f3ecc1d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ecad9d640 1 -- 172.21.15.6:0/96525871 reap_dead start 2024-09-16T05:49:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ecad9d640 1 -- 172.21.15.6:0/96525871 reap_dead start 2024-09-16T05:49:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 shutdown_connections 2024-09-16T05:49:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 >> 172.21.15.6:0/96525871 conn(0x7f3ecc0a1650 msgr2=0x7f3ecc09d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 shutdown_connections 2024-09-16T05:49:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.315+0000 7f3ed092d640 1 -- 172.21.15.6:0/96525871 wait complete. 2024-09-16T05:49:52.340 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:49:52.341 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T05:49:52.341 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:49:52.355 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T05:49:52.355 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:49:52.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:49:52.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient: get_monmap_and_config 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient: build_initial_monmap 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient: monmap: 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:52.512524+0000 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:52.512524+0000 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:52.512 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:52.512 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 1 Processor -- start 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 1 -- start start 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient: init 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient(hunting): picked mon.noname-b con 0x7f901009aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient(hunting): picked mon.noname-a con 0x7f901009e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient(hunting): picked mon.noname-c con 0x7f90100a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90100a1370 con 0x7f901009e230 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f901009ead0 con 0x7f90100a0da0 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f901009edb0 con 0x7f901009aa30 2024-09-16T05:49:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9017da4640 10 monclient(hunting): authenticate will time out at 1854.319275s 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f901533f640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f901009e230 legacy=0x7f90101399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9015b40640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f901009aa30 0x7f901009ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46606/0 (socket says 172.21.15.6:46606) 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9015b40640 1 -- 172.21.15.6:0/3380621297 learned_addr learned my addr 172.21.15.6:0/3380621297 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900061272 0 0) 0x7f901009edb0 con 0x7f901009aa30 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9014b3e640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9014b3e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9014b3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9014b3e640 10 monclient(hunting): my global_id is 164162 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.507+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ffc003530 con 0x7f901009aa30 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1160230855 0 0) 0x7f8ffc003530 con 0x7f901009aa30 2024-09-16T05:49:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 >> v1:172.21.15.63:6789/0 conn(0x7f90100a0da0 legacy=0x7f901013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 >> v1:172.21.15.6:6789/0 conn(0x7f901009e230 legacy=0x7f90101399d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:49:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: _finish_hunting 0 2024-09-16T05:49:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 1 monclient: found mon.noname-b 2024-09-16T05:49:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f901009dd50 con 0x7f901009aa30 2024-09-16T05:49:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: _finish_auth 0 2024-09-16T05:49:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: _check_auth_tickets 2024-09-16T05:49:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f900c001620 con 0x7f901009aa30 2024-09-16T05:49:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: dump: 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 5 monclient: authenticate success, global_id 164162 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f900c001910 con 0x7f901009aa30 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 1 -- 172.21.15.6:0/3380621297 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f900c001d20 con 0x7f901009aa30 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9014b3e640 10 monclient: dump: 2024-09-16T05:49:52.524 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 4 set_mon_vals no callback set 2024-09-16T05:49:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: shutdown 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- 172.21.15.6:0/3380621297 >> v1:172.21.15.6:6790/0 conn(0x7f901009aa30 legacy=0x7f901009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- 172.21.15.6:0/3380621297 shutdown_connections 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- 172.21.15.6:0/3380621297 >> 172.21.15.6:0/3380621297 conn(0x7f90100a1650 msgr2=0x7f90100a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- 172.21.15.6:0/3380621297 shutdown_connections 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- 172.21.15.6:0/3380621297 wait complete. 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: build_initial_monmap 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: monmap: 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:52.527 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:52.527 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.527 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:52.527 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:52.527 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:52.527 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 Processor -- start 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- start start 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: init 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient(hunting): picked mon.noname-b con 0x7f901009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient(hunting): picked mon.noname-c con 0x7f901009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient(hunting): picked mon.noname-a con 0x7f90100a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f901009c960 con 0x7f90100a0da0 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f901009da60 con 0x7f901009aa30 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f901006ea00 con 0x7f901009e230 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient(hunting): authenticate will time out at 1854.322618s 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9016341640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f90100a0da0 legacy=0x7f90101d3930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f901533f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f901009e230 0x7f901009c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46616/0 (socket says 172.21.15.6:46616) 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f901533f640 1 -- 172.21.15.6:0/2302661141 learned_addr learned my addr 172.21.15.6:0/2302661141 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014948222 0 0) 0x7f901006ea00 con 0x7f901009e230 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): my global_id is 164165 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ff4003560 con 0x7f901009e230 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 150546473 0 0) 0x7f8ff4003560 con 0x7f901009e230 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f901006ea00 con 0x7f901009e230 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9004004160 con 0x7f901009e230 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): dump: 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3544625846 0 0) 0x7f901009da60 con 0x7f901009aa30 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient(hunting): my global_id is 154312 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ff4003560 con 0x7f901009aa30 2024-09-16T05:49:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1710348448 0 0) 0x7f901006ea00 con 0x7f901009e230 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 >> v1:172.21.15.63:6789/0 conn(0x7f901009aa30 legacy=0x7f901009bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 >> v1:172.21.15.6:6789/0 conn(0x7f90100a0da0 legacy=0x7f90101d3930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient: _finish_hunting 0 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 monclient: found mon.c 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9010136b70 con 0x7f901009e230 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient: _finish_auth 0 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient: _check_auth_tickets 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 5 monclient: authenticate success, global_id 164165 2024-09-16T05:49:52.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: _renew_subs 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f901009d360 con 0x7f901009e230 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: _renew_subs 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f90101350c0 con 0x7f901009e230 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f90040043c0 con 0x7f901009e230 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f9004004710 con 0x7f901009e230 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f9003fff640 4 set_mon_vals no callback set 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.511+0000 7f90027fc640 10 monclient: dump: 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:52.535 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.515+0000 7f9017da4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.515+0000 7f9017da4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.515+0000 7f9017da4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.515+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f90040048b0 con 0x7f901009e230 2024-09-16T05:49:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.515+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff8005090 con 0x7f901009e230 2024-09-16T05:49:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.515+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f9004044e30 con 0x7f901009e230 2024-09-16T05:49:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.515+0000 7f9015b40640 10 monclient: get_auth_request con 0x7f8ff4036c00 auth_method 0 2024-09-16T05:49:52.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.519+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f9004019050 con 0x7f901009e230 2024-09-16T05:49:52.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.519+0000 7f90027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:52.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.519+0000 7f90027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.739+0000 7f9017da4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:49:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.739+0000 7f9017da4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:49:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.739+0000 7f9017da4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:52.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.739+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8ff8005380 con 0x7f901009e230 2024-09-16T05:49:52.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.739+0000 7f90027fc640 1 -- 172.21.15.6:0/2302661141 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3167547149) 0x7f900401d910 con 0x7f901009e230 2024-09-16T05:49:52.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.739+0000 7f90027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:49:52.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.739+0000 7f90027fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:52.745 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:49:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.743+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8ff4036c00 legacy=0x7f8ff40390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.743+0000 7f9017da4640 10 monclient: shutdown 2024-09-16T05:49:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.743+0000 7f9017da4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.743+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 >> v1:172.21.15.6:6790/0 conn(0x7f901009e230 legacy=0x7f901009c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.743+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 shutdown_connections 2024-09-16T05:49:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.743+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 >> 172.21.15.6:0/2302661141 conn(0x7f90100a1650 msgr2=0x7f90100a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.743+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 shutdown_connections 2024-09-16T05:49:52.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:52.743+0000 7f9017da4640 1 -- 172.21.15.6:0/2302661141 wait complete. 2024-09-16T05:49:55.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:49:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient: get_monmap_and_config 2024-09-16T05:49:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient: build_initial_monmap 2024-09-16T05:49:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient: monmap: 2024-09-16T05:49:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:55.896559+0000 2024-09-16T05:49:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:55.896559+0000 2024-09-16T05:49:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:55.896 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 1 Processor -- start 2024-09-16T05:49:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 1 -- start start 2024-09-16T05:49:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient: init 2024-09-16T05:49:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:49:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient(hunting): picked mon.noname-b con 0x7f337409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient(hunting): picked mon.noname-a con 0x7f337409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient(hunting): picked mon.noname-c con 0x7f33740a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33740a1370 con 0x7f337409e230 2024-09-16T05:49:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337409ead0 con 0x7f33740a0da0 2024-09-16T05:49:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337409edb0 con 0x7f337409aa30 2024-09-16T05:49:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f33789c8640 10 monclient(hunting): authenticate will time out at 1857.703643s 2024-09-16T05:49:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f3371d9b640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f337409e230 legacy=0x7f33741399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:49:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f337259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f337409aa30 0x7f337409ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46632/0 (socket says 172.21.15.6:46632) 2024-09-16T05:49:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.891+0000 7f337259c640 1 -- 172.21.15.6:0/363901396 learned_addr learned my addr 172.21.15.6:0/363901396 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:55.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 780611020 0 0) 0x7f337409edb0 con 0x7f337409aa30 2024-09-16T05:49:55.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient(hunting): _negotiate 2024-09-16T05:49:55.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:55.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:55.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient(hunting): my global_id is 164168 2024-09-16T05:49:55.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f335c003530 con 0x7f337409aa30 2024-09-16T05:49:55.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2354508486 0 0) 0x7f335c003530 con 0x7f337409aa30 2024-09-16T05:49:55.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 >> v1:172.21.15.63:6789/0 conn(0x7f33740a0da0 legacy=0x7f337413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:55.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 >> v1:172.21.15.6:6789/0 conn(0x7f337409e230 legacy=0x7f33741399d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:49:55.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: _finish_hunting 0 2024-09-16T05:49:55.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 monclient: found mon.noname-b 2024-09-16T05:49:55.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:55.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:49:55.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337409dd50 con 0x7f337409aa30 2024-09-16T05:49:55.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: _finish_auth 0 2024-09-16T05:49:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: _check_auth_tickets 2024-09-16T05:49:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3364001620 con 0x7f337409aa30 2024-09-16T05:49:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:49:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: dump: 2024-09-16T05:49:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:55.917 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 5 monclient: authenticate success, global_id 164168 2024-09-16T05:49:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:49:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3364001910 con 0x7f337409aa30 2024-09-16T05:49:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 1 -- 172.21.15.6:0/363901396 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3364001d20 con 0x7f337409aa30 2024-09-16T05:49:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337159a640 10 monclient: dump: 2024-09-16T05:49:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:55.920 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: get_monmap_and_config success 2024-09-16T05:49:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 4 set_mon_vals no callback set 2024-09-16T05:49:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: shutdown 2024-09-16T05:49:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- 172.21.15.6:0/363901396 >> v1:172.21.15.6:6790/0 conn(0x7f337409aa30 legacy=0x7f337409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- 172.21.15.6:0/363901396 shutdown_connections 2024-09-16T05:49:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- 172.21.15.6:0/363901396 >> 172.21.15.6:0/363901396 conn(0x7f33740a1650 msgr2=0x7f33740a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- 172.21.15.6:0/363901396 shutdown_connections 2024-09-16T05:49:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- 172.21.15.6:0/363901396 wait complete. 2024-09-16T05:49:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: build_initial_monmap 2024-09-16T05:49:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: monmap: 2024-09-16T05:49:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:49:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:49:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:49:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:49:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:49:55.924 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 Processor -- start 2024-09-16T05:49:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- start start 2024-09-16T05:49:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: init 2024-09-16T05:49:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:49:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient(hunting): picked mon.noname-c con 0x7f337409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient(hunting): picked mon.noname-a con 0x7f337409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient(hunting): picked mon.noname-b con 0x7f33740a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337413ccd0 con 0x7f337409e230 2024-09-16T05:49:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337409c990 con 0x7f33740a0da0 2024-09-16T05:49:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337413cfb0 con 0x7f337409aa30 2024-09-16T05:49:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3371d9b640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f337409e230 legacy=0x7f337409c280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:49:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient(hunting): authenticate will time out at 1857.706791s 2024-09-16T05:49:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f337409aa30 0x7f337409bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:46638/0 (socket says 172.21.15.6:46638) 2024-09-16T05:49:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f337259c640 1 -- 172.21.15.6:0/33821051 learned_addr learned my addr 172.21.15.6:0/33821051 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:55.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363441167 0 0) 0x7f337413cfb0 con 0x7f337409aa30 2024-09-16T05:49:55.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:49:55.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:49:55.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:49:55.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient(hunting): my global_id is 164171 2024-09-16T05:49:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3354003530 con 0x7f337409aa30 2024-09-16T05:49:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3702095128 0 0) 0x7f3354003530 con 0x7f337409aa30 2024-09-16T05:49:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f337413cfb0 con 0x7f337409aa30 2024-09-16T05:49:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f33640015e0 con 0x7f337409aa30 2024-09-16T05:49:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:49:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient(hunting): dump: 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1854528701 0 0) 0x7f337413cfb0 con 0x7f337409aa30 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 >> v1:172.21.15.63:6789/0 conn(0x7f33740a0da0 legacy=0x7f33741d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 >> v1:172.21.15.6:6789/0 conn(0x7f337409e230 legacy=0x7f337409c280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 monclient: found mon.c 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337413dac0 con 0x7f337409aa30 2024-09-16T05:49:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient: _finish_auth 0 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 5 monclient: authenticate success, global_id 164171 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: _renew_subs 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f337406ea40 con 0x7f337409aa30 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: _renew_subs 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f337413bb60 con 0x7f337409aa30 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3364001910 con 0x7f337409aa30 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:49:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3364004e70 con 0x7f337409aa30 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3362ffd640 10 monclient: dump: 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:49:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.895+0000 7f3370d99640 4 set_mon_vals no callback set 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.899+0000 7f33789c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.899+0000 7f33789c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.899+0000 7f33789c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.899+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3358005090 con 0x7f337409aa30 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.899+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3364005010 con 0x7f337409aa30 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.899+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f3364044ff0 con 0x7f337409aa30 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.899+0000 7f3371d9b640 10 monclient: get_auth_request con 0x7f3354036980 auth_method 0 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.903+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f3364015680 con 0x7f337409aa30 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.903+0000 7f3362ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:49:55.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:55.903+0000 7f3362ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:49:56.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.123+0000 7f33789c8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:49:56.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.123+0000 7f33789c8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:49:56.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.123+0000 7f33789c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:49:56.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.123+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3358005380 con 0x7f337409aa30 2024-09-16T05:49:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.123+0000 7f3362ffd640 1 -- 172.21.15.6:0/33821051 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2988521377) 0x7f3364014f80 con 0x7f337409aa30 2024-09-16T05:49:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.123+0000 7f3362ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:49:56.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.123+0000 7f3362ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:49:56.130 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:49:56.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.127+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3354036980 legacy=0x7f3354038e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:56.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.127+0000 7f33789c8640 10 monclient: shutdown 2024-09-16T05:49:56.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.127+0000 7f33789c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:49:56.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.127+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 >> v1:172.21.15.6:6790/0 conn(0x7f337409aa30 legacy=0x7f337409bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:49:56.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.127+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 shutdown_connections 2024-09-16T05:49:56.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.127+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 >> 172.21.15.6:0/33821051 conn(0x7f33740a1650 msgr2=0x7f337413e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:49:56.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.127+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 shutdown_connections 2024-09-16T05:49:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:56.127+0000 7f33789c8640 1 -- 172.21.15.6:0/33821051 wait complete. 2024-09-16T05:49:59.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:49:59.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient: get_monmap_and_config 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient: build_initial_monmap 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient: monmap: 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:49:59.280474+0000 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:49:59.280474+0000 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:49:59.279 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 1 Processor -- start 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 1 -- start start 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient: init 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient: _reopen_session rank -1 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae80a27e0 addr v1:172.21.15.6:6790/0 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae8136a10 addr v1:172.21.15.63:6789/0 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae813cf70 addr v1:172.21.15.6:6789/0 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae80a0f80 con 0x7f2ae813cf70 2024-09-16T05:49:59.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae80a4c40 con 0x7f2ae8136a10 2024-09-16T05:49:59.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae80a06e0 con 0x7f2ae80a27e0 2024-09-16T05:49:59.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient(hunting): _renew_subs 2024-09-16T05:49:59.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aef452640 10 monclient(hunting): authenticate will time out at 1861.087697s 2024-09-16T05:49:59.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aed9ef640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2ae813f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:49:59.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aed1ee640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2ae80a27e0 0x7f2ae80a2bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53174/0 (socket says 172.21.15.6:53174) 2024-09-16T05:49:59.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.275+0000 7f2aed1ee640 1 -- 172.21.15.6:0/2666332469 learned_addr learned my addr 172.21.15.6:0/2666332469 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:49:59.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.479+0000 7f2aed9ef640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2ae813f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:49:59.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:49:59.883+0000 7f2aed9ef640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2ae813f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:00.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:00.687+0000 7f2aed9ef640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2ae813f360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:02.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 10 monclient(hunting): tick 2024-09-16T05:50:02.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 1 monclient(hunting): continuing hunt 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6790/0 conn(0x7f2ae80a27e0 legacy=0x7f2ae80a2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.63:6789/0 conn(0x7f2ae8136a10 legacy=0x7f2ae813b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2ae813f360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f2acc0036d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae813cf70 addr v1:172.21.15.6:6789/0 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.275+0000 7f2adeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f2acc00ab90 addr v1:172.21.15.6:6790/0 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae806dff0 con 0x7f2ae813cf70 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae80a0f80 con 0x7f2acc0036d0 2024-09-16T05:50:02.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2acc0108d0 con 0x7f2acc00ab90 2024-09-16T05:50:02.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.279+0000 7f2adeffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:50:02.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.279+0000 7f2aec9ed640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:02.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.483+0000 7f2aec9ed640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:02.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:02.887+0000 7f2aec9ed640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:03.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:03.691+0000 7f2aec9ed640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 10 monclient(hunting): tick 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 1 monclient(hunting): continuing hunt 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6790/0 conn(0x7f2acc00ab90 legacy=0x7f2acc00d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.63:6789/0 conn(0x7f2acc0036d0 legacy=0x7f2acc005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:50:05.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f2acc00f9a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:50:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae813cf70 addr v1:172.21.15.6:6789/0 2024-09-16T05:50:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f2acc0036d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:50:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2acc011b00 con 0x7f2ae813cf70 2024-09-16T05:50:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae806dff0 con 0x7f2acc0036d0 2024-09-16T05:50:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 1 -- 172.21.15.6:0/2666332469 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2acc00af90 con 0x7f2acc00f9a0 2024-09-16T05:50:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2adeffd640 10 monclient(hunting): _renew_subs 2024-09-16T05:50:05.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.279+0000 7f2aec9ed640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:05.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.483+0000 7f2aec9ed640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:05.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:05.887+0000 7f2aec9ed640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.615+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1924161426 0 0) 0x7f2ae806dff0 con 0x7f2acc0036d0 2024-09-16T05:50:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.615+0000 7f2adffff640 10 monclient(hunting): _negotiate 2024-09-16T05:50:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.615+0000 7f2adffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.615+0000 7f2adffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.615+0000 7f2adffff640 10 monclient(hunting): my global_id is 154321 2024-09-16T05:50:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.615+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ad4003530 con 0x7f2acc0036d0 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 41335089 0 0) 0x7f2ad4003530 con 0x7f2acc0036d0 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6790/0 conn(0x7f2acc00f9a0 legacy=0x7f2acc00fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2acc0093f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: _finish_hunting 0 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 1 monclient: found mon.noname-c 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2acc00c170 con 0x7f2acc0036d0 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: _finish_auth 0 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: _check_auth_tickets 2024-09-16T05:50:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ae00044e0 con 0x7f2acc0036d0 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: dump: 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2aef452640 5 monclient: authenticate success, global_id 154321 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2aef452640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ae0004740 con 0x7f2acc0036d0 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 1 -- 172.21.15.6:0/2666332469 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ae0004b90 con 0x7f2acc0036d0 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2adffff640 10 monclient: dump: 2024-09-16T05:50:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2aef452640 10 monclient: get_monmap_and_config success 2024-09-16T05:50:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2aef452640 4 set_mon_vals no callback set 2024-09-16T05:50:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2aef452640 10 monclient: shutdown 2024-09-16T05:50:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2aef452640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.623+0000 7f2aef452640 1 -- 172.21.15.6:0/2666332469 >> v1:172.21.15.63:6789/0 conn(0x7f2acc0036d0 legacy=0x7f2acc005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 -- 172.21.15.6:0/2666332469 shutdown_connections 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 -- 172.21.15.6:0/2666332469 >> 172.21.15.6:0/2666332469 conn(0x7f2ae80a9880 msgr2=0x7f2ae80a5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 -- 172.21.15.6:0/2666332469 shutdown_connections 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 -- 172.21.15.6:0/2666332469 wait complete. 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient: build_initial_monmap 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient: monmap: 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:50:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 Processor -- start 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 -- start start 2024-09-16T05:50:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient: init 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient: _reopen_session rank -1 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient(hunting): picked mon.noname-c con 0x7f2acc0036d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient(hunting): picked mon.noname-b con 0x7f2acc00f9a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae813cf70 addr v1:172.21.15.6:6789/0 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae813cc60 con 0x7f2ae813cf70 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae80a7120 con 0x7f2acc00f9a0 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae8121110 con 0x7f2acc0036d0 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient(hunting): _renew_subs 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aef452640 10 monclient(hunting): authenticate will time out at 1868.439790s 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.627+0000 7f2aed9ef640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2ae81d38b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aed1ee640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2acc0036d0 0x7f2ae81cb550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53216/0 (socket says 172.21.15.6:53216) 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aed1ee640 1 -- 172.21.15.6:0/13239028 learned_addr learned my addr 172.21.15.6:0/13239028 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075625531 0 0) 0x7f2ae80a7120 con 0x7f2acc00f9a0 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient(hunting): _negotiate 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient(hunting): my global_id is 154324 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2acc00fd80 con 0x7f2acc00f9a0 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2672374617 0 0) 0x7f2acc00fd80 con 0x7f2acc00f9a0 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ae80a7120 con 0x7f2acc00f9a0 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ad80017b0 con 0x7f2acc00f9a0 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:50:06.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient(hunting): dump: 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:06.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2282092311 0 0) 0x7f2ae80a7120 con 0x7f2acc00f9a0 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 >> v1:172.21.15.6:6790/0 conn(0x7f2acc0036d0 legacy=0x7f2ae81cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 >> v1:172.21.15.6:6789/0 conn(0x7f2ae813cf70 legacy=0x7f2ae81d38b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient: _finish_hunting 0 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 monclient: found mon.b 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae81213f0 con 0x7f2acc00f9a0 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient: _finish_auth 0 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient: _check_auth_tickets 2024-09-16T05:50:06.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 5 monclient: authenticate success, global_id 154324 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 10 monclient: _renew_subs 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae811fcc0 con 0x7f2acc00f9a0 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 10 monclient: _renew_subs 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ae8136610 con 0x7f2acc00f9a0 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ad8003620 con 0x7f2acc00f9a0 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2ad80039f0 con 0x7f2acc00f9a0 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2adeffd640 4 set_mon_vals no callback set 2024-09-16T05:50:06.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 10 monclient: dump: 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:06.643 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:06.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:50:06.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:50:06.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:50:06.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac8005090 con 0x7f2acc00f9a0 2024-09-16T05:50:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.631+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2ad8003c10 con 0x7f2acc00f9a0 2024-09-16T05:50:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.635+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f2ad8044e70 con 0x7f2acc00f9a0 2024-09-16T05:50:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.635+0000 7f2aed1ee640 10 monclient: get_auth_request con 0x7f2acc0381a0 auth_method 0 2024-09-16T05:50:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.635+0000 7f2aed1ee640 1 -- 172.21.15.6:0/13239028 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2acc0381a0 legacy=0x7f2acc03a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T05:50:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.635+0000 7f2aed1ee640 1 -- 172.21.15.6:0/13239028 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2acc0381a0 legacy=0x7f2acc03a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:50:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.635+0000 7f2aed1ee640 1 --1- 172.21.15.6:0/13239028 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2acc0381a0 0x7f2acc03a660 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T05:50:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.635+0000 7f2aed1ee640 1 --1- 172.21.15.6:0/13239028 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2acc0381a0 0x7f2acc03a660 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T05:50:06.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.635+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2acc0381a0 legacy=0x7f2acc03a660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:50:06.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.647+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2ad8019090 con 0x7f2acc00f9a0 2024-09-16T05:50:06.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.647+0000 7f2addffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:50:06.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.647+0000 7f2addffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:50:06.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.875+0000 7f2aef452640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:50:06.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.875+0000 7f2aef452640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:50:06.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.875+0000 7f2aef452640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:50:06.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.875+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ac8005380 con 0x7f2acc00f9a0 2024-09-16T05:50:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2addffb640 1 -- 172.21.15.6:0/13239028 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 2817468898) 0x7f2ad801d950 con 0x7f2acc00f9a0 2024-09-16T05:50:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2addffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:50:06.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2addffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:50:06.882 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":66,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:50:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2aef452640 10 monclient: shutdown 2024-09-16T05:50:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2aef452640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:50:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 >> v1:172.21.15.63:6789/0 conn(0x7f2acc00f9a0 legacy=0x7f2ae8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 shutdown_connections 2024-09-16T05:50:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 >> 172.21.15.6:0/13239028 conn(0x7f2ae80a9880 msgr2=0x7f2ae80a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:50:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 shutdown_connections 2024-09-16T05:50:06.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:06.879+0000 7f2aef452640 1 -- 172.21.15.6:0/13239028 wait complete. 2024-09-16T05:50:06.901 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:50:06.901 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:50:06.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:50:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: get_monmap_and_config 2024-09-16T05:50:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: build_initial_monmap 2024-09-16T05:50:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: monmap: 2024-09-16T05:50:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:50:07.021 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:50:07.023103+0000 2024-09-16T05:50:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:50:07.023103+0000 2024-09-16T05:50:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:50:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:50:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:50:07.022 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:50:07.022 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 Processor -- start 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- start start 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: init 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: _reopen_session rank -1 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient(hunting): picked mon.noname-a con 0x7fa548134810 addr v1:172.21.15.6:6789/0 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient(hunting): picked mon.noname-b con 0x7fa548134bf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient(hunting): picked mon.noname-c con 0x7fa548134ff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5480a1c80 con 0x7fa548134810 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54809e310 con 0x7fa548134ff0 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5480a1f00 con 0x7fa548134bf0 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient(hunting): _renew_subs 2024-09-16T05:50:07.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient(hunting): authenticate will time out at 1868.829727s 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54ce94640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fa548134810 legacy=0x7fa54809e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa547fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa548134bf0 0x7fa5481396e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53232/0 (socket says 172.21.15.6:53232) 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa547fff640 1 -- 172.21.15.6:0/2663727809 learned_addr learned my addr 172.21.15.6:0/2663727809 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3902683261 0 0) 0x7fa5480a1f00 con 0x7fa548134bf0 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient(hunting): my global_id is 164174 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa530003530 con 0x7fa548134bf0 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 690172776 0 0) 0x7fa530003530 con 0x7fa548134bf0 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 >> v1:172.21.15.63:6789/0 conn(0x7fa548134ff0 legacy=0x7fa54813ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 >> v1:172.21.15.6:6789/0 conn(0x7fa548134810 legacy=0x7fa54809e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:50:07.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: _finish_hunting 0 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 monclient: found mon.noname-b 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa54809da70 con 0x7fa548134bf0 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: _finish_auth 0 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: _check_auth_tickets 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa538001620 con 0x7fa548134bf0 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: dump: 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.027 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 5 monclient: authenticate success, global_id 164174 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa538001880 con 0x7fa548134bf0 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:50:07.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 1 -- 172.21.15.6:0/2663727809 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa538001c50 con 0x7fa548134bf0 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: get_monmap_and_config success 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 4 set_mon_vals no callback set 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa5477fe640 10 monclient: dump: 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.029 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: shutdown 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- 172.21.15.6:0/2663727809 >> v1:172.21.15.6:6790/0 conn(0x7fa548134bf0 legacy=0x7fa5481396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- 172.21.15.6:0/2663727809 shutdown_connections 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- 172.21.15.6:0/2663727809 >> 172.21.15.6:0/2663727809 conn(0x7fa5480a2760 msgr2=0x7fa5480a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- 172.21.15.6:0/2663727809 shutdown_connections 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 1 -- 172.21.15.6:0/2663727809 wait complete. 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: build_initial_monmap 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.019+0000 7fa54f0f8640 10 monclient: monmap: 2024-09-16T05:50:07.030 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:50:07.031 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 1 Processor -- start 2024-09-16T05:50:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 1 -- start start 2024-09-16T05:50:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient: init 2024-09-16T05:50:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient: _reopen_session rank -1 2024-09-16T05:50:07.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient(hunting): picked mon.noname-a con 0x7fa548134810 addr v1:172.21.15.6:6789/0 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient(hunting): picked mon.noname-c con 0x7fa548134bf0 addr v1:172.21.15.6:6790/0 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient(hunting): picked mon.noname-b con 0x7fa548134ff0 addr v1:172.21.15.63:6789/0 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54812f950 con 0x7fa548134810 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5480a4110 con 0x7fa548134ff0 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5480741c0 con 0x7fa548134bf0 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient(hunting): _renew_subs 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient(hunting): authenticate will time out at 1868.832795s 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54ce94640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7fa548134810 legacy=0x7fa54812eb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa547fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa548134bf0 0x7fa54812f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53236/0 (socket says 172.21.15.6:53236) 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa547fff640 1 -- 172.21.15.6:0/3968773736 learned_addr learned my addr 172.21.15.6:0/3968773736 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2050194744 0 0) 0x7fa5480741c0 con 0x7fa548134bf0 2024-09-16T05:50:07.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): my global_id is 164177 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa534003530 con 0x7fa548134bf0 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2399050310 0 0) 0x7fa534003530 con 0x7fa548134bf0 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5480741c0 con 0x7fa548134bf0 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa538001520 con 0x7fa548134bf0 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): dump: 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130090429 0 0) 0x7fa5480a4110 con 0x7fa548134ff0 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): my global_id is 154330 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa534003530 con 0x7fa548134ff0 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2475804250 0 0) 0x7fa5480741c0 con 0x7fa548134bf0 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 >> v1:172.21.15.63:6789/0 conn(0x7fa548134ff0 legacy=0x7fa5481d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 >> v1:172.21.15.6:6789/0 conn(0x7fa548134810 legacy=0x7fa54812eb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: _finish_hunting 0 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 monclient: found mon.c 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa548131ab0 con 0x7fa548134bf0 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: _finish_auth 0 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: _check_auth_tickets 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:50:07.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 5 monclient: authenticate success, global_id 164177 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient: _renew_subs 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa548130750 con 0x7fa548134bf0 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient: _renew_subs 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5481331c0 con 0x7fa548134bf0 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa538004c00 con 0x7fa548134bf0 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa538004e30 con 0x7fa548134bf0 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa546ffd640 4 set_mon_vals no callback set 2024-09-16T05:50:07.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: dump: 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5280065d0 con 0x7fa528002870 2024-09-16T05:50:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fa53800fbd0 con 0x7fa548134bf0 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fa538044fd0 con 0x7fa548134bf0 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 193918954 0 0) 0x7fa5280065d0 con 0x7fa528002870 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient(hunting): my global_id is 164180 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5480741c0 con 0x7fa528002870 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1097024085 0 0) 0x7fa5480741c0 con 0x7fa528002870 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa54d695640 10 monclient: get_auth_request con 0x7fa534036c30 auth_method 0 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5280065d0 con 0x7fa528002870 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa53c003eb0 con 0x7fa528002870 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: dump: 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2570199948 0 0) 0x7fa5280065d0 con 0x7fa528002870 2024-09-16T05:50:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa52800a160 con 0x7fa528002870 2024-09-16T05:50:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa53c006320 con 0x7fa528002870 2024-09-16T05:50:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:50:07.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:50:07.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 >> v1:172.21.15.6:6790/0 conn(0x7fa528002870 legacy=0x7fa528004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.023+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 >> v1:172.21.15.6:6790/0 conn(0x7fa528002870 legacy=0x7fa528004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:50:07.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:50:07.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa528009e30 con 0x7fa5280090f0 2024-09-16T05:50:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097182471 0 0) 0x7fa528009e30 con 0x7fa5280090f0 2024-09-16T05:50:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient(hunting): my global_id is 164183 2024-09-16T05:50:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5280065d0 con 0x7fa5280090f0 2024-09-16T05:50:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4034859195 0 0) 0x7fa5280065d0 con 0x7fa5280090f0 2024-09-16T05:50:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa528009e30 con 0x7fa5280090f0 2024-09-16T05:50:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fa53c0045f0 con 0x7fa5280090f0 2024-09-16T05:50:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:50:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient: dump: 2024-09-16T05:50:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.050 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 529998574 0 0) 0x7fa528009e30 con 0x7fa5280090f0 2024-09-16T05:50:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa52800c3a0 con 0x7fa5280090f0 2024-09-16T05:50:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4284 (unknown 0 0 4178052347) 0x7fa53c0059d0 con 0x7fa5280090f0 2024-09-16T05:50:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:50:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:50:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 >> v1:172.21.15.6:6790/0 conn(0x7fa5280090f0 legacy=0x7fa528009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa5457fa640 1 -- 172.21.15.6:0/3968773736 >> v1:172.21.15.6:6790/0 conn(0x7fa5280090f0 legacy=0x7fa528009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:50:07.051 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:50:07.051 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:50:07.052 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:50:07.052 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:50:07.052 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 66, 2024-09-16T05:50:07.052 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 17991, 2024-09-16T05:50:07.052 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:50:07.052 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:50:07.052 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:50:07.053 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.053 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:50:07.053 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:50:07.053 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:50:07.053 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:50:07.053 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:50:07.053 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:50:07.054 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:50:07.054 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:50:07.054 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:50:07.055 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:50:07.055 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:50:07.055 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:50:07.055 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:50:07.055 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:50:07.055 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:50:07.055 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.056 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:50:07.056 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:50:07.056 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:50:07.056 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:50:07.056 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:50:07.056 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:50:07.056 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:50:07.057 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:50:07.057 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:50:07.057 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:50:07.057 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:50:07.057 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:50:07.057 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:50:07.058 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.058 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:50:07.058 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:50:07.058 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:50:07.058 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:50:07.058 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:50:07.059 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:50:07.059 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:50:07.059 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:50:07.059 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:50:07.059 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:50:07.059 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:50:07.060 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:50:07.060 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:50:07.060 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:50:07.060 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:50:07.060 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:50:07.060 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:50:07.060 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:50:07.061 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:50:07.061 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:50:07.061 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:50:07.061 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:50:07.061 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:50:07.061 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:50:07.061 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:50:07.062 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:50:07.062 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:50:07.062 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:50:07.062 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.062 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:50:07.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.063 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:50:07.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.063 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:50:07.063 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:50:07.063 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:50:07.063 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:50:07.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.064 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:50:07.064 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:50:07.064 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:50:07.064 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.064 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.065 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:50:07.065 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:50:07.066 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:50:07.066 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:50:07.066 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:50:07.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.067 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:50:07.067 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:50:07.067 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:50:07.067 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:50:07.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.068 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:50:07.068 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:50:07.069 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:50:07.070 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:50:07.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.072 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:50:07.073 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:50:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 >> v1:172.21.15.63:6812/3643207211 conn(0x7fa534036c30 legacy=0x7fa5340390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 10 monclient: shutdown 2024-09-16T05:50:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:50:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 >> v1:172.21.15.6:6790/0 conn(0x7fa548134bf0 legacy=0x7fa54812f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54d695640 1 -- 172.21.15.6:0/3968773736 reap_dead start 2024-09-16T05:50:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54d695640 1 -- 172.21.15.6:0/3968773736 reap_dead start 2024-09-16T05:50:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 shutdown_connections 2024-09-16T05:50:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 >> 172.21.15.6:0/3968773736 conn(0x7fa5480a2760 msgr2=0x7fa54812d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:50:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 shutdown_connections 2024-09-16T05:50:07.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.031+0000 7fa54f0f8640 1 -- 172.21.15.6:0/3968773736 wait complete. 2024-09-16T05:50:07.076 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:50:07.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:50:07.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:50:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient: get_monmap_and_config 2024-09-16T05:50:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient: build_initial_monmap 2024-09-16T05:50:07.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient: monmap: 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:50:07.195281+0000 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:50:07.195281+0000 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:50:07.194 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 Processor -- start 2024-09-16T05:50:07.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- start start 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient: init 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient: _reopen_session rank -1 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc8072300 addr v1:172.21.15.63:6789/0 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc8072df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc813d320 addr v1:172.21.15.6:6789/0 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fc809b330 con 0x7f7fc813d320 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fc809b050 con 0x7f7fc8072300 2024-09-16T05:50:07.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fc809b610 con 0x7f7fc8072df0 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient(hunting): _renew_subs 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient(hunting): authenticate will time out at 1869.001925s 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcdf30640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f7fc813d320 legacy=0x7f7fc813f710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fccf2e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7fc8072df0 0x7f7fc813bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53262/0 (socket says 172.21.15.6:53262) 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fccf2e640 1 -- 172.21.15.6:0/2006660602 learned_addr learned my addr 172.21.15.6:0/2006660602 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2173590902 0 0) 0x7f7fc809b610 con 0x7f7fc8072df0 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient(hunting): my global_id is 164186 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fb0003530 con 0x7f7fc8072df0 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458887739 0 0) 0x7f7fc809b050 con 0x7f7fc8072300 2024-09-16T05:50:07.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient(hunting): my global_id is 154333 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fc809b610 con 0x7f7fc8072300 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2408294240 0 0) 0x7f7fb0003530 con 0x7f7fc8072df0 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 >> v1:172.21.15.63:6789/0 conn(0x7f7fc8072300 legacy=0x7f7fc80726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 >> v1:172.21.15.6:6789/0 conn(0x7f7fc813d320 legacy=0x7f7fc813f710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 monclient: found mon.noname-b 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc809c070 con 0x7f7fc8072df0 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: _finish_auth 0 2024-09-16T05:50:07.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7fbc001660 con 0x7f7fc8072df0 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 5 monclient: authenticate success, global_id 164186 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: dump: 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.205 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7fbc001930 con 0x7f7fc8072df0 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient: get_monmap_and_config success 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 4 set_mon_vals no callback set 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 1 -- 172.21.15.6:0/2006660602 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7fbc001d00 con 0x7f7fc8072df0 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fc7fff640 10 monclient: dump: 2024-09-16T05:50:07.206 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 10 monclient: shutdown 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- 172.21.15.6:0/2006660602 >> v1:172.21.15.6:6790/0 conn(0x7f7fc8072df0 legacy=0x7f7fc813bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- 172.21.15.6:0/2006660602 shutdown_connections 2024-09-16T05:50:07.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- 172.21.15.6:0/2006660602 >> 172.21.15.6:0/2006660602 conn(0x7f7fc80a1650 msgr2=0x7f7fc80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- 172.21.15.6:0/2006660602 shutdown_connections 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.191+0000 7f7fcf993640 1 -- 172.21.15.6:0/2006660602 wait complete. 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: build_initial_monmap 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: monmap: 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:50:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 1 Processor -- start 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 1 -- start start 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: init 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: _reopen_session rank -1 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc8072300 addr v1:172.21.15.6:6789/0 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc8072df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc813d320 addr v1:172.21.15.6:6790/0 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fc813d0f0 con 0x7f7fc8072300 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fc8139550 con 0x7f7fc8072df0 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fc813f310 con 0x7f7fc813d320 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient(hunting): _renew_subs 2024-09-16T05:50:07.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient(hunting): authenticate will time out at 1869.005127s 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcd72f640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f7fc8072300 legacy=0x7f7fc813e4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcdf30640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7fc813d320 0x7f7fc81d3d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53270/0 (socket says 172.21.15.6:53270) 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcdf30640 1 -- 172.21.15.6:0/379368895 learned_addr learned my addr 172.21.15.6:0/379368895 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048264615 0 0) 0x7f7fc813f310 con 0x7f7fc813d320 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): my global_id is 164189 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fb4003530 con 0x7f7fc813d320 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2589213892 0 0) 0x7f7fb4003530 con 0x7f7fc813d320 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fc813f310 con 0x7f7fc813d320 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7fc0004040 con 0x7f7fc813d320 2024-09-16T05:50:07.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): dump: 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:50:07.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1599723566 0 0) 0x7f7fc8139550 con 0x7f7fc8072df0 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): my global_id is 154336 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fb4003530 con 0x7f7fc8072df0 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4160474032 0 0) 0x7f7fc813f310 con 0x7f7fc813d320 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 >> v1:172.21.15.63:6789/0 conn(0x7f7fc8072df0 legacy=0x7f7fc813ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 >> v1:172.21.15.6:6789/0 conn(0x7f7fc8072300 legacy=0x7f7fc813e4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 monclient: found mon.c 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:50:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc81382c0 con 0x7f7fc813d320 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 5 monclient: authenticate success, global_id 164189 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: _renew_subs 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc80740c0 con 0x7f7fc813d320 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: _renew_subs 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7fc8139270 con 0x7f7fc813d320 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7fc00043a0 con 0x7f7fc813d320 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7fc0004770 con 0x7f7fc813d320 2024-09-16T05:50:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc77fe640 4 set_mon_vals no callback set 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient: dump: 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7fa80065d0 con 0x7f7fa8002870 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7fc0004990 con 0x7f7fc813d320 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f7fc0044f10 con 0x7f7fc813d320 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fccf2e640 10 monclient: get_auth_request con 0x7f7fb4036a80 auth_method 0 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2674924948 0 0) 0x7f7fa80065d0 con 0x7f7fa8002870 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 10 monclient(hunting): my global_id is 154339 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.195+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fc813f310 con 0x7f7fa8002870 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4233596410 0 0) 0x7f7fc813f310 con 0x7f7fa8002870 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fa80065d0 con 0x7f7fa8002870 2024-09-16T05:50:07.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7fb8001a90 con 0x7f7fa8002870 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 10 monclient: dump: 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.216 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2748478888 0 0) 0x7f7fa80065d0 con 0x7f7fa8002870 2024-09-16T05:50:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fa800a160 con 0x7f7fa8002870 2024-09-16T05:50:07.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7fb8005b50 con 0x7f7fa8002870 2024-09-16T05:50:07.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:50:07.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:50:07.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 >> v1:172.21.15.63:6789/0 conn(0x7f7fa8002870 legacy=0x7f7fa8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.199+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 >> v1:172.21.15.63:6789/0 conn(0x7f7fa8002870 legacy=0x7f7fa8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:50:07.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.203+0000 7f7fcf993640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:50:07.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.203+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7fa8009e50 con 0x7f7fa8009110 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1649790895 0 0) 0x7f7fa8009e50 con 0x7f7fa8009110 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient(hunting): my global_id is 154342 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fa80065d0 con 0x7f7fa8009110 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1736803095 0 0) 0x7f7fa80065d0 con 0x7f7fa8009110 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fa8009e50 con 0x7f7fa8009110 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7fb80039a0 con 0x7f7fa8009110 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient: dump: 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:50:07.219 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2039638105 0 0) 0x7f7fa8009e50 con 0x7f7fa8009110 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7fa800c3e0 con 0x7f7fa8009110 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4120 (unknown 0 0 556776723) 0x7f7fb8001fb0 con 0x7f7fa8009110 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:50:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 >> v1:172.21.15.63:6789/0 conn(0x7f7fa8009110 legacy=0x7f7fa80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fc5ffb640 1 -- 172.21.15.6:0/379368895 >> v1:172.21.15.63:6789/0 conn(0x7f7fa8009110 legacy=0x7f7fa80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 66, 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 141950, 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:50:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:50:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:50:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:50:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:50:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:50:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:50:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:50:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:50:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:50:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:50:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:50:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:50:07.233 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:50:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7fb4036a80 legacy=0x7f7fb4038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcf993640 10 monclient: shutdown 2024-09-16T05:50:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcf993640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:50:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 >> v1:172.21.15.6:6790/0 conn(0x7f7fc813d320 legacy=0x7f7fc81d3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:50:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcdf30640 1 -- 172.21.15.6:0/379368895 reap_dead start 2024-09-16T05:50:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcdf30640 1 -- 172.21.15.6:0/379368895 reap_dead start 2024-09-16T05:50:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 shutdown_connections 2024-09-16T05:50:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 >> 172.21.15.6:0/379368895 conn(0x7f7fc80a1650 msgr2=0x7f7fc80a3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:50:07.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 shutdown_connections 2024-09-16T05:50:07.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:50:07.207+0000 7f7fcf993640 1 -- 172.21.15.6:0/379368895 wait complete. 2024-09-16T05:50:07.235 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:50:07.235 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:51:37.235 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:51:37.236 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T05:51:37.236 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T05:51:37.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T05:51:37.241 INFO:tasks.ceph.mon.a:Started 2024-09-16T05:51:37.242 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:51:37.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:51:37.339 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:37.338+0000 7f8510a94e00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 552 -> 0 (sync_force? mon rebuild?) 2024-09-16T05:51:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient: get_monmap_and_config 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient: build_initial_monmap 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient: monmap: 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:37.364372+0000 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:37.364372+0000 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:37.363 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 1 Processor -- start 2024-09-16T05:51:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 1 -- start start 2024-09-16T05:51:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient: init 2024-09-16T05:51:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:51:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient(hunting): picked mon.noname-c con 0x7f67f00a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient(hunting): picked mon.noname-a con 0x7f67f00a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient(hunting): picked mon.noname-b con 0x7f67f00a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f006e520 con 0x7f67f00a4290 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f00a0780 con 0x7f67f00a37a0 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f00a4c70 con 0x7f67f00a1a00 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f6d24640 10 monclient(hunting): authenticate will time out at 1959.170718s 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f52c1640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f67f00a1a00 0x7f67f013f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51970/0 (socket says 172.21.15.6:51970) 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67effff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f67f00a4290 0x7f67f013bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35282/0 (socket says 172.21.15.6:35282) 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67f52c1640 1 -- 172.21.15.6:0/3610571476 learned_addr learned my addr 172.21.15.6:0/3610571476 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956486851 0 0) 0x7f67f00a4c70 con 0x7f67f00a1a00 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient(hunting): my global_id is 164192 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67dc003530 con 0x7f67f00a1a00 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 414108122 0 0) 0x7f67f00a0780 con 0x7f67f00a37a0 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient(hunting): my global_id is 154345 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f00a4c70 con 0x7f67f00a37a0 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2641547221 0 0) 0x7f67f00a4c70 con 0x7f67f00a37a0 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 >> v1:172.21.15.6:6790/0 conn(0x7f67f00a1a00 legacy=0x7f67f013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 >> v1:172.21.15.6:6789/0 conn(0x7f67f00a4290 legacy=0x7f67f013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 monclient: found mon.noname-c 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f00a65f0 con 0x7f67f00a37a0 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.362+0000 7f67ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f67e0001600 con 0x7f67f00a37a0 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 10 monclient: dump: 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.369 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 5 monclient: authenticate success, global_id 154345 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f67e00018f0 con 0x7f67f00a37a0 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 1 -- 172.21.15.6:0/3610571476 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f67e0001cc0 con 0x7f67f00a37a0 2024-09-16T05:51:37.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 4 set_mon_vals no callback set 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ef7fe640 10 monclient: dump: 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.371 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: shutdown 2024-09-16T05:51:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- 172.21.15.6:0/3610571476 >> v1:172.21.15.63:6789/0 conn(0x7f67f00a37a0 legacy=0x7f67f00a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- 172.21.15.6:0/3610571476 shutdown_connections 2024-09-16T05:51:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- 172.21.15.6:0/3610571476 >> 172.21.15.6:0/3610571476 conn(0x7f67f00a58a0 msgr2=0x7f67f00a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- 172.21.15.6:0/3610571476 shutdown_connections 2024-09-16T05:51:37.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- 172.21.15.6:0/3610571476 wait complete. 2024-09-16T05:51:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: build_initial_monmap 2024-09-16T05:51:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: monmap: 2024-09-16T05:51:37.377 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:37.377 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 Processor -- start 2024-09-16T05:51:37.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- start start 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: init 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient(hunting): picked mon.noname-b con 0x7f67f00a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient(hunting): picked mon.noname-c con 0x7f67f00a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient(hunting): picked mon.noname-a con 0x7f67f00a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f0134640 con 0x7f67f00a4290 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f0135830 con 0x7f67f00a1a00 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f0132a90 con 0x7f67f00a37a0 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient(hunting): authenticate will time out at 1959.174693s 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67effff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f67f00a37a0 0x7f67f0132380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51974/0 (socket says 172.21.15.6:51974) 2024-09-16T05:51:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67effff640 1 -- 172.21.15.6:0/4123095668 learned_addr learned my addr 172.21.15.6:0/4123095668 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4655924 0 0) 0x7f67f0132a90 con 0x7f67f00a37a0 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): my global_id is 164195 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67d0003530 con 0x7f67f00a37a0 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 484685783 0 0) 0x7f67d0003530 con 0x7f67f00a37a0 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67f0132a90 con 0x7f67f00a37a0 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f67d8001850 con 0x7f67f00a37a0 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:51:37.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): dump: 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:37.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 305070386 0 0) 0x7f67f0135830 con 0x7f67f00a1a00 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient(hunting): my global_id is 154348 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67d0003530 con 0x7f67f00a1a00 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2505883958 0 0) 0x7f67f0132a90 con 0x7f67f00a37a0 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 >> v1:172.21.15.63:6789/0 conn(0x7f67f00a1a00 legacy=0x7f67f0131c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 >> v1:172.21.15.6:6789/0 conn(0x7f67f00a4290 legacy=0x7f67f01d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 monclient: found mon.c 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f00a0100 con 0x7f67f00a37a0 2024-09-16T05:51:37.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient: _finish_auth 0 2024-09-16T05:51:37.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:51:37.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:37.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 5 monclient: authenticate success, global_id 164195 2024-09-16T05:51:37.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: _renew_subs 2024-09-16T05:51:37.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67f0135550 con 0x7f67f00a37a0 2024-09-16T05:51:37.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: _renew_subs 2024-09-16T05:51:37.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f67f0133c60 con 0x7f67f00a37a0 2024-09-16T05:51:37.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f67d80047f0 con 0x7f67f00a37a0 2024-09-16T05:51:37.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f67d8004b40 con 0x7f67f00a37a0 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67eeffd640 4 set_mon_vals no callback set 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.366+0000 7f67ed7fa640 10 monclient: dump: 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.387 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.370+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f67d8004ce0 con 0x7f67f00a37a0 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.370+0000 7f67f6d24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.370+0000 7f67f6d24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.370+0000 7f67f6d24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.370+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d4005090 con 0x7f67f00a37a0 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.370+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f67d8044e90 con 0x7f67f00a37a0 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.370+0000 7f67f4ac0640 10 monclient: get_auth_request con 0x7f67d0036b40 auth_method 0 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.374+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f67d8015460 con 0x7f67f00a37a0 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.374+0000 7f67ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:37.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.374+0000 7f67ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.610+0000 7f67f6d24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:51:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.610+0000 7f67f6d24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.610+0000 7f67f6d24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.610+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f67d4005380 con 0x7f67f00a37a0 2024-09-16T05:51:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.610+0000 7f67ed7fa640 1 -- 172.21.15.6:0/4123095668 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 810803840) 0x7f67d8014d60 con 0x7f67f00a37a0 2024-09-16T05:51:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.610+0000 7f67ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.610+0000 7f67ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:37.613 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:51:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.614+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 >> v1:172.21.15.63:6812/3643207211 conn(0x7f67d0036b40 legacy=0x7f67d0039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.614+0000 7f67f6d24640 10 monclient: shutdown 2024-09-16T05:51:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.614+0000 7f67f6d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.614+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 >> v1:172.21.15.6:6790/0 conn(0x7f67f00a37a0 legacy=0x7f67f0132380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.614+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 shutdown_connections 2024-09-16T05:51:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.614+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 >> 172.21.15.6:0/4123095668 conn(0x7f67f00a58a0 msgr2=0x7f67f00a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.614+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 shutdown_connections 2024-09-16T05:51:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.614+0000 7f67f6d24640 1 -- 172.21.15.6:0/4123095668 wait complete. 2024-09-16T05:51:37.629 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:51:37.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:51:37.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:51:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient: get_monmap_and_config 2024-09-16T05:51:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient: build_initial_monmap 2024-09-16T05:51:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient: monmap: 2024-09-16T05:51:37.750 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:37.752024+0000 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:37.752024+0000 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 1 Processor -- start 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 1 -- start start 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient: init 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient(hunting): picked mon.noname-c con 0x7fee2409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient(hunting): picked mon.noname-a con 0x7fee2409b520 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient(hunting): picked mon.noname-b con 0x7fee24141690 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee240a1350 con 0x7fee2409b520 2024-09-16T05:51:37.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2409cdf0 con 0x7fee2409aa30 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2409dad0 con 0x7fee24141690 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient(hunting): authenticate will time out at 1959.558869s 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee295b6640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fee24141690 0x7fee24143a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51988/0 (socket says 172.21.15.6:51988) 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee295b6640 1 -- 172.21.15.6:0/2233743282 learned_addr learned my addr 172.21.15.6:0/2233743282 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee23fff640 1 --1- 172.21.15.6:0/2233743282 >> v1:172.21.15.6:6789/0 conn(0x7fee2409b520 0x7fee2413ff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35298/0 (socket says 172.21.15.6:35298) 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 723148140 0 0) 0x7fee2409dad0 con 0x7fee24141690 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): my global_id is 164198 2024-09-16T05:51:37.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee0c003530 con 0x7fee24141690 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2852729315 0 0) 0x7fee18001050 con 0x7fee2409b520 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): my global_id is 174099 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee2409dad0 con 0x7fee2409b520 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1064570632 0 0) 0x7fee2409cdf0 con 0x7fee2409aa30 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient(hunting): my global_id is 154351 2024-09-16T05:51:37.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee18001050 con 0x7fee2409aa30 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2790057783 0 0) 0x7fee0c003530 con 0x7fee24141690 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 >> v1:172.21.15.63:6789/0 conn(0x7fee2409aa30 legacy=0x7fee2409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 >> v1:172.21.15.6:6789/0 conn(0x7fee2409b520 legacy=0x7fee2413ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 monclient: found mon.noname-b 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee240a10d0 con 0x7fee24141690 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee1c0019a0 con 0x7fee24141690 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 5 monclient: authenticate success, global_id 164198 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: dump: 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.763 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee1c001c00 con 0x7fee24141690 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee2b019640 4 set_mon_vals no callback set 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 1 -- 172.21.15.6:0/2233743282 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee1c001f90 con 0x7fee24141690 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.750+0000 7fee237fe640 10 monclient: dump: 2024-09-16T05:51:37.764 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: shutdown 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- 172.21.15.6:0/2233743282 >> v1:172.21.15.6:6790/0 conn(0x7fee24141690 legacy=0x7fee24143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- 172.21.15.6:0/2233743282 shutdown_connections 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- 172.21.15.6:0/2233743282 >> 172.21.15.6:0/2233743282 conn(0x7fee240a1650 msgr2=0x7fee240a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- 172.21.15.6:0/2233743282 shutdown_connections 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- 172.21.15.6:0/2233743282 wait complete. 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: build_initial_monmap 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: monmap: 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 Processor -- start 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- start start 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: init 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient(hunting): picked mon.noname-a con 0x7fee2409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient(hunting): picked mon.noname-b con 0x7fee2409b520 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient(hunting): picked mon.noname-c con 0x7fee24141690 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee24140f60 con 0x7fee2409aa30 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee24124d50 con 0x7fee2409b520 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2413b1d0 con 0x7fee24141690 2024-09-16T05:51:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient(hunting): authenticate will time out at 1959.562474s 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee28db5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fee2409aa30 0x7fee2413a3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35302/0 (socket says 172.21.15.6:35302) 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee28db5640 1 -- 172.21.15.6:0/4262631404 learned_addr learned my addr 172.21.15.6:0/4262631404 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee295b6640 1 --1- 172.21.15.6:0/4262631404 >> v1:172.21.15.6:6790/0 conn(0x7fee24141690 0x7fee241d7c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:51998/0 (socket says 172.21.15.6:51998) 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1540310625 0 0) 0x7fee2413b1d0 con 0x7fee24141690 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): my global_id is 164201 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee10003530 con 0x7fee24141690 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004728079 0 0) 0x7fee24140f60 con 0x7fee2409aa30 2024-09-16T05:51:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): my global_id is 174102 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee2413b1d0 con 0x7fee2409aa30 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1325610379 0 0) 0x7fee10003530 con 0x7fee24141690 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee24140f60 con 0x7fee24141690 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee1c0018a0 con 0x7fee24141690 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): dump: 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3386939831 0 0) 0x7fee2413b1d0 con 0x7fee2409aa30 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee10003530 con 0x7fee2409aa30 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee14001920 con 0x7fee2409aa30 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): dump: 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997309668 0 0) 0x7fee18001980 con 0x7fee2409b520 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient(hunting): my global_id is 154354 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee2413b1d0 con 0x7fee2409b520 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3744881440 0 0) 0x7fee24140f60 con 0x7fee24141690 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 >> v1:172.21.15.63:6789/0 conn(0x7fee2409b520 legacy=0x7fee2413aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 >> v1:172.21.15.6:6789/0 conn(0x7fee2409aa30 legacy=0x7fee2413a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient: _finish_hunting 0 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 monclient: found mon.c 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee241407d0 con 0x7fee24141690 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient: _finish_auth 0 2024-09-16T05:51:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient: _check_auth_tickets 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 5 monclient: authenticate success, global_id 164201 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: _renew_subs 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee2413fe30 con 0x7fee24141690 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: _renew_subs 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fee241d93a0 con 0x7fee24141690 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee1c001c00 con 0x7fee24141690 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee1c004f80 con 0x7fee24141690 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee22ffd640 4 set_mon_vals no callback set 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.754+0000 7fee217fa640 10 monclient: dump: 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee2b019640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fee040065d0 con 0x7fee04002870 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fee1c00fd20 con 0x7fee24141690 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fee1c045100 con 0x7fee24141690 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331877888 0 0) 0x7fee040065d0 con 0x7fee04002870 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient(hunting): my global_id is 174105 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee24140f60 con 0x7fee04002870 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee23fff640 10 monclient: get_auth_request con 0x7fee10036ba0 auth_method 0 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3503794114 0 0) 0x7fee24140f60 con 0x7fee04002870 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee040065d0 con 0x7fee04002870 2024-09-16T05:51:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee140037c0 con 0x7fee04002870 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient: dump: 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.776 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 553071785 0 0) 0x7fee040065d0 con 0x7fee04002870 2024-09-16T05:51:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee0400a160 con 0x7fee04002870 2024-09-16T05:51:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fee140057d0 con 0x7fee04002870 2024-09-16T05:51:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 >> v1:172.21.15.6:6789/0 conn(0x7fee04002870 legacy=0x7fee04004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.758+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 >> v1:172.21.15.6:6789/0 conn(0x7fee04002870 legacy=0x7fee04004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.762+0000 7fee2b019640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.762+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fee04009e30 con 0x7fee040090f0 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 856218831 0 0) 0x7fee04009e30 con 0x7fee040090f0 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient(hunting): my global_id is 174108 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee040065d0 con 0x7fee040090f0 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4042430918 0 0) 0x7fee040065d0 con 0x7fee040090f0 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee04009e30 con 0x7fee040090f0 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fee14003e60 con 0x7fee040090f0 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient: dump: 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 234433331 0 0) 0x7fee04009e30 con 0x7fee040090f0 2024-09-16T05:51:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee0400c3a0 con 0x7fee040090f0 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3961 (unknown 0 0 954686302) 0x7fee140040c0 con 0x7fee040090f0 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 >> v1:172.21.15.6:6789/0 conn(0x7fee040090f0 legacy=0x7fee04009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee217fa640 1 -- 172.21.15.6:0/4262631404 >> v1:172.21.15.6:6789/0 conn(0x7fee040090f0 legacy=0x7fee04009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:37.781 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 70, 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 439, 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:37.782 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:51:37.783 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.784 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:51:37.785 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:51:37.786 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:51:37.786 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:37.786 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:51:37.786 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:37.786 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:37.786 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:37.786 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:37.787 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.788 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:51:37.789 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:51:37.790 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:37.791 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:37.792 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:51:37.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.794 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:37.794 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:37.794 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:37.794 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.794 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.794 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:51:37.794 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:51:37.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 >> v1:172.21.15.63:6812/3643207211 conn(0x7fee10036ba0 legacy=0x7fee10039060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee2b019640 10 monclient: shutdown 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee2b019640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 >> v1:172.21.15.6:6790/0 conn(0x7fee24141690 legacy=0x7fee241d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee295b6640 1 -- 172.21.15.6:0/4262631404 reap_dead start 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee295b6640 1 -- 172.21.15.6:0/4262631404 reap_dead start 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 shutdown_connections 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 >> 172.21.15.6:0/4262631404 conn(0x7fee240a1650 msgr2=0x7fee240a2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 shutdown_connections 2024-09-16T05:51:37.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.766+0000 7fee2b019640 1 -- 172.21.15.6:0/4262631404 wait complete. 2024-09-16T05:51:37.796 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:51:37.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:51:37.797 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:51:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.918+0000 7f5967280640 10 monclient: get_monmap_and_config 2024-09-16T05:51:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.918+0000 7f5967280640 10 monclient: build_initial_monmap 2024-09-16T05:51:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.918+0000 7f5967280640 10 monclient: monmap: 2024-09-16T05:51:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:37.922853+0000 2024-09-16T05:51:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:37.922853+0000 2024-09-16T05:51:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:37.922 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 Processor -- start 2024-09-16T05:51:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- start start 2024-09-16T05:51:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient: init 2024-09-16T05:51:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:37.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:51:37.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient(hunting): picked mon.noname-b con 0x7f596006dd90 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:37.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient(hunting): picked mon.noname-a con 0x7f59600a2c00 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:37.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient(hunting): picked mon.noname-c con 0x7f596013cfe0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:37.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59600a4c20 con 0x7f59600a2c00 2024-09-16T05:51:37.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59600a49a0 con 0x7f596013cfe0 2024-09-16T05:51:37.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59600a0720 con 0x7f596006dd90 2024-09-16T05:51:37.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient(hunting): authenticate will time out at 1959.729716s 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f596501c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f596006dd90 0x7f59600a24f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52014/0 (socket says 172.21.15.6:52014) 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f596501c640 1 -- 172.21.15.6:0/2356491840 learned_addr learned my addr 172.21.15.6:0/2356491840 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1966161081 0 0) 0x7f59600a4c20 con 0x7f59600a2c00 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient(hunting): my global_id is 174111 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f594c003530 con 0x7f59600a2c00 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738536856 0 0) 0x7f59600a0720 con 0x7f596006dd90 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient(hunting): my global_id is 164204 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59600a4c20 con 0x7f596006dd90 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3466386384 0 0) 0x7f594c003530 con 0x7f59600a2c00 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 >> v1:172.21.15.6:6790/0 conn(0x7f596006dd90 legacy=0x7f59600a24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 >> v1:172.21.15.63:6789/0 conn(0x7f596013cfe0 legacy=0x7f596013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: _finish_hunting 0 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 monclient: found mon.noname-a 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5960136a80 con 0x7f59600a2c00 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: _finish_auth 0 2024-09-16T05:51:37.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: _check_auth_tickets 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5950001620 con 0x7f59600a2c00 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 5 monclient: authenticate success, global_id 174111 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: dump: 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.934 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5950001880 con 0x7f59600a2c00 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 1 -- 172.21.15.6:0/2356491840 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5950001c90 con 0x7f59600a2c00 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 4 set_mon_vals no callback set 2024-09-16T05:51:37.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5957fff640 10 monclient: dump: 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 10 monclient: shutdown 2024-09-16T05:51:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- 172.21.15.6:0/2356491840 >> v1:172.21.15.6:6789/0 conn(0x7f59600a2c00 legacy=0x7f596013b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- 172.21.15.6:0/2356491840 shutdown_connections 2024-09-16T05:51:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- 172.21.15.6:0/2356491840 >> 172.21.15.6:0/2356491840 conn(0x7f59600a5800 msgr2=0x7f59600a5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:37.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- 172.21.15.6:0/2356491840 shutdown_connections 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.922+0000 7f5967280640 1 -- 172.21.15.6:0/2356491840 wait complete. 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: build_initial_monmap 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: monmap: 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:37.938 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 1 Processor -- start 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 1 -- start start 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: init 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient(hunting): picked mon.noname-c con 0x7f596006dd90 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient(hunting): picked mon.noname-b con 0x7f59600a2c00 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient(hunting): picked mon.noname-a con 0x7f596013cfe0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5960140790 con 0x7f596013cfe0 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59601403f0 con 0x7f59600a2c00 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f596006ecf0 con 0x7f596006dd90 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient(hunting): authenticate will time out at 1959.733074s 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f596581d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f596013cfe0 0x7f59601d3a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35318/0 (socket says 172.21.15.6:35318) 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f596581d640 1 -- 172.21.15.6:0/981643705 learned_addr learned my addr 172.21.15.6:0/981643705 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635254893 0 0) 0x7f5958001650 con 0x7f596013cfe0 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): my global_id is 174114 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5944003530 con 0x7f596013cfe0 2024-09-16T05:51:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175546709 0 0) 0x7f596006ecf0 con 0x7f596006dd90 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): my global_id is 164207 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5958001650 con 0x7f596006dd90 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2951807206 0 0) 0x7f5944003530 con 0x7f596013cfe0 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f596006ecf0 con 0x7f596013cfe0 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5958001ea0 con 0x7f596013cfe0 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): dump: 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.941 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 929401231 0 0) 0x7f5958001650 con 0x7f596006dd90 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5944003530 con 0x7f596006dd90 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 949905736 0 0) 0x7f59601403f0 con 0x7f59600a2c00 2024-09-16T05:51:37.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): my global_id is 154360 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5958001650 con 0x7f59600a2c00 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f595c0010c0 con 0x7f596006dd90 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): dump: 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.943 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2505620513 0 0) 0x7f596006ecf0 con 0x7f596013cfe0 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 >> v1:172.21.15.6:6790/0 conn(0x7f596006dd90 legacy=0x7f596009f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 >> v1:172.21.15.63:6789/0 conn(0x7f59600a2c00 legacy=0x7f59601d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 monclient: found mon.a 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59601c8010 con 0x7f596013cfe0 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: _finish_auth 0 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 5 monclient: authenticate success, global_id 174114 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: _renew_subs 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f596009f930 con 0x7f596013cfe0 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: _renew_subs 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f596009c730 con 0x7f596013cfe0 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5958003c50 con 0x7f596013cfe0 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:37.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5958004020 con 0x7f596013cfe0 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f59577fe640 4 set_mon_vals no callback set 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: dump: 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.946 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5958004240 con 0x7f596013cfe0 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59400065d0 con 0x7f5940002870 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f59580453c0 con 0x7f596013cfe0 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1498167106 0 0) 0x7f59400065d0 con 0x7f5940002870 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient(hunting): my global_id is 164210 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f596006ecf0 con 0x7f5940002870 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f596481b640 10 monclient: get_auth_request con 0x7f5944036bd0 auth_method 0 2024-09-16T05:51:37.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1176217910 0 0) 0x7f596006ecf0 con 0x7f5940002870 2024-09-16T05:51:37.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59400065d0 con 0x7f5940002870 2024-09-16T05:51:37.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f595c004b80 con 0x7f5940002870 2024-09-16T05:51:37.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:37.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 10 monclient: dump: 2024-09-16T05:51:37.948 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2598065366 0 0) 0x7f59400065d0 con 0x7f5940002870 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.926+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f594000a160 con 0x7f5940002870 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.930+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f595c004d70 con 0x7f5940002870 2024-09-16T05:51:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.930+0000 7f5955ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:37.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.930+0000 7f5955ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:37.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.930+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 >> v1:172.21.15.6:6790/0 conn(0x7f5940002870 legacy=0x7f5940004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.930+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 >> v1:172.21.15.6:6790/0 conn(0x7f5940002870 legacy=0x7f5940004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:51:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5940009e20 con 0x7f5940009110 2024-09-16T05:51:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3736075552 0 0) 0x7f5940009e20 con 0x7f5940009110 2024-09-16T05:51:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:51:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:37.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient(hunting): my global_id is 164213 2024-09-16T05:51:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59400065d0 con 0x7f5940009110 2024-09-16T05:51:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2229291373 0 0) 0x7f59400065d0 con 0x7f5940009110 2024-09-16T05:51:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5940009e20 con 0x7f5940009110 2024-09-16T05:51:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f595c0052b0 con 0x7f5940009110 2024-09-16T05:51:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient: dump: 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 419580924 0 0) 0x7f5940009e20 con 0x7f5940009110 2024-09-16T05:51:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f594000c390 con 0x7f5940009110 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 1429337778) 0x7f595c0070d0 con 0x7f5940009110 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 >> v1:172.21.15.6:6790/0 conn(0x7f5940009110 legacy=0x7f59400095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5955ffb640 1 -- 172.21.15.6:0/981643705 >> v1:172.21.15.6:6790/0 conn(0x7f5940009110 legacy=0x7f59400095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:37.956 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 70, 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 108894, 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:37.957 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:37.958 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.959 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:51:37.959 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:51:37.959 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:37.959 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:37.959 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:37.959 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:37.959 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:51:37.960 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:51:37.961 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:51:37.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.964 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.965 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:51:37.966 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.967 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:51:37.968 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:51:37.969 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5944036bd0 legacy=0x7f5944039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 10 monclient: shutdown 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 >> v1:172.21.15.6:6789/0 conn(0x7f596013cfe0 legacy=0x7f59601d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f596581d640 1 -- 172.21.15.6:0/981643705 reap_dead start 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f596581d640 1 -- 172.21.15.6:0/981643705 reap_dead start 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 shutdown_connections 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 >> 172.21.15.6:0/981643705 conn(0x7f59600a5800 msgr2=0x7f59600a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 shutdown_connections 2024-09-16T05:51:37.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:37.934+0000 7f5967280640 1 -- 172.21.15.6:0/981643705 wait complete. 2024-09-16T05:51:37.972 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:51:37.972 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:51:37.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:51:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient: get_monmap_and_config 2024-09-16T05:51:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient: build_initial_monmap 2024-09-16T05:51:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient: monmap: 2024-09-16T05:51:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:38.091319+0000 2024-09-16T05:51:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:38.091319+0000 2024-09-16T05:51:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:38.091 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 Processor -- start 2024-09-16T05:51:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- start start 2024-09-16T05:51:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient: init 2024-09-16T05:51:38.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:38.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:51:38.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient(hunting): picked mon.noname-c con 0x7f0c6009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient(hunting): picked mon.noname-b con 0x7f0c6009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient(hunting): picked mon.noname-a con 0x7f0c600a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c600a1370 con 0x7f0c600a0da0 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6009ead0 con 0x7f0c6009aa30 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6009edb0 con 0x7f0c6009e230 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient(hunting): authenticate will time out at 1959.897846s 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5e7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0c6009e230 0x7f0c601399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52066/0 (socket says 172.21.15.6:52066) 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5e7fc640 1 -- 172.21.15.6:0/492846515 learned_addr learned my addr 172.21.15.6:0/492846515 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143874156 0 0) 0x7f0c600a1370 con 0x7f0c600a0da0 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient(hunting): my global_id is 174117 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c48003530 con 0x7f0c600a0da0 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3163476761 0 0) 0x7f0c6009edb0 con 0x7f0c6009e230 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient(hunting): my global_id is 164216 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c600a1370 con 0x7f0c6009e230 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1309685207 0 0) 0x7f0c48003530 con 0x7f0c600a0da0 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 >> v1:172.21.15.6:6790/0 conn(0x7f0c6009e230 legacy=0x7f0c601399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 >> v1:172.21.15.63:6789/0 conn(0x7f0c6009aa30 legacy=0x7f0c6009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:51:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 monclient: found mon.noname-a 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6009dd50 con 0x7f0c600a0da0 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: _finish_auth 0 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c580019a0 con 0x7f0c600a0da0 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 5 monclient: authenticate success, global_id 174117 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: dump: 2024-09-16T05:51:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c58001c00 con 0x7f0c600a0da0 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 1 -- 172.21.15.6:0/492846515 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c58001f90 con 0x7f0c600a0da0 2024-09-16T05:51:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c5dffb640 10 monclient: dump: 2024-09-16T05:51:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.110 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 4 set_mon_vals no callback set 2024-09-16T05:51:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 10 monclient: shutdown 2024-09-16T05:51:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- 172.21.15.6:0/492846515 >> v1:172.21.15.6:6789/0 conn(0x7f0c600a0da0 legacy=0x7f0c6013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- 172.21.15.6:0/492846515 shutdown_connections 2024-09-16T05:51:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- 172.21.15.6:0/492846515 >> 172.21.15.6:0/492846515 conn(0x7f0c600a1650 msgr2=0x7f0c600a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:38.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- 172.21.15.6:0/492846515 shutdown_connections 2024-09-16T05:51:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.090+0000 7f0c65441640 1 -- 172.21.15.6:0/492846515 wait complete. 2024-09-16T05:51:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: build_initial_monmap 2024-09-16T05:51:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: monmap: 2024-09-16T05:51:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:38.112 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:38.114 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 1 Processor -- start 2024-09-16T05:51:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 1 -- start start 2024-09-16T05:51:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: init 2024-09-16T05:51:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:51:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient(hunting): picked mon.noname-b con 0x7f0c6009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient(hunting): picked mon.noname-c con 0x7f0c6009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient(hunting): picked mon.noname-a con 0x7f0c600a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6009c720 con 0x7f0c600a0da0 2024-09-16T05:51:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6009d570 con 0x7f0c6009aa30 2024-09-16T05:51:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6009d850 con 0x7f0c6009e230 2024-09-16T05:51:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient(hunting): authenticate will time out at 1959.900907s 2024-09-16T05:51:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c5e7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0c6009e230 0x7f0c60133fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52072/0 (socket says 172.21.15.6:52072) 2024-09-16T05:51:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c5e7fc640 1 -- 172.21.15.6:0/1009396822 learned_addr learned my addr 172.21.15.6:0/1009396822 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 119522165 0 0) 0x7f0c6009dd50 con 0x7f0c600a0da0 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): my global_id is 174120 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c4c003530 con 0x7f0c600a0da0 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1041306510 0 0) 0x7f0c6009d850 con 0x7f0c6009e230 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): my global_id is 164219 2024-09-16T05:51:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c6009dd50 con 0x7f0c6009e230 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4210251962 0 0) 0x7f0c4c003530 con 0x7f0c600a0da0 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c6009d850 con 0x7f0c600a0da0 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c580018a0 con 0x7f0c600a0da0 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): dump: 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3708106483 0 0) 0x7f0c6009dd50 con 0x7f0c6009e230 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c4c003530 con 0x7f0c6009e230 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c50001a20 con 0x7f0c6009e230 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): dump: 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932834397 0 0) 0x7f0c6009d570 con 0x7f0c6009aa30 2024-09-16T05:51:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): my global_id is 154366 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c6009dd50 con 0x7f0c6009aa30 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 767426084 0 0) 0x7f0c6009d850 con 0x7f0c600a0da0 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.6:6790/0 conn(0x7f0c6009e230 legacy=0x7f0c60133fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.63:6789/0 conn(0x7f0c6009aa30 legacy=0x7f0c601338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 monclient: found mon.a 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c601346d0 con 0x7f0c600a0da0 2024-09-16T05:51:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 5 monclient: authenticate success, global_id 174120 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: _renew_subs 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6006e900 con 0x7f0c600a0da0 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: _renew_subs 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c6009c400 con 0x7f0c600a0da0 2024-09-16T05:51:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c5f7fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.6:6789/0 conn(0x7f0c600a0da0 legacy=0x7f0c601d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c5f7fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.6:6789/0 conn(0x7f0c600a0da0 legacy=0x7f0c601d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c5f7fe640 1 --1- 172.21.15.6:0/1009396822 >> v1:172.21.15.6:6789/0 conn(0x7f0c600a0da0 0x7f0c601d3720 :-1 s=OPENED pgs=1075 cs=1 l=1).handle_message read tag failed 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c5f7fe640 1 --1- 172.21.15.6:0/1009396822 >> v1:172.21.15.6:6789/0 conn(0x7f0c600a0da0 0x7f0c601d3720 :-1 s=OPENED pgs=1075 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.6:6789/0 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.6:6789/0 conn(0x7f0c600a0da0 legacy=0x7f0c601d3720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): picked mon.b con 0x7f0c6009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): picked mon.c con 0x7f0c6009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): picked mon.a con 0x7f0c4c00c1a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c6009d850 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c4c003530 con 0x7f0c6009aa30 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c6009d570 con 0x7f0c6009e230 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931432107 0 0) 0x7f0c6009d570 con 0x7f0c6009e230 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f0c4c0101e0 con 0x7f0c6009e230 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517828798 0 0) 0x7f0c6009d850 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T05:51:38.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f0c6009d570 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c600740c0 con 0x7f0c6009b520 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 2829363119 0 0) 0x7f0c6009d570 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.6:6790/0 conn(0x7f0c6009e230 legacy=0x7f0c4c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.63:6789/0 conn(0x7f0c6009aa30 legacy=0x7f0c4c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 monclient: found mon.a 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f0c4c0133f0 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c58004e70 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: dump: 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c58004e70 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c5d7fa640 4 set_mon_vals no callback set 2024-09-16T05:51:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0c5800fca0 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.0 v1:172.21.15.6:6789/0 6 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f0c58045030 con 0x7f0c4c00c1a0 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567709341 0 0) 0x7f0c600740c0 con 0x7f0c6009b520 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 10 monclient(hunting): my global_id is 154369 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.094+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c6009d570 con 0x7f0c6009b520 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c5e7fc640 10 monclient: get_auth_request con 0x7f0c600a0da0 auth_method 0 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2607788560 0 0) 0x7f0c6009d570 con 0x7f0c6009b520 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c600740c0 con 0x7f0c6009b520 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c54001870 con 0x7f0c6009b520 2024-09-16T05:51:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 10 monclient: dump: 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1795075965 0 0) 0x7f0c600740c0 con 0x7f0c6009b520 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c600008e0 con 0x7f0c6009b520 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0c54001690 con 0x7f0c6009b520 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.63:6789/0 conn(0x7f0c6009b520 legacy=0x7f0c601338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.098+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.63:6789/0 conn(0x7f0c6009b520 legacy=0x7f0c601338b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.102+0000 7f0c65441640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:51:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.102+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c6002d660 con 0x7f0c6013d600 2024-09-16T05:51:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1875969127 0 0) 0x7f0c6002d660 con 0x7f0c6013d600 2024-09-16T05:51:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient(hunting): my global_id is 154372 2024-09-16T05:51:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c600740c0 con 0x7f0c6013d600 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2378180169 0 0) 0x7f0c600740c0 con 0x7f0c6013d600 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c6002d660 con 0x7f0c6013d600 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0c540037c0 con 0x7f0c6013d600 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient: dump: 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.133 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2892738518 0 0) 0x7f0c6002d660 con 0x7f0c6013d600 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c6009da90 con 0x7f0c6013d600 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 1274055830) 0x7f0c54003a20 con 0x7f0c6013d600 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.63:6789/0 conn(0x7f0c6013d600 legacy=0x7f0c60133fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c477fe640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.63:6789/0 conn(0x7f0c6013d600 legacy=0x7f0c60133fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 70, 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 232845, 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:51:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:38.136 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:38.137 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:51:38.138 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:51:38.139 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:51:38.140 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:38.140 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:51:38.140 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:38.140 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:38.140 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:38.140 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:38.140 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:38.140 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:38.141 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:38.141 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:38.141 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:38.141 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:38.141 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:38.141 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:38.142 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:51:38.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.142 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:51:38.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.142 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:38.142 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:38.143 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:38.143 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:38.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.143 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:38.143 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:51:38.143 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:38.143 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:38.144 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:38.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.144 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:51:38.144 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:51:38.144 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:38.144 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:38.145 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:38.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.145 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:38.145 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:38.146 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:38.146 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:38.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.146 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:38.146 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:51:38.146 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:38.147 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:38.147 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:38.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.147 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:51:38.147 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:51:38.147 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:38.148 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:38.148 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:38.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.148 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:38.148 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:38.149 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:38.149 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:38.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.149 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:38.149 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:51:38.149 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:38.150 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:38.150 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:38.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.150 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:51:38.150 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:51:38.150 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:38.151 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:38.151 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:38.151 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:38.151 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:38.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.151 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:51:38.152 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:51:38.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.152 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:38.152 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:38.152 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:38.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:38.153 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:38.153 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:51:38.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.153 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:38.153 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:51:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:51:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:38.154 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:38.155 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:51:38.155 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:38.155 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:38.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:38.155 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:51:38.155 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:51:38.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0c600a0da0 legacy=0x7f0c4c044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c65441640 10 monclient: shutdown 2024-09-16T05:51:38.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c65441640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:38.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 >> v1:172.21.15.6:6789/0 conn(0x7f0c4c00c1a0 legacy=0x7f0c4c00e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c5f7fe640 1 -- 172.21.15.6:0/1009396822 reap_dead start 2024-09-16T05:51:38.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c5f7fe640 1 -- 172.21.15.6:0/1009396822 reap_dead start 2024-09-16T05:51:38.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 shutdown_connections 2024-09-16T05:51:38.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 >> 172.21.15.6:0/1009396822 conn(0x7f0c600a1650 msgr2=0x7f0c600a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:38.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 shutdown_connections 2024-09-16T05:51:38.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.106+0000 7f0c65441640 1 -- 172.21.15.6:0/1009396822 wait complete. 2024-09-16T05:51:38.159 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:51:38.159 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:51:38.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.274+0000 7fbb64b2d640 10 monclient: get_monmap_and_config 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.274+0000 7fbb64b2d640 10 monclient: build_initial_monmap 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.274+0000 7fbb64b2d640 10 monclient: monmap: 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:38.277441+0000 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:38.277441+0000 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.276 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:38.277 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:38.277 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.274+0000 7fbb64b2d640 1 Processor -- start 2024-09-16T05:51:38.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.274+0000 7fbb64b2d640 1 -- start start 2024-09-16T05:51:38.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.274+0000 7fbb64b2d640 10 monclient: init 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): picked mon.noname-a con 0x7fbb6009bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): picked mon.noname-b con 0x7fbb6009bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): picked mon.noname-c con 0x7fbb601397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb600a1350 con 0x7fbb6009bb40 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb600a10d0 con 0x7fbb601397f0 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb6009d940 con 0x7fbb6009bf20 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:38.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): authenticate will time out at 1960.084231s 2024-09-16T05:51:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5dd9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbb6009bf20 0x7fbb6009c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52098/0 (socket says 172.21.15.6:52098) 2024-09-16T05:51:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5dd9b640 1 -- 172.21.15.6:0/3183946156 learned_addr learned my addr 172.21.15.6:0/3183946156 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 916808712 0 0) 0x7fbb600a1350 con 0x7fbb6009bb40 2024-09-16T05:51:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient(hunting): my global_id is 174123 2024-09-16T05:51:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb4c003530 con 0x7fbb6009bb40 2024-09-16T05:51:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1624116473 0 0) 0x7fbb6009d940 con 0x7fbb6009bf20 2024-09-16T05:51:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient(hunting): my global_id is 164222 2024-09-16T05:51:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb600a1350 con 0x7fbb6009bf20 2024-09-16T05:51:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 26588008 0 0) 0x7fbb600a1350 con 0x7fbb6009bf20 2024-09-16T05:51:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 >> v1:172.21.15.63:6789/0 conn(0x7fbb601397f0 legacy=0x7fbb6013bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 >> v1:172.21.15.6:6789/0 conn(0x7fbb6009bb40 legacy=0x7fbb6009e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: _finish_hunting 0 2024-09-16T05:51:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 monclient: found mon.noname-b 2024-09-16T05:51:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:51:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6009dc20 con 0x7fbb6009bf20 2024-09-16T05:51:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: _finish_auth 0 2024-09-16T05:51:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: _check_auth_tickets 2024-09-16T05:51:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 5 monclient: authenticate success, global_id 164222 2024-09-16T05:51:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb48001660 con 0x7fbb6009bf20 2024-09-16T05:51:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:51:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: dump: 2024-09-16T05:51:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.286 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.287 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.287 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.287 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.287 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.287 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.287 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.288 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:38.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb48001800 con 0x7fbb6009bf20 2024-09-16T05:51:38.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:38.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 1 -- 172.21.15.6:0/3183946156 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb48001b90 con 0x7fbb6009bf20 2024-09-16T05:51:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 4 set_mon_vals no callback set 2024-09-16T05:51:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5d59a640 10 monclient: dump: 2024-09-16T05:51:38.289 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.290 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.291 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: shutdown 2024-09-16T05:51:38.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- 172.21.15.6:0/3183946156 >> v1:172.21.15.6:6790/0 conn(0x7fbb6009bf20 legacy=0x7fbb6009c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- 172.21.15.6:0/3183946156 shutdown_connections 2024-09-16T05:51:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- 172.21.15.6:0/3183946156 >> 172.21.15.6:0/3183946156 conn(0x7fbb600a1650 msgr2=0x7fbb600a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- 172.21.15.6:0/3183946156 shutdown_connections 2024-09-16T05:51:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- 172.21.15.6:0/3183946156 wait complete. 2024-09-16T05:51:38.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: build_initial_monmap 2024-09-16T05:51:38.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: monmap: 2024-09-16T05:51:38.293 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:38.293 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:38.293 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:38.293 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:38.293 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:38.294 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 Processor -- start 2024-09-16T05:51:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- start start 2024-09-16T05:51:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: init 2024-09-16T05:51:38.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:38.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:51:38.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): picked mon.noname-c con 0x7fbb6009bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:38.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): picked mon.noname-a con 0x7fbb6009bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:38.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): picked mon.noname-b con 0x7fbb601397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:38.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb6011c770 con 0x7fbb6009bf20 2024-09-16T05:51:38.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb60072790 con 0x7fbb601397f0 2024-09-16T05:51:38.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb6009b590 con 0x7fbb6009bb40 2024-09-16T05:51:38.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:38.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient(hunting): authenticate will time out at 1960.087043s 2024-09-16T05:51:38.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5dd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbb6009bf20 0x7fbb60072080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35384/0 (socket says 172.21.15.6:35384) 2024-09-16T05:51:38.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb5dd9b640 1 -- 172.21.15.6:0/1646648106 learned_addr learned my addr 172.21.15.6:0/1646648106 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:38.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538324581 0 0) 0x7fbb6011c770 con 0x7fbb6009bf20 2024-09-16T05:51:38.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): my global_id is 174126 2024-09-16T05:51:38.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb40003530 con 0x7fbb6009bf20 2024-09-16T05:51:38.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3949336223 0 0) 0x7fbb6009b590 con 0x7fbb6009bb40 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): my global_id is 164225 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb6011c770 con 0x7fbb6009bb40 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 863133171 0 0) 0x7fbb40003530 con 0x7fbb6009bf20 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb6009b590 con 0x7fbb6009bf20 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb480015a0 con 0x7fbb6009bf20 2024-09-16T05:51:38.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): dump: 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2352420701 0 0) 0x7fbb6011c770 con 0x7fbb6009bb40 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb40003530 con 0x7fbb6009bb40 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb500018a0 con 0x7fbb6009bb40 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): dump: 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.306 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.307 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.307 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.307 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.307 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.307 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354464229 0 0) 0x7fbb60072790 con 0x7fbb601397f0 2024-09-16T05:51:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:51:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient(hunting): my global_id is 154378 2024-09-16T05:51:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb6011c770 con 0x7fbb601397f0 2024-09-16T05:51:38.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2466398001 0 0) 0x7fbb6009b590 con 0x7fbb6009bf20 2024-09-16T05:51:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 >> v1:172.21.15.6:6790/0 conn(0x7fbb6009bb40 legacy=0x7fbb60071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 >> v1:172.21.15.63:6789/0 conn(0x7fbb601397f0 legacy=0x7fbb601d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:51:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 monclient: found mon.a 2024-09-16T05:51:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6011ca50 con 0x7fbb6009bf20 2024-09-16T05:51:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient: _finish_auth 0 2024-09-16T05:51:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:51:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 5 monclient: authenticate success, global_id 174126 2024-09-16T05:51:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: _renew_subs 2024-09-16T05:51:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.278+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6013c540 con 0x7fbb6009bf20 2024-09-16T05:51:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb64b2d640 10 monclient: _renew_subs 2024-09-16T05:51:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb64b2d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb6011c490 con 0x7fbb6009bf20 2024-09-16T05:51:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb48004b20 con 0x7fbb6009bf20 2024-09-16T05:51:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb56ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb5cd99640 4 set_mon_vals no callback set 2024-09-16T05:51:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbb48004d50 con 0x7fbb6009bf20 2024-09-16T05:51:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:38.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:38.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb56ffd640 10 monclient: dump: 2024-09-16T05:51:38.312 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:38.312 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:38.312 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.312 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:38.313 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:38.313 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbb48004f70 con 0x7fbb6009bf20 2024-09-16T05:51:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb64b2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb64b2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb64b2d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb44005090 con 0x7fbb6009bf20 2024-09-16T05:51:38.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fbb48044f30 con 0x7fbb6009bf20 2024-09-16T05:51:38.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.282+0000 7fbb5e59c640 10 monclient: get_auth_request con 0x7fbb40036bb0 auth_method 0 2024-09-16T05:51:38.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.286+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fbb48015560 con 0x7fbb6009bf20 2024-09-16T05:51:38.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.286+0000 7fbb56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:38.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.286+0000 7fbb56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:38.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.514+0000 7fbb64b2d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:51:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.514+0000 7fbb64b2d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:51:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.514+0000 7fbb64b2d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:38.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.514+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fbb44005b90 con 0x7fbb6009bf20 2024-09-16T05:51:38.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb56ffd640 1 -- 172.21.15.6:0/1646648106 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fbb48014e60 con 0x7fbb6009bf20 2024-09-16T05:51:38.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:51:38.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:38.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbb40036bb0 legacy=0x7fbb40039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb64b2d640 10 monclient: shutdown 2024-09-16T05:51:38.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb64b2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:38.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 >> v1:172.21.15.6:6789/0 conn(0x7fbb6009bf20 legacy=0x7fbb60072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:38.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.518+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.518+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=10,logm=52} crc {auth=3938066370,config=3541143443,health=584711527,kv=3130718821,logm=8425301}) 2024-09-16T05:51:38.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.518+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=307451752,config=3541143443,health=1301148488,kv=2361287224,logm=2293244804}) 2024-09-16T05:51:38.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.518+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.518+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=10,logm=52} crc {auth=3938066370,config=3541143443,health=584711527,kv=3130718821,logm=8425301}) 2024-09-16T05:51:38.521 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.518+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=10,kv=11,logm=50} crc {auth=3460750198,config=3541143443,health=3917380090,kv=2361287224,logm=897401934}) 2024-09-16T05:51:38.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 shutdown_connections 2024-09-16T05:51:38.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 >> 172.21.15.6:0/1646648106 conn(0x7fbb600a1650 msgr2=0x7fbb60138f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:38.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 shutdown_connections 2024-09-16T05:51:38.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:38.518+0000 7fbb64b2d640 1 -- 172.21.15.6:0/1646648106 wait complete. 2024-09-16T05:51:38.525 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.522+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.525 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.522+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193431179}) 2024-09-16T05:51:38.525 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.522+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3210407986}) 2024-09-16T05:51:38.525 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.522+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.525 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.522+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193431179}) 2024-09-16T05:51:38.525 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.522+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1448842610}) 2024-09-16T05:51:38.528 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.526+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.528 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.526+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526653420}) 2024-09-16T05:51:38.528 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.526+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1300954464}) 2024-09-16T05:51:38.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.526+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.526+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526653420}) 2024-09-16T05:51:38.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.526+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2938299866}) 2024-09-16T05:51:38.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.530+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.530+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359380050}) 2024-09-16T05:51:38.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.530+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4191195438}) 2024-09-16T05:51:38.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.530+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.530+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359380050}) 2024-09-16T05:51:38.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.530+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=125061901}) 2024-09-16T05:51:38.534 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.534 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159144794}) 2024-09-16T05:51:38.534 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194378971}) 2024-09-16T05:51:38.534 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.534 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159144794}) 2024-09-16T05:51:38.534 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2146751492}) 2024-09-16T05:51:38.536 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:51:38.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119162377}) 2024-09-16T05:51:38.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2424151185}) 2024-09-16T05:51:38.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119162377}) 2024-09-16T05:51:38.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.534+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982649043}) 2024-09-16T05:51:38.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.538+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.538+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2734907844}) 2024-09-16T05:51:38.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.538+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=165382970}) 2024-09-16T05:51:38.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.538+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.538+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2734907844}) 2024-09-16T05:51:38.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.538+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1696691962}) 2024-09-16T05:51:38.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.542+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.542+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435953498}) 2024-09-16T05:51:38.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.542+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3123520854}) 2024-09-16T05:51:38.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.542+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.542+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435953498}) 2024-09-16T05:51:38.543 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.542+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609095058}) 2024-09-16T05:51:38.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3990436222}) 2024-09-16T05:51:38.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2083144945}) 2024-09-16T05:51:38.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3990436222}) 2024-09-16T05:51:38.547 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1927057124}) 2024-09-16T05:51:38.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589757646}) 2024-09-16T05:51:38.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250929727}) 2024-09-16T05:51:38.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589757646}) 2024-09-16T05:51:38.549 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.546+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1779205531}) 2024-09-16T05:51:38.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261304128}) 2024-09-16T05:51:38.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3130095693}) 2024-09-16T05:51:38.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261304128}) 2024-09-16T05:51:38.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=16054020}) 2024-09-16T05:51:38.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622938946}) 2024-09-16T05:51:38.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801928440}) 2024-09-16T05:51:38.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622938946}) 2024-09-16T05:51:38.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.550+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1415335123}) 2024-09-16T05:51:38.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453495110}) 2024-09-16T05:51:38.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536006408}) 2024-09-16T05:51:38.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453495110}) 2024-09-16T05:51:38.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1504044735}) 2024-09-16T05:51:38.555 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.555 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255671772}) 2024-09-16T05:51:38.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779183647}) 2024-09-16T05:51:38.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255671772}) 2024-09-16T05:51:38.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.554+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1965061000}) 2024-09-16T05:51:38.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355649955}) 2024-09-16T05:51:38.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457224523}) 2024-09-16T05:51:38.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355649955}) 2024-09-16T05:51:38.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867520541}) 2024-09-16T05:51:38.559 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792430481}) 2024-09-16T05:51:38.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3501009634}) 2024-09-16T05:51:38.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792430481}) 2024-09-16T05:51:38.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3998173874}) 2024-09-16T05:51:38.561 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.562 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303761495}) 2024-09-16T05:51:38.562 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2177256169}) 2024-09-16T05:51:38.562 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.562 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303761495}) 2024-09-16T05:51:38.562 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.558+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1539451179}) 2024-09-16T05:51:38.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858542206}) 2024-09-16T05:51:38.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=357796077}) 2024-09-16T05:51:38.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858542206}) 2024-09-16T05:51:38.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624937449}) 2024-09-16T05:51:38.565 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.565 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016572403}) 2024-09-16T05:51:38.565 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838577945}) 2024-09-16T05:51:38.565 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.565 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016572403}) 2024-09-16T05:51:38.565 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.562+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1550927021}) 2024-09-16T05:51:38.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097667933}) 2024-09-16T05:51:38.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=665814963}) 2024-09-16T05:51:38.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097667933}) 2024-09-16T05:51:38.567 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=724273788}) 2024-09-16T05:51:38.568 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010266795}) 2024-09-16T05:51:38.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2168607417}) 2024-09-16T05:51:38.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010266795}) 2024-09-16T05:51:38.569 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.566+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787128481}) 2024-09-16T05:51:38.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231682624}) 2024-09-16T05:51:38.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1227187511}) 2024-09-16T05:51:38.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231682624}) 2024-09-16T05:51:38.570 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=257767054}) 2024-09-16T05:51:38.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574380108}) 2024-09-16T05:51:38.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3655283794}) 2024-09-16T05:51:38.573 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.573 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574380108}) 2024-09-16T05:51:38.573 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.570+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734447907}) 2024-09-16T05:51:38.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117047811}) 2024-09-16T05:51:38.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=942090446}) 2024-09-16T05:51:38.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117047811}) 2024-09-16T05:51:38.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196750889}) 2024-09-16T05:51:38.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662610742}) 2024-09-16T05:51:38.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748269827}) 2024-09-16T05:51:38.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662610742}) 2024-09-16T05:51:38.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.574+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=986932380}) 2024-09-16T05:51:38.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893844164}) 2024-09-16T05:51:38.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2337026530}) 2024-09-16T05:51:38.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893844164}) 2024-09-16T05:51:38.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092038616}) 2024-09-16T05:51:38.579 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.580 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167907249}) 2024-09-16T05:51:38.580 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208081233}) 2024-09-16T05:51:38.580 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.580 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167907249}) 2024-09-16T05:51:38.580 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.578+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184204774}) 2024-09-16T05:51:38.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722305261}) 2024-09-16T05:51:38.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238069773}) 2024-09-16T05:51:38.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722305261}) 2024-09-16T05:51:38.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=569711419}) 2024-09-16T05:51:38.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785856357}) 2024-09-16T05:51:38.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2600218760}) 2024-09-16T05:51:38.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785856357}) 2024-09-16T05:51:38.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3108844194}) 2024-09-16T05:51:38.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206751932}) 2024-09-16T05:51:38.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646246698}) 2024-09-16T05:51:38.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206751932}) 2024-09-16T05:51:38.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.582+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=130363588}) 2024-09-16T05:51:38.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519810091}) 2024-09-16T05:51:38.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=205761193}) 2024-09-16T05:51:38.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519810091}) 2024-09-16T05:51:38.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3393503026}) 2024-09-16T05:51:38.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652900327}) 2024-09-16T05:51:38.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550495658}) 2024-09-16T05:51:38.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652900327}) 2024-09-16T05:51:38.590 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.586+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689122409}) 2024-09-16T05:51:38.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3530622016}) 2024-09-16T05:51:38.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837363656}) 2024-09-16T05:51:38.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3530622016}) 2024-09-16T05:51:38.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924366676}) 2024-09-16T05:51:38.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564789358}) 2024-09-16T05:51:38.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4004119001}) 2024-09-16T05:51:38.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564789358}) 2024-09-16T05:51:38.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.590+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461631472}) 2024-09-16T05:51:38.595 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.595 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3508189205}) 2024-09-16T05:51:38.595 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3404273915}) 2024-09-16T05:51:38.595 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.595 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3508189205}) 2024-09-16T05:51:38.595 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244701053}) 2024-09-16T05:51:38.597 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.597 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721833204}) 2024-09-16T05:51:38.597 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=700647833}) 2024-09-16T05:51:38.597 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.597 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721833204}) 2024-09-16T05:51:38.597 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.594+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2104205406}) 2024-09-16T05:51:38.599 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.599 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=96207956}) 2024-09-16T05:51:38.599 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3309283511}) 2024-09-16T05:51:38.599 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.599 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=96207956}) 2024-09-16T05:51:38.599 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3318872562}) 2024-09-16T05:51:38.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206079054}) 2024-09-16T05:51:38.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2519677261}) 2024-09-16T05:51:38.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206079054}) 2024-09-16T05:51:38.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.598+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701934809}) 2024-09-16T05:51:38.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804789191}) 2024-09-16T05:51:38.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1399291006}) 2024-09-16T05:51:38.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804789191}) 2024-09-16T05:51:38.603 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1637176270}) 2024-09-16T05:51:38.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847501197}) 2024-09-16T05:51:38.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751300278}) 2024-09-16T05:51:38.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847501197}) 2024-09-16T05:51:38.605 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.602+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606221835}) 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230233642}) 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=736831285}) 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230233642}) 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3249658648}) 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105325128}) 2024-09-16T05:51:38.609 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596802882}) 2024-09-16T05:51:38.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105325128}) 2024-09-16T05:51:38.610 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.606+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246776235}) 2024-09-16T05:51:38.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775132538}) 2024-09-16T05:51:38.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2898983692}) 2024-09-16T05:51:38.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775132538}) 2024-09-16T05:51:38.611 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=649739932}) 2024-09-16T05:51:38.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190746664}) 2024-09-16T05:51:38.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=308393794}) 2024-09-16T05:51:38.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190746664}) 2024-09-16T05:51:38.613 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.610+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1068337921}) 2024-09-16T05:51:38.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409116074}) 2024-09-16T05:51:38.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488969659}) 2024-09-16T05:51:38.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409116074}) 2024-09-16T05:51:38.615 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2330954087}) 2024-09-16T05:51:38.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572646468}) 2024-09-16T05:51:38.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4108420105}) 2024-09-16T05:51:38.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572646468}) 2024-09-16T05:51:38.617 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.614+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2890477938}) 2024-09-16T05:51:38.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428353653}) 2024-09-16T05:51:38.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823687420}) 2024-09-16T05:51:38.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428353653}) 2024-09-16T05:51:38.619 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061054942}) 2024-09-16T05:51:38.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420582723}) 2024-09-16T05:51:38.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3366152121}) 2024-09-16T05:51:38.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420582723}) 2024-09-16T05:51:38.621 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.618+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302827868}) 2024-09-16T05:51:38.622 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.623 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921712937}) 2024-09-16T05:51:38.623 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=177776241}) 2024-09-16T05:51:38.623 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.623 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921712937}) 2024-09-16T05:51:38.623 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2976595264}) 2024-09-16T05:51:38.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730374822}) 2024-09-16T05:51:38.624 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81612185}) 2024-09-16T05:51:38.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730374822}) 2024-09-16T05:51:38.625 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.622+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1759930037}) 2024-09-16T05:51:38.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=777017563}) 2024-09-16T05:51:38.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387296865}) 2024-09-16T05:51:38.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.626 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=777017563}) 2024-09-16T05:51:38.627 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3920549413}) 2024-09-16T05:51:38.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780458219}) 2024-09-16T05:51:38.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322720503}) 2024-09-16T05:51:38.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780458219}) 2024-09-16T05:51:38.628 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.626+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1063461314}) 2024-09-16T05:51:38.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363317512}) 2024-09-16T05:51:38.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3152865235}) 2024-09-16T05:51:38.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363317512}) 2024-09-16T05:51:38.630 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760509316}) 2024-09-16T05:51:38.631 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3581323339}) 2024-09-16T05:51:38.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585995501}) 2024-09-16T05:51:38.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3581323339}) 2024-09-16T05:51:38.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.630+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458703057}) 2024-09-16T05:51:38.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545637386}) 2024-09-16T05:51:38.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=575746844}) 2024-09-16T05:51:38.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.636 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545637386}) 2024-09-16T05:51:38.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448950315}) 2024-09-16T05:51:38.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796765092}) 2024-09-16T05:51:38.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=594327773}) 2024-09-16T05:51:38.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796765092}) 2024-09-16T05:51:38.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.634+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072845165}) 2024-09-16T05:51:38.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004696400}) 2024-09-16T05:51:38.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968343317}) 2024-09-16T05:51:38.638 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004696400}) 2024-09-16T05:51:38.639 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2855936857}) 2024-09-16T05:51:38.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674868206}) 2024-09-16T05:51:38.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3903445088}) 2024-09-16T05:51:38.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674868206}) 2024-09-16T05:51:38.640 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.638+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411363323}) 2024-09-16T05:51:38.642 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632695521}) 2024-09-16T05:51:38.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3427842712}) 2024-09-16T05:51:38.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632695521}) 2024-09-16T05:51:38.643 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3492826122}) 2024-09-16T05:51:38.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=3955852106,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=4150644442,mon_config_key=4074902064,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=2182199915}) 2024-09-16T05:51:38.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=4} crc {logm=2777515172,mdsmap=2670817759,mgr=1552851386,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=4159795391,mon_config_key=782542404,monmap=3625042558,osd_metadata=1245031373,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=4188563455}) 2024-09-16T05:51:38.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=5} crc {logm=3955852106,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=4150644442,mon_config_key=4074902064,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=2182199915}) 2024-09-16T05:51:38.645 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.642+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=3} crc {logm=2940194077,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3579548406,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1478793609}) 2024-09-16T05:51:38.653 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.654+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.653 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.654+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=1313791104}) 2024-09-16T05:51:38.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.654+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=48} crc {osdmap=1357300049}) 2024-09-16T05:51:38.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.654+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:51:38.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.654+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=1313791104}) 2024-09-16T05:51:38.654 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:51:38.654+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=47} crc {osdmap=2036107347}) 2024-09-16T05:51:39.537 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:51:39.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:51:39.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.666+0000 7f11461a5640 10 monclient: get_monmap_and_config 2024-09-16T05:51:39.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.666+0000 7f11461a5640 10 monclient: build_initial_monmap 2024-09-16T05:51:39.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.666+0000 7f11461a5640 10 monclient: monmap: 2024-09-16T05:51:39.668 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:39.668 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:39.669 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:39.669860+0000 2024-09-16T05:51:39.669 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:39.669860+0000 2024-09-16T05:51:39.669 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:39.669 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:39.669 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:39.669 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:39.669 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:39.669 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:39.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.666+0000 7f11461a5640 1 Processor -- start 2024-09-16T05:51:39.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.666+0000 7f11461a5640 1 -- start start 2024-09-16T05:51:39.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.666+0000 7f11461a5640 10 monclient: init 2024-09-16T05:51:39.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:39.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:51:39.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient(hunting): picked mon.noname-c con 0x7f114009b2f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:39.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient(hunting): picked mon.noname-a con 0x7f1140136b70 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:39.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient(hunting): picked mon.noname-b con 0x7f114009b6d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11400a19c0 con 0x7f1140136b70 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1140099ae0 con 0x7f114009b2f0 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1140099dc0 con 0x7f114009b6d0 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient(hunting): authenticate will time out at 1961.476701s 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f114009b6d0 0x7f114013f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42176/0 (socket says 172.21.15.6:42176) 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113ffff640 1 -- 172.21.15.6:0/642808257 learned_addr learned my addr 172.21.15.6:0/642808257 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2146773153 0 0) 0x7f1140099dc0 con 0x7f114009b6d0 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:39.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): my global_id is 164228 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f112c003530 con 0x7f114009b6d0 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1014194703 0 0) 0x7f11400a19c0 con 0x7f1140136b70 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): my global_id is 174129 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1140099dc0 con 0x7f1140136b70 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401836910 0 0) 0x7f1140099ae0 con 0x7f114009b2f0 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient(hunting): my global_id is 154381 2024-09-16T05:51:39.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11400a19c0 con 0x7f114009b2f0 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2887687129 0 0) 0x7f112c003530 con 0x7f114009b6d0 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 >> v1:172.21.15.63:6789/0 conn(0x7f114009b2f0 legacy=0x7f11400a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 >> v1:172.21.15.6:6789/0 conn(0x7f1140136b70 legacy=0x7f11400a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 monclient: found mon.noname-b 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114009bae0 con 0x7f114009b6d0 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 5 monclient: authenticate success, global_id 164228 2024-09-16T05:51:39.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1138001550 con 0x7f114009b6d0 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: dump: 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:39.679 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11380017b0 con 0x7f114009b6d0 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 1 -- 172.21.15.6:0/642808257 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1138001bc0 con 0x7f114009b6d0 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 4 set_mon_vals no callback set 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f113e7fc640 10 monclient: dump: 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:39.680 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: shutdown 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- 172.21.15.6:0/642808257 >> v1:172.21.15.6:6790/0 conn(0x7f114009b6d0 legacy=0x7f114013f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- 172.21.15.6:0/642808257 shutdown_connections 2024-09-16T05:51:39.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- 172.21.15.6:0/642808257 >> 172.21.15.6:0/642808257 conn(0x7f11400a1c90 msgr2=0x7f11400a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- 172.21.15.6:0/642808257 shutdown_connections 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- 172.21.15.6:0/642808257 wait complete. 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: build_initial_monmap 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: monmap: 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:39.682 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 Processor -- start 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- start start 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: init 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient(hunting): picked mon.noname-b con 0x7f114009b2f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient(hunting): picked mon.noname-a con 0x7f114009b6d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 10 monclient(hunting): picked mon.noname-c con 0x7f1140136b70 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1140131d40 con 0x7f114009b6d0 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.670+0000 7f11461a5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11400a3dd0 con 0x7f114009b2f0 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11400a3aa0 con 0x7f1140136b70 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:39.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 monclient(hunting): authenticate will time out at 1961.480030s 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f113effd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f114009b6d0 0x7f11401d0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34098/0 (socket says 172.21.15.6:34098) 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f113effd640 1 -- 172.21.15.6:0/1730942452 learned_addr learned my addr 172.21.15.6:0/1730942452 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2901161835 0 0) 0x7f11400a3aa0 con 0x7f1140136b70 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _negotiate 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): my global_id is 164231 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1120003530 con 0x7f1140136b70 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764660147 0 0) 0x7f1140131d40 con 0x7f114009b6d0 2024-09-16T05:51:39.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _negotiate 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): my global_id is 174132 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11400a3aa0 con 0x7f114009b6d0 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3097234660 0 0) 0x7f11400a3aa0 con 0x7f114009b6d0 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1140131d40 con 0x7f114009b6d0 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1128001bf0 con 0x7f114009b6d0 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:39.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): dump: 2024-09-16T05:51:39.686 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:39.686 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:39.686 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.686 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.686 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:39.686 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:39.686 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:39.686 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:39.687 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:39.687 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:39.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:39.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 157773379 0 0) 0x7f11400a3dd0 con 0x7f114009b2f0 2024-09-16T05:51:39.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _negotiate 2024-09-16T05:51:39.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): my global_id is 154384 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11400a3aa0 con 0x7f114009b2f0 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1253865718 0 0) 0x7f1120003530 con 0x7f1140136b70 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11400a3dd0 con 0x7f1140136b70 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1138001b40 con 0x7f1140136b70 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient(hunting): dump: 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1212731444 0 0) 0x7f1140131d40 con 0x7f114009b6d0 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 >> v1:172.21.15.6:6790/0 conn(0x7f1140136b70 legacy=0x7f11401d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:39.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 >> v1:172.21.15.63:6789/0 conn(0x7f114009b2f0 legacy=0x7f11401cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient: _finish_hunting 0 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 monclient: found mon.a 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114009c380 con 0x7f114009b6d0 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient: _finish_auth 0 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient: _check_auth_tickets 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 5 monclient: authenticate success, global_id 174132 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 monclient: _renew_subs 2024-09-16T05:51:39.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:39.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f114012fd30 con 0x7f114009b6d0 2024-09-16T05:51:39.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 monclient: _renew_subs 2024-09-16T05:51:39.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:39.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1140130a60 con 0x7f114009b6d0 2024-09-16T05:51:39.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1128003980 con 0x7f114009b6d0 2024-09-16T05:51:39.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:39.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1128003cd0 con 0x7f114009b6d0 2024-09-16T05:51:39.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f113dffb640 4 set_mon_vals no callback set 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 10 monclient: dump: 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:39.692 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1128003e70 con 0x7f114009b6d0 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1124005090 con 0x7f114009b6d0 2024-09-16T05:51:39.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f11280450d0 con 0x7f114009b6d0 2024-09-16T05:51:39.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.674+0000 7f113f7fe640 10 monclient: get_auth_request con 0x7f1120036d10 auth_method 0 2024-09-16T05:51:39.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.678+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1140130a60 con 0x7f114009b6d0 2024-09-16T05:51:39.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.678+0000 7f1137fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:39.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.678+0000 7f1137fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.902+0000 7f11461a5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:51:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.902+0000 7f11461a5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.902+0000 7f11461a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:39.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.902+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1124005380 con 0x7f114009b6d0 2024-09-16T05:51:39.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.902+0000 7f1137fff640 1 -- 172.21.15.6:0/1730942452 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 925175510) 0x7f1128015700 con 0x7f114009b6d0 2024-09-16T05:51:39.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.902+0000 7f1137fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:39.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.902+0000 7f1137fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:39.903 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:51:39.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.906+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1120036d10 legacy=0x7f11200391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:39.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.906+0000 7f11461a5640 10 monclient: shutdown 2024-09-16T05:51:39.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.906+0000 7f11461a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:39.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.906+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 >> v1:172.21.15.6:6789/0 conn(0x7f114009b6d0 legacy=0x7f11401d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:39.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.906+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 shutdown_connections 2024-09-16T05:51:39.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.906+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 >> 172.21.15.6:0/1730942452 conn(0x7f11400a1c90 msgr2=0x7f11400a0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:39.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.906+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 shutdown_connections 2024-09-16T05:51:39.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:39.906+0000 7f11461a5640 1 -- 172.21.15.6:0/1730942452 wait complete. 2024-09-16T05:51:39.924 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:51:39.924 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:51:39.924 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:51:39.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:51:40.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient: get_monmap_and_config 2024-09-16T05:51:40.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient: build_initial_monmap 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient: monmap: 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:40.047216+0000 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:40.047216+0000 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:40.046 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 Processor -- start 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- start start 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient: init 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient(hunting): picked mon.noname-b con 0x7f097006dd90 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient(hunting): picked mon.noname-c con 0x7f09700a3d60 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient(hunting): picked mon.noname-a con 0x7f097013d110 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09700a41b0 con 0x7f097013d110 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097009ff10 con 0x7f09700a3d60 2024-09-16T05:51:40.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09700a4c80 con 0x7f097006dd90 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient(hunting): authenticate will time out at 1961.853799s 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f097006dd90 0x7f09700a2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42202/0 (socket says 172.21.15.6:42202) 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096ffff640 1 -- 172.21.15.6:0/737936471 learned_addr learned my addr 172.21.15.6:0/737936471 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3430148689 0 0) 0x7f09700a4c80 con 0x7f097006dd90 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient(hunting): my global_id is 164234 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0958003530 con 0x7f097006dd90 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2084082925 0 0) 0x7f09700a41b0 con 0x7f097013d110 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient(hunting): my global_id is 174135 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09700a4c80 con 0x7f097013d110 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 118119905 0 0) 0x7f0958003530 con 0x7f097006dd90 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 >> v1:172.21.15.63:6789/0 conn(0x7f09700a3d60 legacy=0x7f097013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 >> v1:172.21.15.6:6789/0 conn(0x7f097013d110 legacy=0x7f097013f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: _finish_hunting 0 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 monclient: found mon.noname-b 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097006e950 con 0x7f097006dd90 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: _finish_auth 0 2024-09-16T05:51:40.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: _check_auth_tickets 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f09680019a0 con 0x7f097006dd90 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: dump: 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.058 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 5 monclient: authenticate success, global_id 164234 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0968001cc0 con 0x7f097006dd90 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 4 set_mon_vals no callback set 2024-09-16T05:51:40.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 1 -- 172.21.15.6:0/737936471 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0968002090 con 0x7f097006dd90 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f096effd640 10 monclient: dump: 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.060 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 10 monclient: shutdown 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- 172.21.15.6:0/737936471 >> v1:172.21.15.6:6790/0 conn(0x7f097006dd90 legacy=0x7f09700a2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- 172.21.15.6:0/737936471 shutdown_connections 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- 172.21.15.6:0/737936471 >> 172.21.15.6:0/737936471 conn(0x7f09700a59a0 msgr2=0x7f09700a5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- 172.21.15.6:0/737936471 shutdown_connections 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.046+0000 7f097643a640 1 -- 172.21.15.6:0/737936471 wait complete. 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: build_initial_monmap 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: monmap: 2024-09-16T05:51:40.061 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 1 Processor -- start 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 1 -- start start 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: init 2024-09-16T05:51:40.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient(hunting): picked mon.noname-a con 0x7f097006dd90 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient(hunting): picked mon.noname-c con 0x7f09700a3d60 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient(hunting): picked mon.noname-b con 0x7f097013d110 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0970127cd0 con 0x7f097006dd90 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09700a0100 con 0x7f097013d110 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097009edb0 con 0x7f09700a3d60 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient(hunting): authenticate will time out at 1961.857026s 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f097006dd90 0x7f097013c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34116/0 (socket says 172.21.15.6:34116) 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096ffff640 1 -- 172.21.15.6:0/949302924 learned_addr learned my addr 172.21.15.6:0/949302924 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767253395 0 0) 0x7f097009edb0 con 0x7f09700a3d60 2024-09-16T05:51:40.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): my global_id is 164237 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f095c003530 con 0x7f09700a3d60 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 634287746 0 0) 0x7f0970127cd0 con 0x7f097006dd90 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): my global_id is 174138 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f097009edb0 con 0x7f097006dd90 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 765177493 0 0) 0x7f095c003530 con 0x7f09700a3d60 2024-09-16T05:51:40.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0970127cd0 con 0x7f09700a3d60 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f09640017f0 con 0x7f09700a3d60 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): dump: 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888913012 0 0) 0x7f09700a0100 con 0x7f097013d110 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): my global_id is 154390 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f095c003530 con 0x7f097013d110 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 636092576 0 0) 0x7f097009edb0 con 0x7f097006dd90 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09700a0100 con 0x7f097006dd90 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0968001960 con 0x7f097006dd90 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:40.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): dump: 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 870896854 0 0) 0x7f0970127cd0 con 0x7f09700a3d60 2024-09-16T05:51:40.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 >> v1:172.21.15.63:6789/0 conn(0x7f097013d110 legacy=0x7f09701d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 >> v1:172.21.15.6:6789/0 conn(0x7f097006dd90 legacy=0x7f097013c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 monclient: found mon.c 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09700a8680 con 0x7f09700a3d60 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: _finish_auth 0 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 5 monclient: authenticate success, global_id 164237 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: _renew_subs 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:40.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09701259e0 con 0x7f09700a3d60 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: _renew_subs 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0970126760 con 0x7f09700a3d60 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0964003600 con 0x7f09700a3d60 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0964003950 con 0x7f09700a3d60 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096dffb640 4 set_mon_vals no callback set 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: dump: 2024-09-16T05:51:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0964003b30 con 0x7f09700a3d60 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f09500065d0 con 0x7f0950002870 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f0964043b80 con 0x7f09700a3d60 2024-09-16T05:51:40.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036959433 0 0) 0x7f09500065d0 con 0x7f0950002870 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient(hunting): my global_id is 174141 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0970127cd0 con 0x7f0950002870 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f09749d7640 10 monclient: get_auth_request con 0x7f095c036c40 auth_method 0 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1735008708 0 0) 0x7f0970127cd0 con 0x7f0950002870 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09500065d0 con 0x7f0950002870 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0968004f90 con 0x7f0950002870 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:40.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 10 monclient: dump: 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1815861675 0 0) 0x7f09500065d0 con 0x7f0950002870 2024-09-16T05:51:40.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.050+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f095000a160 con 0x7f0950002870 2024-09-16T05:51:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.054+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0968007360 con 0x7f0950002870 2024-09-16T05:51:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.054+0000 7f096cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.054+0000 7f096cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.054+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 >> v1:172.21.15.6:6789/0 conn(0x7f0950002870 legacy=0x7f0950004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.054+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 >> v1:172.21.15.6:6789/0 conn(0x7f0950002870 legacy=0x7f0950004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:40.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:51:40.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0950009e50 con 0x7f0950009110 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3615633944 0 0) 0x7f0950009e50 con 0x7f0950009110 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient(hunting): my global_id is 174144 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09500065d0 con 0x7f0950009110 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3021152764 0 0) 0x7f09500065d0 con 0x7f0950009110 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0950009e50 con 0x7f0950009110 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0968005790 con 0x7f0950009110 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient: dump: 2024-09-16T05:51:40.075 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3124402274 0 0) 0x7f0950009e50 con 0x7f0950009110 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f095000c3e0 con 0x7f0950009110 2024-09-16T05:51:40.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3962 (unknown 0 0 3112084993) 0x7f0968006ac0 con 0x7f0950009110 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 >> v1:172.21.15.6:6789/0 conn(0x7f0950009110 legacy=0x7f09500095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f096cff9640 1 -- 172.21.15.6:0/949302924 >> v1:172.21.15.6:6789/0 conn(0x7f0950009110 legacy=0x7f09500095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 70, 2024-09-16T05:51:40.077 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 2731, 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.078 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.079 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:51:40.080 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:51:40.081 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.082 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.083 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.087 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.088 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.089 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:51:40.089 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:51:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 >> v1:172.21.15.63:6812/3643207211 conn(0x7f095c036c40 legacy=0x7f095c039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 10 monclient: shutdown 2024-09-16T05:51:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 >> v1:172.21.15.6:6790/0 conn(0x7f09700a3d60 legacy=0x7f09701d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f09749d7640 1 -- 172.21.15.6:0/949302924 reap_dead start 2024-09-16T05:51:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f09749d7640 1 -- 172.21.15.6:0/949302924 reap_dead start 2024-09-16T05:51:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 shutdown_connections 2024-09-16T05:51:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 >> 172.21.15.6:0/949302924 conn(0x7f09700a59a0 msgr2=0x7f097009d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 shutdown_connections 2024-09-16T05:51:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.058+0000 7f097643a640 1 -- 172.21.15.6:0/949302924 wait complete. 2024-09-16T05:51:40.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:51:40.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:51:40.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:51:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient: get_monmap_and_config 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient: build_initial_monmap 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient: monmap: 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:40.207396+0000 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:40.207396+0000 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:40.207 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 Processor -- start 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- start start 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient: init 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient(hunting): picked mon.noname-b con 0x7f482409bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient(hunting): picked mon.noname-c con 0x7f482409bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient(hunting): picked mon.noname-a con 0x7f48241397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48240a1350 con 0x7f48241397f0 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48240a10d0 con 0x7f482409bf20 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482409d940 con 0x7f482409bb40 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient(hunting): authenticate will time out at 1962.014054s 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f4828a44640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f482409bb40 0x7f482409e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42224/0 (socket says 172.21.15.6:42224) 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f4829245640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f48241397f0 0x7f482413bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34156/0 (socket says 172.21.15.6:34156) 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f4828a44640 1 -- 172.21.15.6:0/1446346817 learned_addr learned my addr 172.21.15.6:0/1446346817 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626920881 0 0) 0x7f482409d940 con 0x7f482409bb40 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): my global_id is 164240 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f480c005740 con 0x7f482409bb40 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1685941713 0 0) 0x7f48240a1350 con 0x7f48241397f0 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): my global_id is 174147 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f482409d940 con 0x7f48241397f0 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3493112338 0 0) 0x7f48240a10d0 con 0x7f482409bf20 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient(hunting): my global_id is 154393 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48240a1350 con 0x7f482409bf20 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3206214500 0 0) 0x7f482409d940 con 0x7f48241397f0 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 >> v1:172.21.15.6:6790/0 conn(0x7f482409bb40 legacy=0x7f482409e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 >> v1:172.21.15.63:6789/0 conn(0x7f482409bf20 legacy=0x7f482409c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:40.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 monclient: found mon.noname-a 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482409dc20 con 0x7f48241397f0 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f481c001980 con 0x7f48241397f0 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: dump: 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.217 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 5 monclient: authenticate success, global_id 174147 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f481c001c70 con 0x7f48241397f0 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 1 -- 172.21.15.6:0/1446346817 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f481c002040 con 0x7f48241397f0 2024-09-16T05:51:40.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f48237fe640 10 monclient: dump: 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:40.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 4 set_mon_vals no callback set 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 10 monclient: shutdown 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- 172.21.15.6:0/1446346817 >> v1:172.21.15.6:6789/0 conn(0x7f48241397f0 legacy=0x7f482413bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- 172.21.15.6:0/1446346817 shutdown_connections 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- 172.21.15.6:0/1446346817 >> 172.21.15.6:0/1446346817 conn(0x7f48240a1650 msgr2=0x7f48240a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- 172.21.15.6:0/1446346817 shutdown_connections 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.206+0000 7f482aca8640 1 -- 172.21.15.6:0/1446346817 wait complete. 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: build_initial_monmap 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: monmap: 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:40.220 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 1 Processor -- start 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 1 -- start start 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: init 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient(hunting): picked mon.noname-c con 0x7f482409bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:40.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient(hunting): picked mon.noname-b con 0x7f482409bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient(hunting): picked mon.noname-a con 0x7f48241397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482409b8f0 con 0x7f48241397f0 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48240740c0 con 0x7f482409bf20 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4824130140 con 0x7f482409bb40 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient(hunting): authenticate will time out at 1962.017387s 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f4829245640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f48241397f0 0x7f48241d3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34168/0 (socket says 172.21.15.6:34168) 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f4829245640 1 -- 172.21.15.6:0/2517999006 learned_addr learned my addr 172.21.15.6:0/2517999006 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951454677 0 0) 0x7f482409b8f0 con 0x7f48241397f0 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): my global_id is 174150 2024-09-16T05:51:40.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4810003530 con 0x7f48241397f0 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64099576 0 0) 0x7f4824130140 con 0x7f482409bb40 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): my global_id is 164243 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f482409b8f0 con 0x7f482409bb40 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3251987528 0 0) 0x7f4810003530 con 0x7f48241397f0 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4824130140 con 0x7f48241397f0 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 403519249 0 0) 0x7f482409b8f0 con 0x7f482409bb40 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4810003530 con 0x7f482409bb40 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897462462 0 0) 0x7f48240740c0 con 0x7f482409bf20 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): my global_id is 154396 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f482409b8f0 con 0x7f482409bf20 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f481c001940 con 0x7f48241397f0 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): dump: 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.224 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4814001920 con 0x7f482409bb40 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:40.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): dump: 2024-09-16T05:51:40.226 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.226 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.226 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.226 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.226 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.226 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2877879410 0 0) 0x7f4824130140 con 0x7f48241397f0 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 >> v1:172.21.15.6:6790/0 conn(0x7f482409bb40 legacy=0x7f482412fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 >> v1:172.21.15.63:6789/0 conn(0x7f482409bf20 legacy=0x7f48241d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient: _finish_hunting 0 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 monclient: found mon.a 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482411cd30 con 0x7f48241397f0 2024-09-16T05:51:40.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient: _finish_auth 0 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient: _check_auth_tickets 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 5 monclient: authenticate success, global_id 174150 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: _renew_subs 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f482411e420 con 0x7f48241397f0 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: _renew_subs 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f48241c7fb0 con 0x7f48241397f0 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f481c001c70 con 0x7f48241397f0 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:40.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f481c004f00 con 0x7f48241397f0 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient: dump: 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.229 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.230 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.230 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.230 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.230 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f4822ffd640 4 set_mon_vals no callback set 2024-09-16T05:51:40.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f481c0050a0 con 0x7f48241397f0 2024-09-16T05:51:40.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48040065d0 con 0x7f4804002870 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f481c0451e0 con 0x7f48241397f0 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409656034 0 0) 0x7f48040065d0 con 0x7f4804002870 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 10 monclient(hunting): my global_id is 164246 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4824130140 con 0x7f4804002870 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.210+0000 7f4823fff640 10 monclient: get_auth_request con 0x7f4810036ba0 auth_method 0 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4248594253 0 0) 0x7f4824130140 con 0x7f4804002870 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48040065d0 con 0x7f4804002870 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4814004b80 con 0x7f4804002870 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 10 monclient: dump: 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.232 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1699424660 0 0) 0x7f48040065d0 con 0x7f4804002870 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f480400a160 con 0x7f4804002870 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f48140060c0 con 0x7f4804002870 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 >> v1:172.21.15.6:6790/0 conn(0x7f4804002870 legacy=0x7f4804004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.214+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 >> v1:172.21.15.6:6790/0 conn(0x7f4804002870 legacy=0x7f4804004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f482aca8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4804009e10 con 0x7f48040090f0 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1161062122 0 0) 0x7f4804009e10 con 0x7f48040090f0 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient(hunting): my global_id is 164249 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48040065d0 con 0x7f48040090f0 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3576123678 0 0) 0x7f48040065d0 con 0x7f48040090f0 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4804009e10 con 0x7f48040090f0 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4814006810 con 0x7f48040090f0 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient: dump: 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 949907357 0 0) 0x7f4804009e10 con 0x7f48040090f0 2024-09-16T05:51:40.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f480400c390 con 0x7f48040090f0 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 1485319617) 0x7f4814006ac0 con 0x7f48040090f0 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 >> v1:172.21.15.6:6790/0 conn(0x7f48040090f0 legacy=0x7f4804009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.218+0000 7f48217fa640 1 -- 172.21.15.6:0/2517999006 >> v1:172.21.15.6:6790/0 conn(0x7f48040090f0 legacy=0x7f4804009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:40.237 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 70, 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 111178, 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:51:40.238 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.239 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.240 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:51:40.241 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:51:40.242 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:51:40.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.245 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.246 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:51:40.246 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.246 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.246 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.246 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.247 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.248 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.249 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.250 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.251 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.252 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:40.252 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.252 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.252 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:51:40.252 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:51:40.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 >> v1:172.21.15.63:6812/3643207211 conn(0x7f4810036ba0 legacy=0x7f4810039060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f482aca8640 10 monclient: shutdown 2024-09-16T05:51:40.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f482aca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:40.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 >> v1:172.21.15.6:6789/0 conn(0x7f48241397f0 legacy=0x7f48241d3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f4829245640 1 -- 172.21.15.6:0/2517999006 reap_dead start 2024-09-16T05:51:40.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f4829245640 1 -- 172.21.15.6:0/2517999006 reap_dead start 2024-09-16T05:51:40.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 shutdown_connections 2024-09-16T05:51:40.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 >> 172.21.15.6:0/2517999006 conn(0x7f48240a1650 msgr2=0x7f48240a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:40.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 shutdown_connections 2024-09-16T05:51:40.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.222+0000 7f482aca8640 1 -- 172.21.15.6:0/2517999006 wait complete. 2024-09-16T05:51:40.254 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:51:40.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:51:40.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient: get_monmap_and_config 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient: build_initial_monmap 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient: monmap: 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:40.371520+0000 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:40.371520+0000 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:40.371 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 Processor -- start 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- start start 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient: init 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient(hunting): picked mon.noname-b con 0x7f52a809aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient(hunting): picked mon.noname-a con 0x7f52a809e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient(hunting): picked mon.noname-c con 0x7f52a80a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a80a1370 con 0x7f52a809e230 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a809ead0 con 0x7f52a80a0da0 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a809edb0 con 0x7f52a809aa30 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient(hunting): authenticate will time out at 1962.177927s 2024-09-16T05:51:40.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52ac9b3640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f52a809aa30 0x7f52a809ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42260/0 (socket says 172.21.15.6:42260) 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52ac9b3640 1 -- 172.21.15.6:0/2500068212 learned_addr learned my addr 172.21.15.6:0/2500068212 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814799795 0 0) 0x7f52a809edb0 con 0x7f52a809aa30 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient(hunting): my global_id is 164252 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5290003530 con 0x7f52a809aa30 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701653864 0 0) 0x7f52a80a1370 con 0x7f52a809e230 2024-09-16T05:51:40.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient(hunting): my global_id is 174153 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52a809edb0 con 0x7f52a809e230 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 122894673 0 0) 0x7f5290003530 con 0x7f52a809aa30 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 >> v1:172.21.15.63:6789/0 conn(0x7f52a80a0da0 legacy=0x7f52a813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 >> v1:172.21.15.6:6789/0 conn(0x7f52a809e230 legacy=0x7f52a81399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 monclient: found mon.noname-b 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:40.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a809dd50 con 0x7f52a809aa30 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5298001620 con 0x7f52a809aa30 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: dump: 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.381 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 5 monclient: authenticate success, global_id 164252 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52980018d0 con 0x7f52a809aa30 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 1 -- 172.21.15.6:0/2500068212 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5298001ce0 con 0x7f52a809aa30 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 4 set_mon_vals no callback set 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52a77fe640 10 monclient: dump: 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 10 monclient: shutdown 2024-09-16T05:51:40.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- 172.21.15.6:0/2500068212 >> v1:172.21.15.6:6790/0 conn(0x7f52a809aa30 legacy=0x7f52a809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- 172.21.15.6:0/2500068212 shutdown_connections 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- 172.21.15.6:0/2500068212 >> 172.21.15.6:0/2500068212 conn(0x7f52a80a1650 msgr2=0x7f52a80a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- 172.21.15.6:0/2500068212 shutdown_connections 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.370+0000 7f52aec17640 1 -- 172.21.15.6:0/2500068212 wait complete. 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: build_initial_monmap 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: monmap: 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:40.384 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 1 Processor -- start 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 1 -- start start 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: init 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient(hunting): picked mon.noname-a con 0x7f52a809aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient(hunting): picked mon.noname-b con 0x7f52a809e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient(hunting): picked mon.noname-c con 0x7f52a80a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:40.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a8137c60 con 0x7f52a809aa30 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a813d8c0 con 0x7f52a809e230 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a8137980 con 0x7f52a80a0da0 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient(hunting): authenticate will time out at 1962.181005s 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52ac9b3640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f52a809aa30 0x7f52a813d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34188/0 (socket says 172.21.15.6:34188) 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52ad1b4640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f52a80a0da0 0x7f52a81d1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42268/0 (socket says 172.21.15.6:42268) 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52ac9b3640 1 -- 172.21.15.6:0/701389620 learned_addr learned my addr 172.21.15.6:0/701389620 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516906216 0 0) 0x7f52a8137980 con 0x7f52a80a0da0 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): my global_id is 164255 2024-09-16T05:51:40.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5294003560 con 0x7f52a80a0da0 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237578727 0 0) 0x7f52a8137c60 con 0x7f52a809aa30 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): my global_id is 174156 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52a8137980 con 0x7f52a809aa30 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3518666373 0 0) 0x7f5294003560 con 0x7f52a80a0da0 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52a8137c60 con 0x7f52a80a0da0 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52a0001bc0 con 0x7f52a80a0da0 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): dump: 2024-09-16T05:51:40.387 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2647663854 0 0) 0x7f52a8137980 con 0x7f52a809aa30 2024-09-16T05:51:40.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5294003560 con 0x7f52a809aa30 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5298001c20 con 0x7f52a809aa30 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): dump: 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.389 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2999609324 0 0) 0x7f52a813d8c0 con 0x7f52a809e230 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): my global_id is 154402 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52a8137980 con 0x7f52a809e230 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4172701107 0 0) 0x7f52a8137c60 con 0x7f52a80a0da0 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 >> v1:172.21.15.63:6789/0 conn(0x7f52a809e230 legacy=0x7f52a81ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 >> v1:172.21.15.6:6789/0 conn(0x7f52a809aa30 legacy=0x7f52a813d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: _finish_hunting 0 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 monclient: found mon.c 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:40.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a81364d0 con 0x7f52a80a0da0 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: _finish_auth 0 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: _check_auth_tickets 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 5 monclient: authenticate success, global_id 164255 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: _renew_subs 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52a81c8ce0 con 0x7f52a80a0da0 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: _renew_subs 2024-09-16T05:51:40.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f52a81d2eb0 con 0x7f52a80a0da0 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f52a00039a0 con 0x7f52a80a0da0 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52a0003d70 con 0x7f52a80a0da0 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a6ffd640 4 set_mon_vals no callback set 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: dump: 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.392 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f52880065d0 con 0x7f5288002870 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f52a0003f10 con 0x7f52a80a0da0 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f52a0045030 con 0x7f52a80a0da0 2024-09-16T05:51:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a7fff640 10 monclient: get_auth_request con 0x7f5294036c20 auth_method 0 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2109768350 0 0) 0x7f52880065d0 con 0x7f5288002870 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient(hunting): my global_id is 154405 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52a8137c60 con 0x7f5288002870 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2039020322 0 0) 0x7f52a8137c60 con 0x7f5288002870 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52880065d0 con 0x7f5288002870 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f52980049e0 con 0x7f5288002870 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.374+0000 7f52a57fa640 10 monclient: dump: 2024-09-16T05:51:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.378+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 724019881 0 0) 0x7f52880065d0 con 0x7f5288002870 2024-09-16T05:51:40.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.378+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f528800a160 con 0x7f5288002870 2024-09-16T05:51:40.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.378+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5298004d10 con 0x7f5288002870 2024-09-16T05:51:40.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.378+0000 7f52a57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.378+0000 7f52a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:40.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.378+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 >> v1:172.21.15.63:6789/0 conn(0x7f5288002870 legacy=0x7f5288004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.378+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 >> v1:172.21.15.63:6789/0 conn(0x7f5288002870 legacy=0x7f5288004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:40.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.382+0000 7f52aec17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:51:40.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.382+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5288009e30 con 0x7f52880090f0 2024-09-16T05:51:40.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.382+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717414653 0 0) 0x7f5288009e30 con 0x7f52880090f0 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.382+0000 7f52a57fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.382+0000 7f52a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.382+0000 7f52a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.382+0000 7f52a57fa640 10 monclient(hunting): my global_id is 154408 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.382+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52880065d0 con 0x7f52880090f0 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3291474741 0 0) 0x7f52880065d0 con 0x7f52880090f0 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5288009e30 con 0x7f52880090f0 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5298005290 con 0x7f52880090f0 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:51:40.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 10 monclient: dump: 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 106556613 0 0) 0x7f5288009e30 con 0x7f52880090f0 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f528800c3a0 con 0x7f52880090f0 2024-09-16T05:51:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 2400933565) 0x7f52980065c0 con 0x7f52880090f0 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 >> v1:172.21.15.63:6789/0 conn(0x7f52880090f0 legacy=0x7f5288009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52a57fa640 1 -- 172.21.15.6:0/701389620 >> v1:172.21.15.63:6789/0 conn(0x7f52880090f0 legacy=0x7f5288009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:51:40.400 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 70, 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 235125, 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.401 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.402 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:51:40.403 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:51:40.404 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:40.405 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.406 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.407 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:51:40.409 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:40.410 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:51:40.411 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:40.412 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:40.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:40.412 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:51:40.412 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:51:40.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5294036c20 legacy=0x7f52940390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52aec17640 10 monclient: shutdown 2024-09-16T05:51:40.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52aec17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:40.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 >> v1:172.21.15.6:6790/0 conn(0x7f52a80a0da0 legacy=0x7f52a81d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52ad1b4640 1 -- 172.21.15.6:0/701389620 reap_dead start 2024-09-16T05:51:40.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52ad1b4640 1 -- 172.21.15.6:0/701389620 reap_dead start 2024-09-16T05:51:40.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 shutdown_connections 2024-09-16T05:51:40.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 >> 172.21.15.6:0/701389620 conn(0x7f52a80a1650 msgr2=0x7f52a8071890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:40.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 shutdown_connections 2024-09-16T05:51:40.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.386+0000 7f52aec17640 1 -- 172.21.15.6:0/701389620 wait complete. 2024-09-16T05:51:40.414 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:51:40.414 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T05:51:40.414 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T05:51:40.414 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T05:51:40.414 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:51:40.415 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:51:40.415 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:51:40.436 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:51:40.436 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:51:40.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:51:40.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient: get_monmap_and_config 2024-09-16T05:51:40.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient: build_initial_monmap 2024-09-16T05:51:40.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient: monmap: 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:40.547951+0000 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:40.547951+0000 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:40.547 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 1 Processor -- start 2024-09-16T05:51:40.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 1 -- start start 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient: init 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient(hunting): picked mon.noname-c con 0x7f100c0a0df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient(hunting): picked mon.noname-b con 0x7f100c135330 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient(hunting): picked mon.noname-a con 0x7f100c138e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100c09d2f0 con 0x7f100c138e40 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100c13d960 con 0x7f100c0a0df0 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100c13eb40 con 0x7f100c135330 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1013114640 10 monclient(hunting): authenticate will time out at 1962.355374s 2024-09-16T05:51:40.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f100bfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f100c135330 0x7f100c137720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42280/0 (socket says 172.21.15.6:42280) 2024-09-16T05:51:40.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f10116b1640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f100c138e40 0x7f100c13b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34198/0 (socket says 172.21.15.6:34198) 2024-09-16T05:51:40.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f1010eb0640 1 -- 172.21.15.6:0/1186143534 >> v1:172.21.15.63:6789/0 conn(0x7f100c0a0df0 legacy=0x7f100c09de60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:40.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f100bfff640 1 -- 172.21.15.6:0/1186143534 learned_addr learned my addr 172.21.15.6:0/1186143534 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:40.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 850618456 0 0) 0x7f100c09d2f0 con 0x7f100c138e40 2024-09-16T05:51:40.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f100b7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f100b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f100b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f100b7fe640 10 monclient(hunting): my global_id is 174159 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.546+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff4003530 con 0x7f100c138e40 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 66285678 0 0) 0x7f0ff4003530 con 0x7f100c138e40 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 >> v1:172.21.15.6:6790/0 conn(0x7f100c135330 legacy=0x7f100c137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 >> v1:172.21.15.63:6789/0 conn(0x7f100c0a0df0 legacy=0x7f100c09de60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 1 monclient: found mon.noname-a 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100c13fd20 con 0x7f100c138e40 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:40.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f10040019a0 con 0x7f100c138e40 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 5 monclient: authenticate success, global_id 174159 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: dump: 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.558 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1004001c90 con 0x7f100c138e40 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 4 set_mon_vals no callback set 2024-09-16T05:51:40.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 1 -- 172.21.15.6:0/1186143534 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f10040020a0 con 0x7f100c138e40 2024-09-16T05:51:40.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100b7fe640 10 monclient: dump: 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: shutdown 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- 172.21.15.6:0/1186143534 >> v1:172.21.15.6:6789/0 conn(0x7f100c138e40 legacy=0x7f100c13b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- 172.21.15.6:0/1186143534 shutdown_connections 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- 172.21.15.6:0/1186143534 >> 172.21.15.6:0/1186143534 conn(0x7f100c0a1650 msgr2=0x7f100c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- 172.21.15.6:0/1186143534 shutdown_connections 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- 172.21.15.6:0/1186143534 wait complete. 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: build_initial_monmap 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: monmap: 2024-09-16T05:51:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 Processor -- start 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- start start 2024-09-16T05:51:40.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: init 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient(hunting): picked mon.noname-c con 0x7f100c0a0df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient(hunting): picked mon.noname-b con 0x7f100c135330 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient(hunting): picked mon.noname-a con 0x7f100c138e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100c0740c0 con 0x7f100c138e40 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100c11bd30 con 0x7f100c135330 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100c13df30 con 0x7f100c0a0df0 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient(hunting): authenticate will time out at 1962.358715s 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1010eb0640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f100c0a0df0 0x7f100c134810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42296/0 (socket says 172.21.15.6:42296) 2024-09-16T05:51:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1010eb0640 1 -- 172.21.15.6:0/1260323773 learned_addr learned my addr 172.21.15.6:0/1260323773 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100bfff640 1 -- 172.21.15.6:0/1260323773 >> v1:172.21.15.63:6789/0 conn(0x7f100c135330 legacy=0x7f100c1d03e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277500616 0 0) 0x7f100c0740c0 con 0x7f100c138e40 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient(hunting): my global_id is 174162 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff8003560 con 0x7f100c138e40 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 448418289 0 0) 0x7f0ff8003560 con 0x7f100c138e40 2024-09-16T05:51:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f100c0740c0 con 0x7f100c138e40 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1004001960 con 0x7f100c138e40 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient(hunting): dump: 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3738846794 0 0) 0x7f100c0740c0 con 0x7f100c138e40 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 >> v1:172.21.15.6:6790/0 conn(0x7f100c0a0df0 legacy=0x7f100c134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 >> v1:172.21.15.63:6789/0 conn(0x7f100c135330 legacy=0x7f100c1d03e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient: _finish_hunting 0 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 monclient: found mon.a 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100c11ba90 con 0x7f100c138e40 2024-09-16T05:51:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient: _finish_auth 0 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient: _check_auth_tickets 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 5 monclient: authenticate success, global_id 174162 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: _renew_subs 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f100c11cea0 con 0x7f100c138e40 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: _renew_subs 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f100c134f20 con 0x7f100c138e40 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1004004f90 con 0x7f100c138e40 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1004005140 con 0x7f100c138e40 2024-09-16T05:51:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f100affd640 4 set_mon_vals no callback set 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.550+0000 7f10097fa640 10 monclient: dump: 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:40.570 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.554+0000 7f1013114640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.554+0000 7f1013114640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.554+0000 7f1013114640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.554+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f100400fee0 con 0x7f100c138e40 2024-09-16T05:51:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.554+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fec005090 con 0x7f100c138e40 2024-09-16T05:51:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.554+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f1004045260 con 0x7f100c138e40 2024-09-16T05:51:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.554+0000 7f1010eb0640 10 monclient: get_auth_request con 0x7f0ff8036bf0 auth_method 0 2024-09-16T05:51:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.558+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1004019480 con 0x7f100c138e40 2024-09-16T05:51:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.558+0000 7f10097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.558+0000 7f10097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:40.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:51:40.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:40.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:40.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0fec005380 con 0x7f100c138e40 2024-09-16T05:51:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f10097fa640 1 -- 172.21.15.6:0/1260323773 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 882800893) 0x7f100401dd40 con 0x7f100c138e40 2024-09-16T05:51:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f10097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:40.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f10097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:40.786 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:51:40.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0ff8036bf0 legacy=0x7f0ff80390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 10 monclient: shutdown 2024-09-16T05:51:40.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:40.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 >> v1:172.21.15.6:6789/0 conn(0x7f100c138e40 legacy=0x7f100c1d3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 shutdown_connections 2024-09-16T05:51:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 >> 172.21.15.6:0/1260323773 conn(0x7f100c0a1650 msgr2=0x7f100c13ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 shutdown_connections 2024-09-16T05:51:40.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:40.786+0000 7f1013114640 1 -- 172.21.15.6:0/1260323773 wait complete. 2024-09-16T05:51:43.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:51:43.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient: get_monmap_and_config 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient: build_initial_monmap 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient: monmap: 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:43.936886+0000 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:43.936886+0000 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:43.939 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 1 Processor -- start 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 1 -- start start 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient: init 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient(hunting): picked mon.noname-a con 0x7f403c09aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient(hunting): picked mon.noname-b con 0x7f403c09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient(hunting): picked mon.noname-c con 0x7f403c0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403c0a1370 con 0x7f403c09aa30 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403c09ead0 con 0x7f403c0a0da0 2024-09-16T05:51:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403c09edb0 con 0x7f403c09e230 2024-09-16T05:51:43.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:43.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f4041683640 10 monclient(hunting): authenticate will time out at 1965.743697s 2024-09-16T05:51:43.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f403a7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f403c09e230 0x7f403c1399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42312/0 (socket says 172.21.15.6:42312) 2024-09-16T05:51:43.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f403a7fc640 1 -- 172.21.15.6:0/1089022653 learned_addr learned my addr 172.21.15.6:0/1089022653 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:43.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.933+0000 7f403b7fe640 1 -- 172.21.15.6:0/1089022653 >> v1:172.21.15.63:6789/0 conn(0x7f403c0a0da0 legacy=0x7f403c13d100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:43.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3754269412 0 0) 0x7f403c0a1370 con 0x7f403c09aa30 2024-09-16T05:51:43.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:51:43.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:43.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:43.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient(hunting): my global_id is 174165 2024-09-16T05:51:43.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4028003530 con 0x7f403c09aa30 2024-09-16T05:51:43.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3591447902 0 0) 0x7f4028003530 con 0x7f403c09aa30 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 >> v1:172.21.15.6:6790/0 conn(0x7f403c09e230 legacy=0x7f403c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 >> v1:172.21.15.63:6789/0 conn(0x7f403c0a0da0 legacy=0x7f403c13d100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 monclient: found mon.noname-a 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403c09dd50 con 0x7f403c09aa30 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: _finish_auth 0 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:43.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4024001620 con 0x7f403c09aa30 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 5 monclient: authenticate success, global_id 174165 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: dump: 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:43.945 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4024001880 con 0x7f403c09aa30 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 1 -- 172.21.15.6:0/1089022653 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4024001c50 con 0x7f403c09aa30 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4039ffb640 10 monclient: dump: 2024-09-16T05:51:43.946 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:43.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:43.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 4 set_mon_vals no callback set 2024-09-16T05:51:43.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient: shutdown 2024-09-16T05:51:43.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:43.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- 172.21.15.6:0/1089022653 >> v1:172.21.15.6:6789/0 conn(0x7f403c09aa30 legacy=0x7f403c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:43.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- 172.21.15.6:0/1089022653 shutdown_connections 2024-09-16T05:51:43.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- 172.21.15.6:0/1089022653 >> 172.21.15.6:0/1089022653 conn(0x7f403c0a1650 msgr2=0x7f403c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:43.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- 172.21.15.6:0/1089022653 shutdown_connections 2024-09-16T05:51:43.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- 172.21.15.6:0/1089022653 wait complete. 2024-09-16T05:51:43.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient: build_initial_monmap 2024-09-16T05:51:43.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient: monmap: 2024-09-16T05:51:43.953 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:43.953 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:43.953 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:43.953 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 Processor -- start 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- start start 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient: init 2024-09-16T05:51:43.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient(hunting): picked mon.noname-c con 0x7f403c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient(hunting): picked mon.noname-b con 0x7f403c09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient(hunting): picked mon.noname-a con 0x7f403c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403c09c990 con 0x7f403c0a0da0 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403c09d7c0 con 0x7f403c09e230 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403c13cd30 con 0x7f403c09aa30 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f4041683640 10 monclient(hunting): authenticate will time out at 1965.747429s 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f403affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f403c09aa30 0x7f403c09bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42316/0 (socket says 172.21.15.6:42316) 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f403affd640 1 -- 172.21.15.6:0/2713721481 learned_addr learned my addr 172.21.15.6:0/2713721481 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f403a7fc640 1 -- 172.21.15.6:0/2713721481 >> v1:172.21.15.63:6789/0 conn(0x7f403c09e230 legacy=0x7f403c09c280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:43.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2337719072 0 0) 0x7f403c09c990 con 0x7f403c0a0da0 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f40337fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f40337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f40337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f40337fe640 10 monclient(hunting): my global_id is 174168 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.937+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f401c003530 con 0x7f403c0a0da0 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 277446909 0 0) 0x7f401c003530 con 0x7f403c0a0da0 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f403c09c990 con 0x7f403c0a0da0 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4034004040 con 0x7f403c0a0da0 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient(hunting): dump: 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2518257441 0 0) 0x7f403c09c990 con 0x7f403c0a0da0 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 >> v1:172.21.15.6:6790/0 conn(0x7f403c09aa30 legacy=0x7f403c09bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 >> v1:172.21.15.63:6789/0 conn(0x7f403c09e230 legacy=0x7f403c09c280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:51:43.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 monclient: found mon.a 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403c13b8e0 con 0x7f403c0a0da0 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 5 monclient: authenticate success, global_id 174168 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 10 monclient: _renew_subs 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f403c13c490 con 0x7f403c0a0da0 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 10 monclient: _renew_subs 2024-09-16T05:51:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f403c13b5a0 con 0x7f403c0a0da0 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4034004370 con 0x7f403c0a0da0 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f40340046c0 con 0x7f403c0a0da0 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40397fa640 4 set_mon_vals no callback set 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 10 monclient: dump: 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4020005090 con 0x7f403c0a0da0 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f4034004920 con 0x7f403c0a0da0 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f4034044e50 con 0x7f403c0a0da0 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.941+0000 7f403affd640 10 monclient: get_auth_request con 0x7f401c036950 auth_method 0 2024-09-16T05:51:43.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.945+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f4034019070 con 0x7f403c0a0da0 2024-09-16T05:51:43.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.945+0000 7f40337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:43.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:43.945+0000 7f40337fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:44.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:51:44.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:44.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:44.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4020005380 con 0x7f403c0a0da0 2024-09-16T05:51:44.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f40337fe640 1 -- 172.21.15.6:0/2713721481 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 969463418) 0x7f403401d930 con 0x7f403c0a0da0 2024-09-16T05:51:44.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f40337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:44.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f40337fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:44.171 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:51:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 >> v1:172.21.15.63:6812/3643207211 conn(0x7f401c036950 legacy=0x7f401c038e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 10 monclient: shutdown 2024-09-16T05:51:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 >> v1:172.21.15.6:6789/0 conn(0x7f403c0a0da0 legacy=0x7f403c1d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 shutdown_connections 2024-09-16T05:51:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 >> 172.21.15.6:0/2713721481 conn(0x7f403c0a1650 msgr2=0x7f403c06ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 shutdown_connections 2024-09-16T05:51:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:44.169+0000 7f4041683640 1 -- 172.21.15.6:0/2713721481 wait complete. 2024-09-16T05:51:47.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient: get_monmap_and_config 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient: build_initial_monmap 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient: monmap: 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:47.324617+0000 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:47.324617+0000 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:47.323 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:47.324 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:47.324 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:47.324 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:47.324 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 1 Processor -- start 2024-09-16T05:51:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 1 -- start start 2024-09-16T05:51:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient: init 2024-09-16T05:51:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:51:47.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient(hunting): picked mon.noname-b con 0x7f281809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient(hunting): picked mon.noname-c con 0x7f281809bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient(hunting): picked mon.noname-a con 0x7f28181397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28180a1350 con 0x7f28181397f0 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28180a10d0 con 0x7f281809bf20 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f281809d940 con 0x7f281809bb40 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281e878640 10 monclient(hunting): authenticate will time out at 1969.131252s 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f28177fe640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f281809c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281ce15640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f28181397f0 0x7f281813bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34234/0 (socket says 172.21.15.6:34234) 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f2817fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f281809bb40 0x7f281809e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:42322/0 (socket says 172.21.15.6:42322) 2024-09-16T05:51:47.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.321+0000 7f281ce15640 1 -- 172.21.15.6:0/3042121233 learned_addr learned my addr 172.21.15.6:0/3042121233 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:47.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.525+0000 7f28177fe640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f281809c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:47.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:47.929+0000 7f28177fe640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f281809c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:48.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:48.733+0000 7f28177fe640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f281809c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 10 monclient(hunting): tick 2024-09-16T05:51:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:51:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 1 monclient(hunting): continuing hunt 2024-09-16T05:51:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.6:6790/0 conn(0x7f281809bb40 legacy=0x7f281809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f281809c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.6:6789/0 conn(0x7f28181397f0 legacy=0x7f281813bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f28040036d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f281809bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f281809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f28177fe640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f2804009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f281809dc20 con 0x7f28040036d0 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28180a10d0 con 0x7f281809bf20 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28040103e0 con 0x7f281809bb40 2024-09-16T05:51:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.321+0000 7f2815ffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:50.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.525+0000 7f28177fe640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f2804009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:50.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:50.929+0000 7f28177fe640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f2804009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:51.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:51.733+0000 7f28177fe640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f2804009420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 10 monclient(hunting): tick 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 1 monclient(hunting): continuing hunt 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.6:6790/0 conn(0x7f281809bb40 legacy=0x7f280400cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f281809bf20 legacy=0x7f2804009420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.6:6789/0 conn(0x7f28040036d0 legacy=0x7f2804005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f28040106b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:53.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f2804009b50 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f2804009f30 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28040116f0 con 0x7f2804009f30 2024-09-16T05:51:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28180a10d0 con 0x7f28040106b0 2024-09-16T05:51:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 1 -- 172.21.15.6:0/3042121233 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2804005ad0 con 0x7f2804009b50 2024-09-16T05:51:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2815ffb640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:53.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.325+0000 7f2817fff640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f28040106b0 legacy=0x7f2804010af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:53.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.529+0000 7f2817fff640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f28040106b0 legacy=0x7f2804010af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:53.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:53.933+0000 7f2817fff640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f28040106b0 legacy=0x7f2804010af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1671978960 0 0) 0x7f2804005ad0 con 0x7f2804009b50 2024-09-16T05:51:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:51:54.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient(hunting): my global_id is 164258 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2800003530 con 0x7f2804009b50 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3145103166 0 0) 0x7f2800003530 con 0x7f2804009b50 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.63:6789/0 conn(0x7f28040106b0 legacy=0x7f2804010af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.6:6789/0 conn(0x7f2804009f30 legacy=0x7f280400a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 monclient: found mon.noname-b 2024-09-16T05:51:54.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2804005ca0 con 0x7f2804009b50 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: _finish_auth 0 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f280c004240 con 0x7f2804009b50 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: dump: 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 5 monclient: authenticate success, global_id 164258 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f280c0044a0 con 0x7f2804009b50 2024-09-16T05:51:54.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:54.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 1 -- 172.21.15.6:0/3042121233 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f280c0048b0 con 0x7f2804009b50 2024-09-16T05:51:54.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:54.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 4 set_mon_vals no callback set 2024-09-16T05:51:54.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:54.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:54.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f2816ffd640 10 monclient: dump: 2024-09-16T05:51:54.587 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:54.587 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.588 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.588 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.588 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:54.588 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.588 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:54.588 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:54.588 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:54.588 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 10 monclient: shutdown 2024-09-16T05:51:54.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:54.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 1 -- 172.21.15.6:0/3042121233 >> v1:172.21.15.6:6790/0 conn(0x7f2804009b50 legacy=0x7f2804009420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:54.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 1 -- 172.21.15.6:0/3042121233 shutdown_connections 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 1 -- 172.21.15.6:0/3042121233 >> 172.21.15.6:0/3042121233 conn(0x7f28180a1650 msgr2=0x7f28180a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 1 -- 172.21.15.6:0/3042121233 shutdown_connections 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.581+0000 7f281e878640 1 -- 172.21.15.6:0/3042121233 wait complete. 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: build_initial_monmap 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: monmap: 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:54.596 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 1 Processor -- start 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 1 -- start start 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: init 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient(hunting): picked mon.noname-a con 0x7f2804009b50 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient(hunting): picked mon.noname-c con 0x7f2804009f30 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient(hunting): picked mon.noname-b con 0x7f28040106b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:54.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28180740c0 con 0x7f2804009b50 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2818071a90 con 0x7f28040106b0 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2818135610 con 0x7f2804009f30 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient(hunting): authenticate will time out at 1976.394048s 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281ce15640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f28040106b0 legacy=0x7f28181d3a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2817fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2804009b50 0x7f28181347f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60804/0 (socket says 172.21.15.6:60804) 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f28177fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2804009f30 0x7f2818134f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60842/0 (socket says 172.21.15.6:60842) 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2817fff640 1 -- 172.21.15.6:0/2899216497 learned_addr learned my addr 172.21.15.6:0/2899216497 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:54.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2893301553 0 0) 0x7f2818135610 con 0x7f2804009f30 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient(hunting): my global_id is 164261 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2804005f10 con 0x7f2804009f30 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3863393293 0 0) 0x7f2804005f10 con 0x7f2804009f30 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2818135610 con 0x7f2804009f30 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f280c004020 con 0x7f2804009f30 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient(hunting): dump: 2024-09-16T05:51:54.599 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:54.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3360181548 0 0) 0x7f2818135610 con 0x7f2804009f30 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 >> v1:172.21.15.63:6789/0 conn(0x7f28040106b0 legacy=0x7f28181d3a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 >> v1:172.21.15.6:6789/0 conn(0x7f2804009b50 legacy=0x7f28181347f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 monclient: found mon.c 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2818071d70 con 0x7f2804009f30 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient: _finish_auth 0 2024-09-16T05:51:54.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 5 monclient: authenticate success, global_id 164261 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: _renew_subs 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2818138860 con 0x7f2804009f30 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: _renew_subs 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2818070640 con 0x7f2804009f30 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f280c004dd0 con 0x7f2804009f30 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f280c005120 con 0x7f2804009f30 2024-09-16T05:51:54.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f28167fc640 4 set_mon_vals no callback set 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.585+0000 7f2814ff9640 10 monclient: dump: 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:54.603 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.589+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f280c00fec0 con 0x7f2804009f30 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.589+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f280c044df0 con 0x7f2804009f30 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.589+0000 7f281e878640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.589+0000 7f281e878640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.589+0000 7f281e878640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.589+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f8005090 con 0x7f2804009f30 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.589+0000 7f2817fff640 10 monclient: get_auth_request con 0x7f2804038510 auth_method 0 2024-09-16T05:51:54.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.593+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f280c015540 con 0x7f2804009f30 2024-09-16T05:51:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.593+0000 7f2814ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:54.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.593+0000 7f2814ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:54.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:51:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f27f8005380 con 0x7f2804009f30 2024-09-16T05:51:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f2814ff9640 1 -- 172.21.15.6:0/2899216497 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 3889661118) 0x7f280c014e40 con 0x7f2804009f30 2024-09-16T05:51:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f2814ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:51:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f2814ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:54.828 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":74,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:51:54.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2804038510 legacy=0x7f280403a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 10 monclient: shutdown 2024-09-16T05:51:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 >> v1:172.21.15.6:6790/0 conn(0x7f2804009f30 legacy=0x7f2818134f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 shutdown_connections 2024-09-16T05:51:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 >> 172.21.15.6:0/2899216497 conn(0x7f28180a1650 msgr2=0x7f28181339f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 shutdown_connections 2024-09-16T05:51:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.825+0000 7f281e878640 1 -- 172.21.15.6:0/2899216497 wait complete. 2024-09-16T05:51:54.846 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:51:54.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T05:51:54.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:51:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.969+0000 7fe60c1a8640 10 monclient: get_monmap_and_config 2024-09-16T05:51:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.969+0000 7fe60c1a8640 10 monclient: build_initial_monmap 2024-09-16T05:51:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.969+0000 7fe60c1a8640 10 monclient: monmap: 2024-09-16T05:51:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.973 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:54.974822+0000 2024-09-16T05:51:54.974 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:54.974822+0000 2024-09-16T05:51:54.974 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:54.974 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.974 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:54.974 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:54.974 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:54.974 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 Processor -- start 2024-09-16T05:51:54.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- start start 2024-09-16T05:51:54.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient: init 2024-09-16T05:51:54.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:54.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:51:54.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe60409aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe60409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe6040a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6040a1370 con 0x7fe6040a0da0 2024-09-16T05:51:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60409ead0 con 0x7fe60409aa30 2024-09-16T05:51:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60409edb0 con 0x7fe60409e230 2024-09-16T05:51:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient(hunting): authenticate will time out at 1976.781491s 2024-09-16T05:51:54.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe609f44640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7fe60409aa30 legacy=0x7fe60409ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe609743640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe60409e230 0x7fe6041399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60846/0 (socket says 172.21.15.6:60846) 2024-09-16T05:51:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe609743640 1 -- 172.21.15.6:0/2392133860 learned_addr learned my addr 172.21.15.6:0/2392133860 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698389658 0 0) 0x7fe6040a1370 con 0x7fe6040a0da0 2024-09-16T05:51:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient(hunting): _negotiate 2024-09-16T05:51:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient(hunting): my global_id is 174171 2024-09-16T05:51:54.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5ec003530 con 0x7fe6040a0da0 2024-09-16T05:51:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775238934 0 0) 0x7fe60409edb0 con 0x7fe60409e230 2024-09-16T05:51:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient(hunting): _negotiate 2024-09-16T05:51:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient(hunting): my global_id is 164264 2024-09-16T05:51:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6040a1370 con 0x7fe60409e230 2024-09-16T05:51:54.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 830650881 0 0) 0x7fe5ec003530 con 0x7fe6040a0da0 2024-09-16T05:51:54.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 >> v1:172.21.15.6:6790/0 conn(0x7fe60409e230 legacy=0x7fe6041399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:54.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 >> v1:172.21.15.63:6789/0 conn(0x7fe60409aa30 legacy=0x7fe60409ae10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:51:54.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: _finish_hunting 0 2024-09-16T05:51:54.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 monclient: found mon.noname-a 2024-09-16T05:51:54.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:54.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:51:54.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60409dd50 con 0x7fe6040a0da0 2024-09-16T05:51:54.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: _finish_auth 0 2024-09-16T05:51:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: _check_auth_tickets 2024-09-16T05:51:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 5 monclient: authenticate success, global_id 174171 2024-09-16T05:51:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5fc001620 con 0x7fe6040a0da0 2024-09-16T05:51:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:54.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: dump: 2024-09-16T05:51:54.988 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:54.988 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.988 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.988 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.988 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:54.988 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.988 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:54.989 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5fc001880 con 0x7fe6040a0da0 2024-09-16T05:51:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 1 -- 172.21.15.6:0/2392133860 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5fc001c10 con 0x7fe6040a0da0 2024-09-16T05:51:54.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 4 set_mon_vals no callback set 2024-09-16T05:51:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe608f42640 10 monclient: dump: 2024-09-16T05:51:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.990 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.991 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:54.991 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.991 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:54.991 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:54.991 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:54.991 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient: shutdown 2024-09-16T05:51:54.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- 172.21.15.6:0/2392133860 >> v1:172.21.15.6:6789/0 conn(0x7fe6040a0da0 legacy=0x7fe60413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- 172.21.15.6:0/2392133860 shutdown_connections 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- 172.21.15.6:0/2392133860 >> 172.21.15.6:0/2392133860 conn(0x7fe6040a1650 msgr2=0x7fe6040a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- 172.21.15.6:0/2392133860 shutdown_connections 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 1 -- 172.21.15.6:0/2392133860 wait complete. 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient: build_initial_monmap 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.973+0000 7fe60c1a8640 10 monclient: monmap: 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:54.992 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 1 Processor -- start 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 1 -- start start 2024-09-16T05:51:54.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient: init 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe60409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe60409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe6040a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6040a0150 con 0x7fe60409e230 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60406e6d0 con 0x7fe6040a0da0 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6040740c0 con 0x7fe60409aa30 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient(hunting): authenticate will time out at 1976.784679s 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe609f44640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe60409aa30 0x7fe60413d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60862/0 (socket says 172.21.15.6:60862) 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe609f44640 1 -- 172.21.15.6:0/3666377286 learned_addr learned my addr 172.21.15.6:0/3666377286 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:54.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe609743640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe60409e230 0x7fe6041cdf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60822/0 (socket says 172.21.15.6:60822) 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60a745640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.63:6789/0 conn(0x7fe6040a0da0 legacy=0x7fe6041d1630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1715839931 0 0) 0x7fe6040a0150 con 0x7fe60409e230 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): my global_id is 174174 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5f0003530 con 0x7fe60409e230 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833540492 0 0) 0x7fe6040740c0 con 0x7fe60409aa30 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:54.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): my global_id is 164267 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6040a0150 con 0x7fe60409aa30 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1907973367 0 0) 0x7fe5f0003530 con 0x7fe60409e230 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6040740c0 con 0x7fe60409e230 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5f8001950 con 0x7fe60409e230 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): dump: 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.996 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2232960811 0 0) 0x7fe6040a0150 con 0x7fe60409aa30 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5f0003530 con 0x7fe60409aa30 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5f4004410 con 0x7fe60409aa30 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): dump: 2024-09-16T05:51:54.997 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1329743118 0 0) 0x7fe6040740c0 con 0x7fe60409e230 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.6:6790/0 conn(0x7fe60409aa30 legacy=0x7fe60413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.63:6789/0 conn(0x7fe6040a0da0 legacy=0x7fe6041d1630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:51:54.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: _finish_hunting 0 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 monclient: found mon.a 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6041374c0 con 0x7fe60409e230 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: _finish_auth 0 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: _check_auth_tickets 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 5 monclient: authenticate success, global_id 174174 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient: _renew_subs 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6041c5b90 con 0x7fe60409e230 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient: _renew_subs 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:51:54.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe604136ef0 con 0x7fe60409e230 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5f8001c60 con 0x7fe60409e230 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5f8004c40 con 0x7fe60409e230 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: dump: 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe603fff640 4 set_mon_vals no callback set 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe6040008e0 con 0x7fe60409b4a0 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3233064630 0 0) 0x7fe6040008e0 con 0x7fe60409b4a0 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient(hunting): my global_id is 174177 2024-09-16T05:51:55.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6040740c0 con 0x7fe60409b4a0 2024-09-16T05:51:55.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe5f800f9e0 con 0x7fe60409e230 2024-09-16T05:51:55.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fe5f8044e10 con 0x7fe60409e230 2024-09-16T05:51:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3508319381 0 0) 0x7fe6040740c0 con 0x7fe60409b4a0 2024-09-16T05:51:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6040008e0 con 0x7fe60409b4a0 2024-09-16T05:51:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5f4004db0 con 0x7fe60409b4a0 2024-09-16T05:51:55.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:55.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:55.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: dump: 2024-09-16T05:51:55.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.005 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.005 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.006 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.006 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.006 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.006 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.006 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.006 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.007 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2973660226 0 0) 0x7fe6040008e0 con 0x7fe60409b4a0 2024-09-16T05:51:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6040635b0 con 0x7fe60409b4a0 2024-09-16T05:51:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe60a745640 10 monclient: get_auth_request con 0x7fe5f0036ae0 auth_method 0 2024-09-16T05:51:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe5f40070f0 con 0x7fe60409b4a0 2024-09-16T05:51:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:55.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:55.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.6:6789/0 conn(0x7fe60409b4a0 legacy=0x7fe604062ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.977+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.6:6789/0 conn(0x7fe60409b4a0 legacy=0x7fe604062ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:55.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:51:55.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe604130c10 con 0x7fe60402d7d0 2024-09-16T05:51:55.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2069294708 0 0) 0x7fe604130c10 con 0x7fe60402d7d0 2024-09-16T05:51:55.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient(hunting): _negotiate 2024-09-16T05:51:55.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:55.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:55.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient(hunting): my global_id is 174180 2024-09-16T05:51:55.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6040008e0 con 0x7fe60402d7d0 2024-09-16T05:51:55.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3633306556 0 0) 0x7fe6040008e0 con 0x7fe60402d7d0 2024-09-16T05:51:55.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe604130c10 con 0x7fe60402d7d0 2024-09-16T05:51:55.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe5f40053e0 con 0x7fe60402d7d0 2024-09-16T05:51:55.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:55.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:55.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient: dump: 2024-09-16T05:51:55.013 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.014 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.014 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.014 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.015 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.015 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.015 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.015 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.015 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.016 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3940630723 0 0) 0x7fe604130c10 con 0x7fe60402d7d0 2024-09-16T05:51:55.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe60402dbb0 con 0x7fe60402d7d0 2024-09-16T05:51:55.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4119 (unknown 0 0 75832425) 0x7fe5f40067f0 con 0x7fe60402d7d0 2024-09-16T05:51:55.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:51:55.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:55.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.6:6789/0 conn(0x7fe60402d7d0 legacy=0x7fe604130500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe6027fc640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.6:6789/0 conn(0x7fe60402d7d0 legacy=0x7fe604130500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:55.018 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:51:55.018 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:55.018 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:55.018 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:51:55.018 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 74, 2024-09-16T05:51:55.018 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 17659, 2024-09-16T05:51:55.019 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:51:55.019 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:51:55.019 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:51:55.019 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.019 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:51:55.020 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:55.020 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:51:55.020 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:51:55.020 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:55.020 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:55.020 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:55.021 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:55.021 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:55.021 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:55.021 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:55.021 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:55.021 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:55.022 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:55.023 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:55.024 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.025 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:51:55.026 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:55.027 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:51:55.028 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:55.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.028 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.028 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:51:55.029 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:51:55.029 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:55.029 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:55.029 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:55.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.030 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:55.030 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:55.030 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:55.030 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:55.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.030 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:55.030 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:51:55.031 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:55.031 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.031 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.031 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:51:55.031 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:51:55.031 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:55.031 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:55.032 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:55.032 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.032 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.032 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:51:55.032 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:51:55.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.033 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:55.033 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:55.033 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:55.033 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.033 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.033 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:51:55.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.034 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:55.034 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:55.034 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:55.034 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.034 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:51:55.035 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:51:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe5f0036ae0 legacy=0x7fe5f0038fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 10 monclient: shutdown 2024-09-16T05:51:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 >> v1:172.21.15.6:6789/0 conn(0x7fe60409e230 legacy=0x7fe6041cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60a745640 1 -- 172.21.15.6:0/3666377286 reap_dead start 2024-09-16T05:51:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60a745640 1 -- 172.21.15.6:0/3666377286 reap_dead start 2024-09-16T05:51:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 shutdown_connections 2024-09-16T05:51:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 >> 172.21.15.6:0/3666377286 conn(0x7fe6040a1650 msgr2=0x7fe604071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 shutdown_connections 2024-09-16T05:51:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:54.985+0000 7fe60c1a8640 1 -- 172.21.15.6:0/3666377286 wait complete. 2024-09-16T05:51:55.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:51:55.038 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:51:55.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:51:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient: get_monmap_and_config 2024-09-16T05:51:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient: build_initial_monmap 2024-09-16T05:51:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient: monmap: 2024-09-16T05:51:55.157 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:51:55.159094+0000 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:51:55.159094+0000 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:51:55.158 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 Processor -- start 2024-09-16T05:51:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- start start 2024-09-16T05:51:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient: init 2024-09-16T05:51:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:51:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient(hunting): picked mon.noname-b con 0x7f60100a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient(hunting): picked mon.noname-c con 0x7f60100a2570 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:55.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient(hunting): picked mon.noname-a con 0x7f60100a2980 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60100a33a0 con 0x7f60100a2980 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60100a07a0 con 0x7f60100a2570 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60100a2d90 con 0x7f60100a2190 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient(hunting): authenticate will time out at 1976.965787s 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6014a57640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f60100a2190 0x7f60100a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60872/0 (socket says 172.21.15.6:60872) 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600ffff640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f60100a2570 legacy=0x7f601013b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6014a57640 1 -- 172.21.15.6:0/2924815134 learned_addr learned my addr 172.21.15.6:0/2924815134 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1966240085 0 0) 0x7f60100a33a0 con 0x7f60100a2980 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:55.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient(hunting): my global_id is 174183 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ffc003530 con 0x7f60100a2980 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530001211 0 0) 0x7f60100a2d90 con 0x7f60100a2190 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient(hunting): my global_id is 164270 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60100a33a0 con 0x7f60100a2190 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2980007289 0 0) 0x7f60100a33a0 con 0x7f60100a2190 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 >> v1:172.21.15.63:6789/0 conn(0x7f60100a2570 legacy=0x7f601013b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 >> v1:172.21.15.6:6789/0 conn(0x7f60100a2980 legacy=0x7f601013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:51:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 monclient: found mon.noname-b 2024-09-16T05:51:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:51:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60100a66e0 con 0x7f60100a2190 2024-09-16T05:51:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:51:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:51:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:55.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ff8001600 con 0x7f60100a2190 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: dump: 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.170 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 5 monclient: authenticate success, global_id 164270 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ff80018e0 con 0x7f60100a2190 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 1 -- 172.21.15.6:0/2924815134 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ff8001cf0 con 0x7f60100a2190 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:55.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f600f7fe640 10 monclient: dump: 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient: get_monmap_and_config success 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 4 set_mon_vals no callback set 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 10 monclient: shutdown 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- 172.21.15.6:0/2924815134 >> v1:172.21.15.6:6790/0 conn(0x7f60100a2190 legacy=0x7f60100a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- 172.21.15.6:0/2924815134 shutdown_connections 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- 172.21.15.6:0/2924815134 >> 172.21.15.6:0/2924815134 conn(0x7f60100a9850 msgr2=0x7f60100a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- 172.21.15.6:0/2924815134 shutdown_connections 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.157+0000 7f6016cbb640 1 -- 172.21.15.6:0/2924815134 wait complete. 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: build_initial_monmap 2024-09-16T05:51:55.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: monmap: 2024-09-16T05:51:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:51:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:51:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:51:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:51:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:51:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.174 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 1 Processor -- start 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 1 -- start start 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: init 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: _reopen_session rank -1 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient(hunting): picked mon.noname-c con 0x7f60100a2190 addr v1:172.21.15.6:6790/0 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient(hunting): picked mon.noname-a con 0x7f60100a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:51:55.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient(hunting): picked mon.noname-b con 0x7f60100a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6010137730 con 0x7f60100a2570 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60101348a0 con 0x7f60100a2980 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6010137a10 con 0x7f60100a2190 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient(hunting): _renew_subs 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient(hunting): authenticate will time out at 1976.969121s 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6014a57640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f60100a2190 0x7f6010133a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60884/0 (socket says 172.21.15.6:60884) 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6014a57640 1 -- 172.21.15.6:0/2364366371 learned_addr learned my addr 172.21.15.6:0/2364366371 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f60100a2570 0x7f6010134190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60862/0 (socket says 172.21.15.6:60862) 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6015258640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.63:6789/0 conn(0x7f60100a2980 legacy=0x7f60101d3ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3431910153 0 0) 0x7f6010137730 con 0x7f60100a2570 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:55.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): my global_id is 174186 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ff0003530 con 0x7f60100a2570 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2411494134 0 0) 0x7f6010137a10 con 0x7f60100a2190 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): my global_id is 164273 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6010137730 con 0x7f60100a2190 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1864966769 0 0) 0x7f6010137730 con 0x7f60100a2190 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6010137a10 con 0x7f60100a2190 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ff8001c30 con 0x7f60100a2190 2024-09-16T05:51:55.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): dump: 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 631891586 0 0) 0x7f5ff0003530 con 0x7f60100a2570 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6010137730 con 0x7f60100a2570 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6000004300 con 0x7f60100a2570 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): dump: 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.179 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.180 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.180 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.180 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 82606842 0 0) 0x7f6010137a10 con 0x7f60100a2190 2024-09-16T05:51:55.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.63:6789/0 conn(0x7f60100a2980 legacy=0x7f60101d3ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:51:55.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.6:6789/0 conn(0x7f60100a2570 legacy=0x7f6010134190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:51:55.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 monclient: found mon.c 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60100a7c20 con 0x7f60100a2190 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 5 monclient: authenticate success, global_id 164273 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: _renew_subs 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60100a8170 con 0x7f60100a2190 2024-09-16T05:51:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: _renew_subs 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6010137450 con 0x7f60100a2190 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ff80018e0 con 0x7f60100a2190 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ff8004e20 con 0x7f60100a2190 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: dump: 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.182 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600effd640 4 set_mon_vals no callback set 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ff40065d0 con 0x7f5ff4002870 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5ff8004fc0 con 0x7f60100a2190 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f5ff8044f30 con 0x7f60100a2190 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2004496952 0 0) 0x7f5ff40065d0 con 0x7f5ff4002870 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:55.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient(hunting): my global_id is 164276 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6010137a10 con 0x7f5ff4002870 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f6015258640 10 monclient: get_auth_request con 0x7f5ff0036a30 auth_method 0 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 977805978 0 0) 0x7f6010137a10 con 0x7f5ff4002870 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ff40065d0 con 0x7f5ff4002870 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6000004a80 con 0x7f5ff4002870 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.161+0000 7f600d7fa640 10 monclient: dump: 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.184 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.165+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 900338341 0 0) 0x7f5ff40065d0 con 0x7f5ff4002870 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.165+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ff400a160 con 0x7f5ff4002870 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.165+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f60000058c0 con 0x7f5ff4002870 2024-09-16T05:51:55.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.165+0000 7f600d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:51:55.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.165+0000 7f600d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:51:55.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.165+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.6:6790/0 conn(0x7f5ff4002870 legacy=0x7f5ff4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.165+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.6:6790/0 conn(0x7f5ff4002870 legacy=0x7f5ff4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:55.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:51:55.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ff4009e50 con 0x7f5ff4009110 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2883174364 0 0) 0x7f5ff4009e50 con 0x7f5ff4009110 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient(hunting): my global_id is 164279 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ff40065d0 con 0x7f5ff4009110 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1686331987 0 0) 0x7f5ff40065d0 con 0x7f5ff4009110 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ff4009e50 con 0x7f5ff4009110 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f60000051a0 con 0x7f5ff4009110 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:51:55.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient: dump: 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1342409282 0 0) 0x7f5ff4009e50 con 0x7f5ff4009110 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ff400c3e0 con 0x7f5ff4009110 2024-09-16T05:51:55.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4285 (unknown 0 0 1130818028) 0x7f6000005030 con 0x7f5ff4009110 2024-09-16T05:51:55.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:51:55.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:51:55.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.6:6790/0 conn(0x7f5ff4009110 legacy=0x7f5ff40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f600d7fa640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.6:6790/0 conn(0x7f5ff4009110 legacy=0x7f5ff40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:51:55.190 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:51:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:55.190 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:55.191 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:51:55.191 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 74, 2024-09-16T05:51:55.191 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 126130, 2024-09-16T05:51:55.191 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:51:55.191 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:51:55.191 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:55.192 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:55.193 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:55.194 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:51:55.195 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:51:55.196 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:51:55.197 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:55.198 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.199 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:51:55.200 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:51:55.201 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:51:55.202 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.202 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.202 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.202 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:51:55.202 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:51:55.202 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.202 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:55.202 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:55.203 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:55.203 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.203 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.203 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:51:55.203 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.203 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:55.203 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:55.204 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:51:55.204 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.204 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.204 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:51:55.205 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.205 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:55.205 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:55.205 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 7 2024-09-16T05:51:55.206 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.206 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:51:55.206 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:51:55.206 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:51:55.206 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:51:55.207 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:51:55.207 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:51:55.207 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:51:55.207 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:51:55.207 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:51:55.207 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:51:55.208 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:51:55.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5ff0036a30 legacy=0x7f5ff0038ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 10 monclient: shutdown 2024-09-16T05:51:55.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 >> v1:172.21.15.6:6790/0 conn(0x7f60100a2190 legacy=0x7f6010133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6015258640 1 -- 172.21.15.6:0/2364366371 reap_dead start 2024-09-16T05:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6015258640 1 -- 172.21.15.6:0/2364366371 reap_dead start 2024-09-16T05:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 shutdown_connections 2024-09-16T05:51:55.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 >> 172.21.15.6:0/2364366371 conn(0x7f60100a9850 msgr2=0x7f601006ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:51:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.169+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 shutdown_connections 2024-09-16T05:51:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:51:55.173+0000 7f6016cbb640 1 -- 172.21.15.6:0/2364366371 wait complete. 2024-09-16T05:51:55.212 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:51:55.212 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:53:04.537 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=110 2024-09-16T05:53:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 110 -ne 110 ]] 2024-09-16T05:53:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:53:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:+ export CEPH_ARGS= 2024-09-16T05:53:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:+ CEPH_ARGS= 2024-09-16T05:53:04.538 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph auth del client.bar 2024-09-16T05:53:04.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient: get_monmap_and_config 2024-09-16T05:53:04.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient: build_initial_monmap 2024-09-16T05:53:04.645 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient: monmap: 2024-09-16T05:53:04.645 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:04.645 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:04.645 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:53:04.646064+0000 2024-09-16T05:53:04.645 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:53:04.646064+0000 2024-09-16T05:53:04.645 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:04.646 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:04.646 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:04.646 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:04.646 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:04.646 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:04.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 1 Processor -- start 2024-09-16T05:53:04.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 1 -- start start 2024-09-16T05:53:04.647 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient: init 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient(hunting): picked mon.noname-c con 0x7f0ac40a3a90 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient(hunting): picked mon.noname-b con 0x7f0ac40a4580 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient(hunting): picked mon.noname-a con 0x7f0ac413d510 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac40a0620 con 0x7f0ac413d510 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac40a1d30 con 0x7f0ac40a3a90 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac40a4fb0 con 0x7f0ac40a4580 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:04.648 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0ac8e40640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0ac40a3a90 legacy=0x7f0ac40a3e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:53:04.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.642+0000 7f0acb0a4640 10 monclient(hunting): authenticate will time out at 2041.423765s 2024-09-16T05:53:04.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0ac40a4580 0x7f0ac413bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.63:39266/0 (socket says 172.21.15.63:39266) 2024-09-16T05:53:04.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac9641640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0ac413d510 0x7f0ac413f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40392/0 (socket says 172.21.15.63:40392) 2024-09-16T05:53:04.649 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac3fff640 1 -- 172.21.15.63:0/912042953 learned_addr learned my addr 172.21.15.63:0/912042953 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587024844 0 0) 0x7f0abc0014a0 con 0x7f0ac413d510 2024-09-16T05:53:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:53:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient(hunting): my global_id is 174189 2024-09-16T05:53:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ab4003530 con 0x7f0ac413d510 2024-09-16T05:53:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2475096544 0 0) 0x7f0ac40a4fb0 con 0x7f0ac40a4580 2024-09-16T05:53:04.655 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient(hunting): my global_id is 164282 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0abc0014a0 con 0x7f0ac40a4580 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1479921362 0 0) 0x7f0ab4003530 con 0x7f0ac413d510 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 >> v1:172.21.15.6:6790/0 conn(0x7f0ac40a4580 legacy=0x7f0ac413bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 >> v1:172.21.15.63:6789/0 conn(0x7f0ac40a3a90 legacy=0x7f0ac40a3e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 monclient: found mon.noname-a 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:53:04.656 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac40a4ce0 con 0x7f0ac413d510 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: _finish_auth 0 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 5 monclient: authenticate success, global_id 174189 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0abc001b70 con 0x7f0ac413d510 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: dump: 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:04.657 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0abc001e90 con 0x7f0ac413d510 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:04.658 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 1 -- 172.21.15.63:0/912042953 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0abc004d10 con 0x7f0ac413d510 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 4 set_mon_vals no callback set 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0ac37fe640 10 monclient: dump: 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:04.659 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 10 monclient: shutdown 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 1 -- 172.21.15.63:0/912042953 >> v1:172.21.15.6:6789/0 conn(0x7f0ac413d510 legacy=0x7f0ac413f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 1 -- 172.21.15.63:0/912042953 shutdown_connections 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 1 -- 172.21.15.63:0/912042953 >> 172.21.15.63:0/912042953 conn(0x7f0ac40a9ab0 msgr2=0x7f0ac40a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 1 -- 172.21.15.63:0/912042953 shutdown_connections 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 1 -- 172.21.15.63:0/912042953 wait complete. 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 10 monclient: build_initial_monmap 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 10 monclient: monmap: 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:04.660 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 1 Processor -- start 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 1 -- start start 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.646+0000 7f0acb0a4640 10 monclient: init 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:53:04.661 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient(hunting): picked mon.noname-c con 0x7f0ac40a3a90 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient(hunting): picked mon.noname-a con 0x7f0ac40a4580 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient(hunting): picked mon.noname-b con 0x7f0ac413d510 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac40a0280 con 0x7f0ac40a4580 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac41395a0 con 0x7f0ac413d510 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac413c9c0 con 0x7f0ac40a3a90 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient(hunting): authenticate will time out at 2041.428328s 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac9641640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0ac413d510 legacy=0x7f0ac41d4070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.63:6789/0 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0ac40a4580 0x7f0ac41d0940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:40408/0 (socket says 172.21.15.63:40408) 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac3fff640 1 -- 172.21.15.63:0/3391409131 learned_addr learned my addr 172.21.15.63:0/3391409131 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3687256089 0 0) 0x7f0ac40a0280 con 0x7f0ac40a4580 2024-09-16T05:53:04.662 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): my global_id is 174192 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0aa8003530 con 0x7f0ac40a4580 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 525078759 0 0) 0x7f0ac413c9c0 con 0x7f0ac40a3a90 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): my global_id is 164285 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ac40a0280 con 0x7f0ac40a3a90 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3831690850 0 0) 0x7f0aa8003530 con 0x7f0ac40a4580 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ac413c9c0 con 0x7f0ac40a4580 2024-09-16T05:53:04.663 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ab00019b0 con 0x7f0ac40a4580 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): dump: 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:04.664 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2507044449 0 0) 0x7f0ac40a0280 con 0x7f0ac40a3a90 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0aa8003530 con 0x7f0ac40a3a90 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ab80040d0 con 0x7f0ac40a3a90 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient(hunting): dump: 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.665 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1411956683 0 0) 0x7f0ac413c9c0 con 0x7f0ac40a4580 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 >> v1:172.21.15.6:6790/0 conn(0x7f0ac40a3a90 legacy=0x7f0ac41cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 >> v1:172.21.15.63:6789/0 conn(0x7f0ac413d510 legacy=0x7f0ac41d4070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient: _finish_hunting 0 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 monclient: found mon.a 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:04.666 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac4138fe0 con 0x7f0ac40a4580 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient: _finish_auth 0 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient: _check_auth_tickets 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 5 monclient: authenticate success, global_id 174192 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient: _renew_subs 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac4138d00 con 0x7f0ac40a4580 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient: _renew_subs 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ac409df10 con 0x7f0ac40a4580 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ab00047f0 con 0x7f0ac40a4580 2024-09-16T05:53:04.667 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0ab0004b40 con 0x7f0ac40a4580 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac2ffd640 4 set_mon_vals no callback set 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 10 monclient: dump: 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:04.668 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aa4005090 con 0x7f0ac40a4580 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0ab0004d60 con 0x7f0ac40a4580 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.650+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f0ab0044e60 con 0x7f0ac40a4580 2024-09-16T05:53:04.669 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.654+0000 7f0ac8e40640 10 monclient: get_auth_request con 0x7f0aa8036a20 auth_method 0 2024-09-16T05:53:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.666+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f0ab0019080 con 0x7f0ac40a4580 2024-09-16T05:53:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.666+0000 7f0ac17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:04.670 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.666+0000 7f0ac17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:04.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.886+0000 7f0acb0a4640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T05:53:04.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.886+0000 7f0acb0a4640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T05:53:04.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.886+0000 7f0acb0a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:04.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.886+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) -- 0x7f0aa4005380 con 0x7f0ac40a4580 2024-09-16T05:53:04.893 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.890+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v28) ==== 80+0+0 (unknown 2522925092 0 0) 0x7f0ab001d940 con 0x7f0ac40a4580 2024-09-16T05:53:04.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.890+0000 7f0ac17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T05:53:04.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:04.890+0000 7f0ac17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:05.052 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0acb0a4640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T05:53:05.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0acb0a4640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T05:53:05.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0acb0a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:05.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) -- 0x7f0aa4005770 con 0x7f0ac40a4580 2024-09-16T05:53:05.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0ac17fa640 1 -- 172.21.15.63:0/3391409131 <== mon.0 v1:172.21.15.6:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v28) ==== 80+0+0 (unknown 2522925092 0 0) 0x7f0ab00402a0 con 0x7f0ac40a4580 2024-09-16T05:53:05.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0ac17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T05:53:05.053 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0ac17fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:53:05.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 >> v1:172.21.15.63:6812/3643207211 conn(0x7f0aa8036a20 legacy=0x7f0aa8038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:05.055 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0acb0a4640 10 monclient: shutdown 2024-09-16T05:53:05.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0acb0a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:05.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.050+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 >> v1:172.21.15.6:6789/0 conn(0x7f0ac40a4580 legacy=0x7f0ac41d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:05.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.054+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 shutdown_connections 2024-09-16T05:53:05.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.054+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 >> 172.21.15.63:0/3391409131 conn(0x7f0ac40a9ab0 msgr2=0x7f0ac40a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:05.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.054+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 shutdown_connections 2024-09-16T05:53:05.056 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:05.054+0000 7f0acb0a4640 1 -- 172.21.15.63:0/3391409131 wait complete. 2024-09-16T05:53:05.069 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13 2024-09-16T05:53:05.069 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-16T05:53:05.069 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=13 2024-09-16T05:53:05.069 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T05:53:05.069 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T05:53:05.069 INFO:tasks.workunit.client.0.smithi063.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T05:53:25.214 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T05:53:25.214 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T05:53:25.214 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T05:53:25.215 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T05:53:25.218 INFO:tasks.ceph.mon.b:Started 2024-09-16T05:53:25.218 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:53:25.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:53:25.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.348+0000 7fb39878b640 10 monclient: get_monmap_and_config 2024-09-16T05:53:25.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.348+0000 7fb39878b640 10 monclient: build_initial_monmap 2024-09-16T05:53:25.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.348+0000 7fb39878b640 10 monclient: monmap: 2024-09-16T05:53:25.353 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:25.353 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.353 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:25.354577+0000 2024-09-16T05:53:25.353 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:25.354577+0000 2024-09-16T05:53:25.354 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:25.354 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.354 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:25.354 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:25.354 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:25.354 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 1 Processor -- start 2024-09-16T05:53:25.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 1 -- start start 2024-09-16T05:53:25.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 10 monclient: init 2024-09-16T05:53:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:53:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 10 monclient(hunting): picked mon.noname-a con 0x7fb3900a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 10 monclient(hunting): picked mon.noname-c con 0x7fb3900a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 10 monclient(hunting): picked mon.noname-b con 0x7fb3900a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39006e520 con 0x7fb3900a37a0 2024-09-16T05:53:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3900a0780 con 0x7fb3900a4290 2024-09-16T05:53:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3900a4c70 con 0x7fb3900a1a00 2024-09-16T05:53:25.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:25.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb39878b640 10 monclient(hunting): authenticate will time out at 2067.161494s 2024-09-16T05:53:25.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb396527640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb3900a37a0 0x7fb3900a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53098/0 (socket says 172.21.15.6:53098) 2024-09-16T05:53:25.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.352+0000 7fb396527640 1 -- 172.21.15.6:0/3176084317 learned_addr learned my addr 172.21.15.6:0/3176084317 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:25.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3150165508 0 0) 0x7fb3900a4c70 con 0x7fb3900a1a00 2024-09-16T05:53:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient(hunting): my global_id is 164309 2024-09-16T05:53:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb37c003530 con 0x7fb3900a1a00 2024-09-16T05:53:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 651108891 0 0) 0x7fb37c003530 con 0x7fb3900a1a00 2024-09-16T05:53:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 >> v1:172.21.15.63:6789/0 conn(0x7fb3900a4290 legacy=0x7fb39013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 >> v1:172.21.15.6:6789/0 conn(0x7fb3900a37a0 legacy=0x7fb3900a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: _finish_hunting 0 2024-09-16T05:53:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 monclient: found mon.noname-b 2024-09-16T05:53:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:25.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:53:25.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900a65f0 con 0x7fb3900a1a00 2024-09-16T05:53:25.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: _finish_auth 0 2024-09-16T05:53:25.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: _check_auth_tickets 2024-09-16T05:53:25.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:25.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb3880019a0 con 0x7fb3900a1a00 2024-09-16T05:53:25.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:25.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: dump: 2024-09-16T05:53:25.393 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.393 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.395 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 5 monclient: authenticate success, global_id 164309 2024-09-16T05:53:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb388001c00 con 0x7fb3900a1a00 2024-09-16T05:53:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 1 -- 172.21.15.6:0/3176084317 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb388001fd0 con 0x7fb3900a1a00 2024-09-16T05:53:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb395525640 10 monclient: dump: 2024-09-16T05:53:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.398 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 4 set_mon_vals no callback set 2024-09-16T05:53:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 10 monclient: shutdown 2024-09-16T05:53:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 1 -- 172.21.15.6:0/3176084317 >> v1:172.21.15.6:6790/0 conn(0x7fb3900a1a00 legacy=0x7fb39013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 1 -- 172.21.15.6:0/3176084317 shutdown_connections 2024-09-16T05:53:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 1 -- 172.21.15.6:0/3176084317 >> 172.21.15.6:0/3176084317 conn(0x7fb3900a58a0 msgr2=0x7fb3900a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 1 -- 172.21.15.6:0/3176084317 shutdown_connections 2024-09-16T05:53:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 1 -- 172.21.15.6:0/3176084317 wait complete. 2024-09-16T05:53:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 10 monclient: build_initial_monmap 2024-09-16T05:53:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.376+0000 7fb39878b640 10 monclient: monmap: 2024-09-16T05:53:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:25.402 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 1 Processor -- start 2024-09-16T05:53:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 1 -- start start 2024-09-16T05:53:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient: init 2024-09-16T05:53:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:53:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient(hunting): picked mon.noname-a con 0x7fb3900a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient(hunting): picked mon.noname-c con 0x7fb3900a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient(hunting): picked mon.noname-b con 0x7fb3900a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39013ecf0 con 0x7fb3900a1a00 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39013efd0 con 0x7fb3900a4290 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3900a8310 con 0x7fb3900a37a0 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient(hunting): authenticate will time out at 2067.188741s 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb395d26640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb3900a37a0 0x7fb39013bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34112/0 (socket says 172.21.15.6:34112) 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb396527640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb3900a1a00 0x7fb39013b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53102/0 (socket says 172.21.15.6:53102) 2024-09-16T05:53:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb395d26640 1 -- 172.21.15.6:0/855903415 learned_addr learned my addr 172.21.15.6:0/855903415 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211307398 0 0) 0x7fb39013ecf0 con 0x7fb3900a1a00 2024-09-16T05:53:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): my global_id is 174216 2024-09-16T05:53:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb374003530 con 0x7fb3900a1a00 2024-09-16T05:53:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1076107700 0 0) 0x7fb3900a8310 con 0x7fb3900a37a0 2024-09-16T05:53:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): my global_id is 164312 2024-09-16T05:53:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb39013ecf0 con 0x7fb3900a37a0 2024-09-16T05:53:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1711116961 0 0) 0x7fb39013ecf0 con 0x7fb3900a37a0 2024-09-16T05:53:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3900a8310 con 0x7fb3900a37a0 2024-09-16T05:53:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb380001940 con 0x7fb3900a37a0 2024-09-16T05:53:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): dump: 2024-09-16T05:53:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.407 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 42631194 0 0) 0x7fb3900a8310 con 0x7fb3900a37a0 2024-09-16T05:53:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 >> v1:172.21.15.63:6789/0 conn(0x7fb3900a4290 legacy=0x7fb3901d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 >> v1:172.21.15.6:6789/0 conn(0x7fb3900a1a00 legacy=0x7fb39013b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:53:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 monclient: found mon.c 2024-09-16T05:53:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39013f2b0 con 0x7fb3900a37a0 2024-09-16T05:53:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient: _finish_auth 0 2024-09-16T05:53:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:53:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 5 monclient: authenticate success, global_id 164312 2024-09-16T05:53:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient: _renew_subs 2024-09-16T05:53:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb39013ea10 con 0x7fb3900a37a0 2024-09-16T05:53:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient: _renew_subs 2024-09-16T05:53:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb39013c4b0 con 0x7fb3900a37a0 2024-09-16T05:53:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb3800038d0 con 0x7fb3900a37a0 2024-09-16T05:53:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb380003b00 con 0x7fb3900a37a0 2024-09-16T05:53:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb394d24640 4 set_mon_vals no callback set 2024-09-16T05:53:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.380+0000 7fb386ffd640 10 monclient: dump: 2024-09-16T05:53:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.412 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.384+0000 7fb39878b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:25.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.384+0000 7fb39878b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:25.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.384+0000 7fb39878b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:25.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.384+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb380003d20 con 0x7fb3900a37a0 2024-09-16T05:53:25.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.384+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb378005090 con 0x7fb3900a37a0 2024-09-16T05:53:25.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.384+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fb380044fe0 con 0x7fb3900a37a0 2024-09-16T05:53:25.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.384+0000 7fb396527640 10 monclient: get_auth_request con 0x7fb374036b60 auth_method 0 2024-09-16T05:53:25.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.388+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fb3800155d0 con 0x7fb3900a37a0 2024-09-16T05:53:25.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.388+0000 7fb386ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:25.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.388+0000 7fb386ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:25.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.616+0000 7fb39878b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:53:25.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.616+0000 7fb39878b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:53:25.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.616+0000 7fb39878b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:25.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.616+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb378005380 con 0x7fb3900a37a0 2024-09-16T05:53:25.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.616+0000 7fb386ffd640 1 -- 172.21.15.6:0/855903415 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3241322063) 0x7fb380014ed0 con 0x7fb3900a37a0 2024-09-16T05:53:25.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.616+0000 7fb386ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:53:25.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.616+0000 7fb386ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:25.620 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:53:25.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.620+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb374036b60 legacy=0x7fb374039020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.620+0000 7fb39878b640 10 monclient: shutdown 2024-09-16T05:53:25.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.620+0000 7fb39878b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:25.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.620+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 >> v1:172.21.15.6:6790/0 conn(0x7fb3900a37a0 legacy=0x7fb39013bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.620+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 shutdown_connections 2024-09-16T05:53:25.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.620+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 >> 172.21.15.6:0/855903415 conn(0x7fb3900a58a0 msgr2=0x7fb390140040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.620+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 shutdown_connections 2024-09-16T05:53:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.620+0000 7fb39878b640 1 -- 172.21.15.6:0/855903415 wait complete. 2024-09-16T05:53:25.637 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:53:25.637 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:53:25.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:53:25.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: get_monmap_and_config 2024-09-16T05:53:25.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: build_initial_monmap 2024-09-16T05:53:25.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: monmap: 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:25.751113+0000 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:25.751113+0000 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:25.750 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 Processor -- start 2024-09-16T05:53:25.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- start start 2024-09-16T05:53:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: init 2024-09-16T05:53:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:53:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed00a2190 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed00a2570 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed00a2980 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:25.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed00a33a0 con 0x7f5ed00a2190 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed00a07a0 con 0x7f5ed00a2980 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed596a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ed00a2190 0x7f5ed00a1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53106/0 (socket says 172.21.15.6:53106) 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed00a2d90 con 0x7f5ed00a2570 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed596a640 1 -- 172.21.15.6:0/3642657814 learned_addr learned my addr 172.21.15.6:0/3642657814 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient(hunting): authenticate will time out at 2067.557840s 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434893807 0 0) 0x7f5ed00a33a0 con 0x7f5ed00a2190 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient(hunting): my global_id is 174219 2024-09-16T05:53:25.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5eb8003530 con 0x7f5ed00a2190 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2374033868 0 0) 0x7f5ed00a2d90 con 0x7f5ed00a2570 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient(hunting): my global_id is 164315 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed00a33a0 con 0x7f5ed00a2570 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 118229188 0 0) 0x7f5eb8003530 con 0x7f5ed00a2190 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 >> v1:172.21.15.6:6790/0 conn(0x7f5ed00a2570 legacy=0x7f5ed013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 >> v1:172.21.15.63:6789/0 conn(0x7f5ed00a2980 legacy=0x7f5ed013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: _finish_hunting 0 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 monclient: found mon.noname-a 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed00a66e0 con 0x7f5ed00a2190 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: _finish_auth 0 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: _check_auth_tickets 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec00019a0 con 0x7f5ed00a2190 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 5 monclient: authenticate success, global_id 174219 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: dump: 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.761 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ec0001c80 con 0x7f5ed00a2190 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 4 set_mon_vals no callback set 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 1 -- 172.21.15.6:0/3642657814 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec0002090 con 0x7f5ed00a2190 2024-09-16T05:53:25.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed4968640 10 monclient: dump: 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: shutdown 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- 172.21.15.6:0/3642657814 >> v1:172.21.15.6:6789/0 conn(0x7f5ed00a2190 legacy=0x7f5ed00a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- 172.21.15.6:0/3642657814 shutdown_connections 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- 172.21.15.6:0/3642657814 >> 172.21.15.6:0/3642657814 conn(0x7f5ed00a9850 msgr2=0x7f5ed00a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- 172.21.15.6:0/3642657814 shutdown_connections 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 1 -- 172.21.15.6:0/3642657814 wait complete. 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: build_initial_monmap 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.748+0000 7f5ed7bce640 10 monclient: monmap: 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:25.764 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 1 Processor -- start 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 1 -- start start 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient: init 2024-09-16T05:53:25.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed00a2190 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed00a2570 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed00a2980 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed0137490 con 0x7f5ed00a2570 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed0137770 con 0x7f5ed00a2190 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed00a8270 con 0x7f5ed00a2980 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient(hunting): authenticate will time out at 2067.560703s 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed5169640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ed00a2570 0x7f5ed01cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53114/0 (socket says 172.21.15.6:53114) 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed5169640 1 -- 172.21.15.6:0/1369390616 learned_addr learned my addr 172.21.15.6:0/1369390616 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2004571975 0 0) 0x7f5ed0137490 con 0x7f5ed00a2570 2024-09-16T05:53:25.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): my global_id is 174222 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ebc003530 con 0x7f5ed00a2570 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090704611 0 0) 0x7f5ed00a8270 con 0x7f5ed00a2980 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): my global_id is 164318 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed0137490 con 0x7f5ed00a2980 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2760917918 0 0) 0x7f5ebc003530 con 0x7f5ed00a2570 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ed00a8270 con 0x7f5ed00a2570 2024-09-16T05:53:25.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec8001a40 con 0x7f5ed00a2570 2024-09-16T05:53:25.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:25.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:25.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): dump: 2024-09-16T05:53:25.768 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.768 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.768 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.768 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 971427879 0 0) 0x7f5ed0137490 con 0x7f5ed00a2980 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ebc003530 con 0x7f5ed00a2980 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec4001800 con 0x7f5ed00a2980 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:25.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): dump: 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2958950597 0 0) 0x7f5ed0137770 con 0x7f5ed00a2190 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): my global_id is 184102 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed0137490 con 0x7f5ed00a2190 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3826905042 0 0) 0x7f5ed00a8270 con 0x7f5ed00a2570 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 >> v1:172.21.15.6:6790/0 conn(0x7f5ed00a2980 legacy=0x7f5ed01d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 >> v1:172.21.15.63:6789/0 conn(0x7f5ed00a2190 legacy=0x7f5ed0140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 monclient: found mon.a 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed00a7d40 con 0x7f5ed00a2570 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: _finish_auth 0 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:53:25.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 5 monclient: authenticate success, global_id 174222 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient: _renew_subs 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed0136040 con 0x7f5ed00a2570 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient: _renew_subs 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ed01d4960 con 0x7f5ed00a2570 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ec8001620 con 0x7f5ed00a2570 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec8003ad0 con 0x7f5ed00a2570 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:25.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: dump: 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ecffff640 4 set_mon_vals no callback set 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5eb00065d0 con 0x7f5eb0002870 2024-09-16T05:53:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f5ec8003cf0 con 0x7f5ed00a2570 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f5ec8044f80 con 0x7f5ed00a2570 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017426771 0 0) 0x7f5eb00065d0 con 0x7f5eb0002870 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient(hunting): my global_id is 174225 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed00a8270 con 0x7f5eb0002870 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 575371187 0 0) 0x7f5ed00a8270 con 0x7f5eb0002870 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5eb00065d0 con 0x7f5eb0002870 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec0001960 con 0x7f5eb0002870 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:25.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: dump: 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ed616b640 10 monclient: get_auth_request con 0x7f5ebc036bd0 auth_method 0 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2390758849 0 0) 0x7f5eb00065d0 con 0x7f5eb0002870 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5eb000a160 con 0x7f5eb0002870 2024-09-16T05:53:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5ec0001d90 con 0x7f5eb0002870 2024-09-16T05:53:25.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:25.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:25.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 >> v1:172.21.15.6:6789/0 conn(0x7f5eb0002870 legacy=0x7f5eb0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.752+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 >> v1:172.21.15.6:6789/0 conn(0x7f5eb0002870 legacy=0x7f5eb0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:25.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:53:25.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5eb0009e50 con 0x7f5eb0009110 2024-09-16T05:53:25.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1372654745 0 0) 0x7f5eb0009e50 con 0x7f5eb0009110 2024-09-16T05:53:25.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient(hunting): my global_id is 174228 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5eb00065d0 con 0x7f5eb0009110 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3287257823 0 0) 0x7f5eb00065d0 con 0x7f5eb0009110 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5eb0009e50 con 0x7f5eb0009110 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f5ec00051a0 con 0x7f5eb0009110 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:25.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient: dump: 2024-09-16T05:53:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2217848282 0 0) 0x7f5eb0009e50 con 0x7f5eb0009110 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5eb000c3e0 con 0x7f5eb0009110 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 1938401169) 0x7f5ec00073a0 con 0x7f5eb0009110 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 >> v1:172.21.15.6:6789/0 conn(0x7f5eb0009110 legacy=0x7f5eb00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ece7fc640 1 -- 172.21.15.6:0/1369390616 >> v1:172.21.15.6:6789/0 conn(0x7f5eb0009110 legacy=0x7f5eb00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 80, 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 108434, 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.781 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:25.782 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:25.783 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:53:25.784 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:25.786 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:53:25.788 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:53:25.789 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:53:25.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:53:25.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.793 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:53:25.793 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:53:25.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 >> v1:172.21.15.63:6812/3643207211 conn(0x7f5ebc036bd0 legacy=0x7f5ebc039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 10 monclient: shutdown 2024-09-16T05:53:25.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:25.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 >> v1:172.21.15.6:6789/0 conn(0x7f5ed00a2570 legacy=0x7f5ed01cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed616b640 1 -- 172.21.15.6:0/1369390616 reap_dead start 2024-09-16T05:53:25.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed616b640 1 -- 172.21.15.6:0/1369390616 reap_dead start 2024-09-16T05:53:25.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 shutdown_connections 2024-09-16T05:53:25.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 >> 172.21.15.6:0/1369390616 conn(0x7f5ed00a9850 msgr2=0x7f5ed012fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:25.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 shutdown_connections 2024-09-16T05:53:25.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.760+0000 7f5ed7bce640 1 -- 172.21.15.6:0/1369390616 wait complete. 2024-09-16T05:53:25.795 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:53:25.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:53:25.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:53:25.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient: get_monmap_and_config 2024-09-16T05:53:25.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient: build_initial_monmap 2024-09-16T05:53:25.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient: monmap: 2024-09-16T05:53:25.906 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:25.908082+0000 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:25.908082+0000 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:25.907 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 1 Processor -- start 2024-09-16T05:53:25.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 1 -- start start 2024-09-16T05:53:25.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient: init 2024-09-16T05:53:25.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:25.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient(hunting): picked mon.noname-c con 0x7f22dc09aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient(hunting): picked mon.noname-b con 0x7f22dc09e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient(hunting): picked mon.noname-a con 0x7f22dc0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc0a1370 con 0x7f22dc0a0da0 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc09ead0 con 0x7f22dc09aa30 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc09edb0 con 0x7f22dc09e230 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e3c5a640 10 monclient(hunting): authenticate will time out at 2067.715256s 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e21f7640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f22dc0a0da0 0x7f22dc13d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53132/0 (socket says 172.21.15.6:53132) 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e21f7640 1 -- 172.21.15.6:0/1454734755 learned_addr learned my addr 172.21.15.6:0/1454734755 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793101147 0 0) 0x7f22dc0a1370 con 0x7f22dc0a0da0 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 10 monclient(hunting): my global_id is 174231 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c8003530 con 0x7f22dc0a0da0 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106496961 0 0) 0x7f22dc09edb0 con 0x7f22dc09e230 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 10 monclient(hunting): my global_id is 164321 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.904+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22dc0a1370 con 0x7f22dc09e230 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4175824203 0 0) 0x7f22c8003530 con 0x7f22dc0a0da0 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 >> v1:172.21.15.6:6790/0 conn(0x7f22dc09e230 legacy=0x7f22dc1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 >> v1:172.21.15.63:6789/0 conn(0x7f22dc09aa30 legacy=0x7f22dc09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: _finish_hunting 0 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 1 monclient: found mon.noname-a 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc09dd50 con 0x7f22dc0a0da0 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: _finish_auth 0 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: _check_auth_tickets 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22d40019a0 con 0x7f22dc0a0da0 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 5 monclient: authenticate success, global_id 174231 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:25.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: dump: 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22d4001c00 con 0x7f22dc0a0da0 2024-09-16T05:53:25.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 1 -- 172.21.15.6:0/1454734755 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22d4001f50 con 0x7f22dc0a0da0 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e09f4640 10 monclient: dump: 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.915 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 4 set_mon_vals no callback set 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: shutdown 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- 172.21.15.6:0/1454734755 >> v1:172.21.15.6:6789/0 conn(0x7f22dc0a0da0 legacy=0x7f22dc13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- 172.21.15.6:0/1454734755 shutdown_connections 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- 172.21.15.6:0/1454734755 >> 172.21.15.6:0/1454734755 conn(0x7f22dc0a1650 msgr2=0x7f22dc0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- 172.21.15.6:0/1454734755 shutdown_connections 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- 172.21.15.6:0/1454734755 wait complete. 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: build_initial_monmap 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: monmap: 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:25.916 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:25.917 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:25.917 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:25.917 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.917 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:25.917 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:25.917 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:25.917 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 Processor -- start 2024-09-16T05:53:25.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- start start 2024-09-16T05:53:25.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: init 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient(hunting): picked mon.noname-b con 0x7f22dc09aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient(hunting): picked mon.noname-a con 0x7f22dc09e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient(hunting): picked mon.noname-c con 0x7f22dc0a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc13c400 con 0x7f22dc09e230 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc11cac0 con 0x7f22dc09aa30 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc06e660 con 0x7f22dc0a0da0 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient(hunting): authenticate will time out at 2067.718462s 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e11f5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f22dc09e230 0x7f22dc13bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53140/0 (socket says 172.21.15.6:53140) 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e11f5640 1 -- 172.21.15.6:0/3755684943 learned_addr learned my addr 172.21.15.6:0/3755684943 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:25.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4166320255 0 0) 0x7f22dc13c400 con 0x7f22dc09e230 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): my global_id is 174234 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c0003530 con 0x7f22dc09e230 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3947757895 0 0) 0x7f22dc06e660 con 0x7f22dc0a0da0 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): my global_id is 164324 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22dc13c400 con 0x7f22dc0a0da0 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3189067483 0 0) 0x7f22c0003530 con 0x7f22dc09e230 2024-09-16T05:53:25.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22dc06e660 con 0x7f22dc09e230 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22cc0019c0 con 0x7f22dc09e230 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): dump: 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.925 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957981818 0 0) 0x7f22dc11cac0 con 0x7f22dc09aa30 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): my global_id is 184108 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c0003530 con 0x7f22dc09aa30 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 377141436 0 0) 0x7f22dc13c400 con 0x7f22dc0a0da0 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22dc11cac0 con 0x7f22dc0a0da0 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22d40018a0 con 0x7f22dc0a0da0 2024-09-16T05:53:25.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient(hunting): dump: 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1835795760 0 0) 0x7f22dc06e660 con 0x7f22dc09e230 2024-09-16T05:53:25.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 >> v1:172.21.15.6:6790/0 conn(0x7f22dc0a0da0 legacy=0x7f22dc1d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 >> v1:172.21.15.63:6789/0 conn(0x7f22dc09aa30 legacy=0x7f22dc13b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient: _finish_hunting 0 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 monclient: found mon.a 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc13da30 con 0x7f22dc09e230 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient: _finish_auth 0 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient: _check_auth_tickets 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 5 monclient: authenticate success, global_id 174234 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: _renew_subs 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:25.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22dc0740c0 con 0x7f22dc09e230 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: _renew_subs 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f22dc13d640 con 0x7f22dc09e230 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22cc001cd0 con 0x7f22dc09e230 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22cc003a50 con 0x7f22dc09e230 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d27fc640 10 monclient: dump: 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.929 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.908+0000 7f22d37fe640 4 set_mon_vals no callback set 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22e3c5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22c40065d0 con 0x7f22c4002870 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f22cc003cf0 con 0x7f22dc09e230 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f22cc044de0 con 0x7f22dc09e230 2024-09-16T05:53:25.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1062776795 0 0) 0x7f22c40065d0 con 0x7f22c4002870 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient(hunting): my global_id is 164327 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22dc06e660 con 0x7f22c4002870 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 63638563 0 0) 0x7f22dc06e660 con 0x7f22c4002870 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22c40065d0 con 0x7f22c4002870 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22d8001720 con 0x7f22c4002870 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22e21f7640 10 monclient: get_auth_request con 0x7f22c0036c10 auth_method 0 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient: dump: 2024-09-16T05:53:25.931 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3309783347 0 0) 0x7f22c40065d0 con 0x7f22c4002870 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22c400a160 con 0x7f22c4002870 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f22d8001b30 con 0x7f22c4002870 2024-09-16T05:53:25.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 >> v1:172.21.15.6:6790/0 conn(0x7f22c4002870 legacy=0x7f22c4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.912+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 >> v1:172.21.15.6:6790/0 conn(0x7f22c4002870 legacy=0x7f22c4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22e3c5a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22c4009e30 con 0x7f22c40090f0 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727505318 0 0) 0x7f22c4009e30 con 0x7f22c40090f0 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22d27fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22d27fc640 10 monclient(hunting): my global_id is 164330 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c40065d0 con 0x7f22c40090f0 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.916+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4164527602 0 0) 0x7f22c40065d0 con 0x7f22c40090f0 2024-09-16T05:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22c4009e30 con 0x7f22c40090f0 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f22d80037a0 con 0x7f22c40090f0 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 10 monclient: dump: 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:25.934 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1361040745 0 0) 0x7f22c4009e30 con 0x7f22c40090f0 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22c400c3a0 con 0x7f22c40090f0 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 2732768647) 0x7f22d80038f0 con 0x7f22c40090f0 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 >> v1:172.21.15.6:6790/0 conn(0x7f22c40090f0 legacy=0x7f22c4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22d27fc640 1 -- 172.21.15.6:0/3755684943 >> v1:172.21.15.6:6790/0 conn(0x7f22c40090f0 legacy=0x7f22c4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:53:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 80, 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 216881, 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:53:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:53:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:53:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:53:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:53:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:53:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:53:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:25.948 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:25.948 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:25.948 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:53:25.948 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:25.948 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:25.948 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:25.948 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:53:25.948 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:53:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 >> v1:172.21.15.63:6812/3643207211 conn(0x7f22c0036c10 legacy=0x7f22c00390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e3c5a640 10 monclient: shutdown 2024-09-16T05:53:25.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e3c5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 >> v1:172.21.15.6:6789/0 conn(0x7f22dc09e230 legacy=0x7f22dc13bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e21f7640 1 -- 172.21.15.6:0/3755684943 reap_dead start 2024-09-16T05:53:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e21f7640 1 -- 172.21.15.6:0/3755684943 reap_dead start 2024-09-16T05:53:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 shutdown_connections 2024-09-16T05:53:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 >> 172.21.15.6:0/3755684943 conn(0x7f22dc0a1650 msgr2=0x7f22dc071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 shutdown_connections 2024-09-16T05:53:25.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:25.920+0000 7f22e3c5a640 1 -- 172.21.15.6:0/3755684943 wait complete. 2024-09-16T05:53:25.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:53:25.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:53:25.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:53:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.064+0000 7ff0dac8a640 10 monclient: get_monmap_and_config 2024-09-16T05:53:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.064+0000 7ff0dac8a640 10 monclient: build_initial_monmap 2024-09-16T05:53:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.064+0000 7ff0dac8a640 10 monclient: monmap: 2024-09-16T05:53:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:26.070682+0000 2024-09-16T05:53:26.069 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:26.070682+0000 2024-09-16T05:53:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:26.070 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 Processor -- start 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- start start 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient: init 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient(hunting): picked mon.noname-a con 0x7ff0d409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient(hunting): picked mon.noname-b con 0x7ff0d409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient(hunting): picked mon.noname-c con 0x7ff0d40a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d40a1370 con 0x7ff0d409aa30 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d409ead0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d409edb0 con 0x7ff0d409e230 2024-09-16T05:53:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient(hunting): authenticate will time out at 2067.877281s 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d8a26640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff0d409aa30 0x7ff0d409ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53148/0 (socket says 172.21.15.6:53148) 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d8a26640 1 -- 172.21.15.6:0/2425924530 learned_addr learned my addr 172.21.15.6:0/2425924530 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d3fff640 1 --1- 172.21.15.6:0/2425924530 >> v1:172.21.15.6:6790/0 conn(0x7ff0d409e230 0x7ff0d41399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34172/0 (socket says 172.21.15.6:34172) 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179861597 0 0) 0x7ff0d40a1370 con 0x7ff0d409aa30 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient(hunting): my global_id is 174237 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0c0003530 con 0x7ff0d409aa30 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3813681009 0 0) 0x7ff0d409edb0 con 0x7ff0d409e230 2024-09-16T05:53:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient(hunting): my global_id is 164333 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0d40a1370 con 0x7ff0d409e230 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2717864788 0 0) 0x7ff0c0003530 con 0x7ff0d409aa30 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 >> v1:172.21.15.6:6790/0 conn(0x7ff0d409e230 legacy=0x7ff0d41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 >> v1:172.21.15.63:6789/0 conn(0x7ff0d40a0da0 legacy=0x7ff0d413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: _finish_hunting 0 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 monclient: found mon.noname-a 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d409dd50 con 0x7ff0d409aa30 2024-09-16T05:53:26.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: _finish_auth 0 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: _check_auth_tickets 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0bc001620 con 0x7ff0d409aa30 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 5 monclient: authenticate success, global_id 174237 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: dump: 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.081 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0bc001810 con 0x7ff0d409aa30 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 1 -- 172.21.15.6:0/2425924530 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0bc001c20 con 0x7ff0d409aa30 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 4 set_mon_vals no callback set 2024-09-16T05:53:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0d37fe640 10 monclient: dump: 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient: shutdown 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- 172.21.15.6:0/2425924530 >> v1:172.21.15.6:6789/0 conn(0x7ff0d409aa30 legacy=0x7ff0d409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- 172.21.15.6:0/2425924530 shutdown_connections 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- 172.21.15.6:0/2425924530 >> 172.21.15.6:0/2425924530 conn(0x7ff0d40a1650 msgr2=0x7ff0d40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- 172.21.15.6:0/2425924530 shutdown_connections 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 -- 172.21.15.6:0/2425924530 wait complete. 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient: build_initial_monmap 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 10 monclient: monmap: 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.068+0000 7ff0dac8a640 1 Processor -- start 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 1 -- start start 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient: init 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:53:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient(hunting): picked mon.noname-c con 0x7ff0d409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient(hunting): picked mon.noname-b con 0x7ff0d409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient(hunting): picked mon.noname-a con 0x7ff0d40a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d411c5a0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d413bfc0 con 0x7ff0d409e230 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d413d2e0 con 0x7ff0d409aa30 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient(hunting): authenticate will time out at 2067.880564s 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d8a26640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff0d409aa30 0x7ff0d413b1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34184/0 (socket says 172.21.15.6:34184) 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d8a26640 1 -- 172.21.15.6:0/1813911976 learned_addr learned my addr 172.21.15.6:0/1813911976 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460176968 0 0) 0x7ff0d413d2e0 con 0x7ff0d409aa30 2024-09-16T05:53:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): my global_id is 164336 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0b4003530 con 0x7ff0d409aa30 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1207331862 0 0) 0x7ff0d411c5a0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): my global_id is 174240 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0d413d2e0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1185260989 0 0) 0x7ff0b4003530 con 0x7ff0d409aa30 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d411c5a0 con 0x7ff0d409aa30 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0bc001ba0 con 0x7ff0d409aa30 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): dump: 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2809815159 0 0) 0x7ff0d413d2e0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0b4003530 con 0x7ff0d40a0da0 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0cc001b00 con 0x7ff0d40a0da0 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): dump: 2024-09-16T05:53:26.089 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1236789179 0 0) 0x7ff0d413bfc0 con 0x7ff0d409e230 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): my global_id is 184114 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0d413d2e0 con 0x7ff0d409e230 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1574969504 0 0) 0x7ff0b4003530 con 0x7ff0d40a0da0 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 >> v1:172.21.15.6:6790/0 conn(0x7ff0d409aa30 legacy=0x7ff0d413b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 >> v1:172.21.15.63:6789/0 conn(0x7ff0d409e230 legacy=0x7ff0d413b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: _finish_hunting 0 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 monclient: found mon.a 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d411c880 con 0x7ff0d40a0da0 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: _finish_auth 0 2024-09-16T05:53:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: _check_auth_tickets 2024-09-16T05:53:26.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:26.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 5 monclient: authenticate success, global_id 174240 2024-09-16T05:53:26.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient: _renew_subs 2024-09-16T05:53:26.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:26.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d411c2c0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient: _renew_subs 2024-09-16T05:53:26.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:26.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0dac8a640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0d4135340 con 0x7ff0d40a0da0 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0cc0039a0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0cc003cf0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: dump: 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.093 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d2ffd640 4 set_mon_vals no callback set 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0caffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0caffd640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff0ac0065d0 con 0x7ff0ac002870 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff0cc003e90 con 0x7ff0d40a0da0 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7ff0cc043eb0 con 0x7ff0d40a0da0 2024-09-16T05:53:26.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d3fff640 10 monclient: get_auth_request con 0x7ff0b4036bb0 auth_method 0 2024-09-16T05:53:26.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 815959688 0 0) 0x7ff0ac0065d0 con 0x7ff0ac002870 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient(hunting): my global_id is 184117 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0b4003530 con 0x7ff0ac002870 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 602394170 0 0) 0x7ff0b4003530 con 0x7ff0ac002870 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0ac0065d0 con 0x7ff0ac002870 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0bc004da0 con 0x7ff0ac002870 2024-09-16T05:53:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 10 monclient: dump: 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4217173474 0 0) 0x7ff0ac0065d0 con 0x7ff0ac002870 2024-09-16T05:53:26.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.072+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0ac00a160 con 0x7ff0ac002870 2024-09-16T05:53:26.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.076+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff0bc007130 con 0x7ff0ac002870 2024-09-16T05:53:26.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.076+0000 7ff0d17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.076+0000 7ff0d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:26.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.076+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 >> v1:172.21.15.63:6789/0 conn(0x7ff0ac002870 legacy=0x7ff0ac004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.076+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 >> v1:172.21.15.63:6789/0 conn(0x7ff0ac002870 legacy=0x7ff0ac004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:26.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.080+0000 7ff0caffd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:53:26.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.080+0000 7ff0caffd640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff0ac009e20 con 0x7ff0ac009110 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714564943 0 0) 0x7ff0ac009e20 con 0x7ff0ac009110 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient(hunting): my global_id is 184120 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac0065d0 con 0x7ff0ac009110 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2470996438 0 0) 0x7ff0ac0065d0 con 0x7ff0ac009110 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0ac009e20 con 0x7ff0ac009110 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff0bc005460 con 0x7ff0ac009110 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient: dump: 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1657485979 0 0) 0x7ff0ac009e20 con 0x7ff0ac009110 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0ac00c390 con 0x7ff0ac009110 2024-09-16T05:53:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3959 (unknown 0 0 3430499026) 0x7ff0bc0053b0 con 0x7ff0ac009110 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 >> v1:172.21.15.63:6789/0 conn(0x7ff0ac009110 legacy=0x7ff0ac0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d17fa640 1 -- 172.21.15.6:0/1813911976 >> v1:172.21.15.63:6789/0 conn(0x7ff0ac009110 legacy=0x7ff0ac0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:53:26.102 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 80, 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 779, 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:26.103 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:26.104 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:26.105 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:26.106 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:26.107 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:26.108 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:26.109 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.110 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:26.111 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.112 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:26.113 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:26.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:26.114 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:53:26.114 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:53:26.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0caffd640 1 -- 172.21.15.6:0/1813911976 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff0b4036bb0 legacy=0x7ff0b4039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0caffd640 10 monclient: shutdown 2024-09-16T05:53:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0caffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0caffd640 1 -- 172.21.15.6:0/1813911976 >> v1:172.21.15.6:6789/0 conn(0x7ff0d40a0da0 legacy=0x7ff0d41d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d9227640 1 -- 172.21.15.6:0/1813911976 reap_dead start 2024-09-16T05:53:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0d9227640 1 -- 172.21.15.6:0/1813911976 reap_dead start 2024-09-16T05:53:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0caffd640 1 -- 172.21.15.6:0/1813911976 shutdown_connections 2024-09-16T05:53:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0caffd640 1 -- 172.21.15.6:0/1813911976 >> 172.21.15.6:0/1813911976 conn(0x7ff0d40a1650 msgr2=0x7ff0d406ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0caffd640 1 -- 172.21.15.6:0/1813911976 shutdown_connections 2024-09-16T05:53:26.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.084+0000 7ff0caffd640 1 -- 172.21.15.6:0/1813911976 wait complete. 2024-09-16T05:53:26.116 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:53:26.117 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:53:26.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:53:26.189 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=13 2024-09-16T05:53:26.190 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T05:53:26.190 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:53:26.190 INFO:tasks.workunit.client.0.smithi063.stderr:+ rm /tmp/cephtest-mon-caps-madness.bazar.keyring /tmp/cephtest-mon-caps-madness.foo.keyring /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T05:53:26.191 INFO:tasks.workunit.client.0.smithi063.stderr:+ cat 2024-09-16T05:53:26.191 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph auth import -i - 2024-09-16T05:53:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.224+0000 7fc737f89640 10 monclient: get_monmap_and_config 2024-09-16T05:53:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.224+0000 7fc737f89640 10 monclient: build_initial_monmap 2024-09-16T05:53:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.224+0000 7fc737f89640 10 monclient: monmap: 2024-09-16T05:53:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:26.229940+0000 2024-09-16T05:53:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:26.229940+0000 2024-09-16T05:53:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:26.229 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 Processor -- start 2024-09-16T05:53:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- start start 2024-09-16T05:53:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient: init 2024-09-16T05:53:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:53:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient(hunting): picked mon.noname-b con 0x7fc730134880 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:26.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient(hunting): picked mon.noname-a con 0x7fc7300a0f20 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient(hunting): picked mon.noname-c con 0x7fc73013ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7300a3180 con 0x7fc7300a0f20 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73009f2a0 con 0x7fc73013ae70 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7300a0260 con 0x7fc730134880 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient(hunting): authenticate will time out at 2068.037123s 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc735524640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc7300a0f20 0x7fc730139750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53156/0 (socket says 172.21.15.6:53156) 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc735524640 1 -- 172.21.15.6:0/2031071017 learned_addr learned my addr 172.21.15.6:0/2031071017 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732571850 0 0) 0x7fc720001120 con 0x7fc7300a0f20 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient(hunting): my global_id is 174246 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc71c003530 con 0x7fc7300a0f20 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2168149060 0 0) 0x7fc7300a0260 con 0x7fc730134880 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient(hunting): my global_id is 164342 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc720001120 con 0x7fc730134880 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3243393960 0 0) 0x7fc71c003530 con 0x7fc7300a0f20 2024-09-16T05:53:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 >> v1:172.21.15.6:6790/0 conn(0x7fc730134880 legacy=0x7fc730134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 >> v1:172.21.15.63:6789/0 conn(0x7fc73013ae70 legacy=0x7fc73013d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: _finish_hunting 0 2024-09-16T05:53:26.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 monclient: found mon.noname-a 2024-09-16T05:53:26.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:26.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7300a0540 con 0x7fc7300a0f20 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: _finish_auth 0 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: _check_auth_tickets 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 5 monclient: authenticate success, global_id 174246 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc7200017f0 con 0x7fc7300a0f20 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: dump: 2024-09-16T05:53:26.237 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc720001a50 con 0x7fc7300a0f20 2024-09-16T05:53:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 1 -- 172.21.15.6:0/2031071017 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc7200048b0 con 0x7fc7300a0f20 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 4 set_mon_vals no callback set 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc734d23640 10 monclient: dump: 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.239 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient: shutdown 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- 172.21.15.6:0/2031071017 >> v1:172.21.15.6:6789/0 conn(0x7fc7300a0f20 legacy=0x7fc730139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- 172.21.15.6:0/2031071017 shutdown_connections 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- 172.21.15.6:0/2031071017 >> 172.21.15.6:0/2031071017 conn(0x7fc7300a37e0 msgr2=0x7fc7300a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- 172.21.15.6:0/2031071017 shutdown_connections 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 -- 172.21.15.6:0/2031071017 wait complete. 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient: build_initial_monmap 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 10 monclient: monmap: 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:26.240 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.228+0000 7fc737f89640 1 Processor -- start 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 1 -- start start 2024-09-16T05:53:26.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: init 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient(hunting): picked mon.noname-c con 0x7fc7300a0f20 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient(hunting): picked mon.noname-a con 0x7fc730134880 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient(hunting): picked mon.noname-b con 0x7fc73013ae70 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73011d330 con 0x7fc730134880 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc73011d610 con 0x7fc73013ae70 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7301cc6e0 con 0x7fc7300a0f20 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient(hunting): authenticate will time out at 2068.040706s 2024-09-16T05:53:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc735524640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc730134880 0x7fc7301cbfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53162/0 (socket says 172.21.15.6:53162) 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc735524640 1 -- 172.21.15.6:0/1481910108 learned_addr learned my addr 172.21.15.6:0/1481910108 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc735d25640 1 --1- 172.21.15.6:0/1481910108 >> v1:172.21.15.6:6790/0 conn(0x7fc7300a0f20 0x7fc7301cb8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34208/0 (socket says 172.21.15.6:34208) 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1571674024 0 0) 0x7fc73011d330 con 0x7fc730134880 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): my global_id is 174249 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc714003530 con 0x7fc730134880 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182161599 0 0) 0x7fc7301cc6e0 con 0x7fc7300a0f20 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): my global_id is 164345 2024-09-16T05:53:26.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc73011d330 con 0x7fc7300a0f20 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2143396420 0 0) 0x7fc73011d330 con 0x7fc7300a0f20 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7301cc6e0 con 0x7fc7300a0f20 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc72c001a40 con 0x7fc7300a0f20 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): dump: 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.247 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3108300851 0 0) 0x7fc714003530 con 0x7fc730134880 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc73011d330 con 0x7fc730134880 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc720000ee0 con 0x7fc730134880 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): dump: 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.248 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.249 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.249 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.249 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.249 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.249 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.250 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 969776910 0 0) 0x7fc73011d610 con 0x7fc73013ae70 2024-09-16T05:53:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient(hunting): my global_id is 184129 2024-09-16T05:53:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc714003530 con 0x7fc73013ae70 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1447561508 0 0) 0x7fc7301cc6e0 con 0x7fc7300a0f20 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 >> v1:172.21.15.63:6789/0 conn(0x7fc73013ae70 legacy=0x7fc7301d3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 >> v1:172.21.15.6:6789/0 conn(0x7fc730134880 legacy=0x7fc7301cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient: _finish_hunting 0 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 monclient: found mon.c 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73006dd30 con 0x7fc7300a0f20 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient: _finish_auth 0 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient: _check_auth_tickets 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:26.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 5 monclient: authenticate success, global_id 164345 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: _renew_subs 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc73011dbd0 con 0x7fc7300a0f20 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc72c001ca0 con 0x7fc7300a0f20 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc72c003ad0 con 0x7fc7300a0f20 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: _renew_subs 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc73013e710 con 0x7fc7300a0f20 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc727fff640 4 set_mon_vals no callback set 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 10 monclient: dump: 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7300008e0 con 0x7fc7300a0f20 2024-09-16T05:53:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc72c003cc0 con 0x7fc7300a0f20 2024-09-16T05:53:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fc72c044df0 con 0x7fc7300a0f20 2024-09-16T05:53:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.232+0000 7fc735524640 10 monclient: get_auth_request con 0x7fc714036be0 auth_method 0 2024-09-16T05:53:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.236+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc72c0153a0 con 0x7fc7300a0f20 2024-09-16T05:53:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.236+0000 7fc7267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.236+0000 7fc7267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:26.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.306+0000 7f7c1e2c1640 10 monclient: get_monmap_and_config 2024-09-16T05:53:26.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.306+0000 7f7c1e2c1640 10 monclient: build_initial_monmap 2024-09-16T05:53:26.310 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.306+0000 7f7c1e2c1640 10 monclient: monmap: 2024-09-16T05:53:26.310 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:26.310 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.311 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:53:26.311446+0000 2024-09-16T05:53:26.311 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:53:26.311446+0000 2024-09-16T05:53:26.311 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:26.311 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.311 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:26.311 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:26.311 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:26.311 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.317 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.306+0000 7f7c1e2c1640 1 Processor -- start 2024-09-16T05:53:26.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.306+0000 7f7c1e2c1640 1 -- start start 2024-09-16T05:53:26.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.306+0000 7f7c1e2c1640 10 monclient: init 2024-09-16T05:53:26.318 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c180a18b0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c1806c6b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c1809bb20 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c1809b140 con 0x7f7c1809bb20 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c1809ae60 con 0x7f7c1806c6b0 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c180a3310 con 0x7f7c180a18b0 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): authenticate will time out at 2063.088534s 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c177fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7c1806c6b0 0x7f7c1813bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48894/0 (socket says 172.21.15.63:48894) 2024-09-16T05:53:26.319 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c177fe640 1 -- 172.21.15.63:0/2022737241 learned_addr learned my addr 172.21.15.63:0/2022737241 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785417642 0 0) 0x7f7c1809ae60 con 0x7f7c1806c6b0 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): my global_id is 184132 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c00003530 con 0x7f7c1806c6b0 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3400056828 0 0) 0x7f7c180a3310 con 0x7f7c180a18b0 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): my global_id is 164348 2024-09-16T05:53:26.320 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c1809ae60 con 0x7f7c180a18b0 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3709073912 0 0) 0x7f7c1809b140 con 0x7f7c1809bb20 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient(hunting): my global_id is 174252 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c180a3310 con 0x7f7c1809bb20 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2258466962 0 0) 0x7f7c00003530 con 0x7f7c1806c6b0 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 >> v1:172.21.15.6:6790/0 conn(0x7f7c180a18b0 legacy=0x7f7c180a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 >> v1:172.21.15.6:6789/0 conn(0x7f7c1809bb20 legacy=0x7f7c1813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 monclient: found mon.noname-c 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:26.321 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c180a23a0 con 0x7f7c1806c6b0 2024-09-16T05:53:26.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: _finish_auth 0 2024-09-16T05:53:26.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:53:26.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:26.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c0c001620 con 0x7f7c1806c6b0 2024-09-16T05:53:26.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.322 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: dump: 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 5 monclient: authenticate success, global_id 184132 2024-09-16T05:53:26.323 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c0c001880 con 0x7f7c1806c6b0 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 4 set_mon_vals no callback set 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 1 -- 172.21.15.63:0/2022737241 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c0c001c50 con 0x7f7c1806c6b0 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c16ffd640 10 monclient: dump: 2024-09-16T05:53:26.324 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: shutdown 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:26.325 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/2022737241 >> v1:172.21.15.63:6789/0 conn(0x7f7c1806c6b0 legacy=0x7f7c1813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/2022737241 shutdown_connections 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/2022737241 >> 172.21.15.63:0/2022737241 conn(0x7f7c180a64a0 msgr2=0x7f7c180a29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/2022737241 shutdown_connections 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/2022737241 wait complete. 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: build_initial_monmap 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: monmap: 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:26.326 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 Processor -- start 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- start start 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: init 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c1806c6b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c1809bb20 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:26.327 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c180a18b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c18138ba0 con 0x7f7c180a18b0 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c18138e80 con 0x7f7c1806c6b0 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c18137470 con 0x7f7c1809bb20 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.310+0000 7f7c1e2c1640 10 monclient(hunting): authenticate will time out at 2063.091756s 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1c85e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7c180a18b0 0x7f7c18135a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.63:49710/0 (socket says 172.21.15.63:49710) 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c17fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7c1806c6b0 0x7f7c18134c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48908/0 (socket says 172.21.15.63:48908) 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1c85e640 1 -- 172.21.15.63:0/644205241 learned_addr learned my addr 172.21.15.63:0/644205241 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734119998 0 0) 0x7f7c18138e80 con 0x7f7c1806c6b0 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.328 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): my global_id is 184135 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c04003530 con 0x7f7c1806c6b0 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 533019481 0 0) 0x7f7c04003530 con 0x7f7c1806c6b0 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c18138e80 con 0x7f7c1806c6b0 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119545528 0 0) 0x7f7c18138ba0 con 0x7f7c180a18b0 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): my global_id is 174255 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c04003530 con 0x7f7c180a18b0 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c08001900 con 0x7f7c1806c6b0 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:26.329 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): dump: 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760294539 0 0) 0x7f7c18137470 con 0x7f7c1809bb20 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.330 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient(hunting): my global_id is 164351 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c18138ba0 con 0x7f7c1809bb20 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 297658279 0 0) 0x7f7c18138e80 con 0x7f7c1806c6b0 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 >> v1:172.21.15.6:6790/0 conn(0x7f7c1809bb20 legacy=0x7f7c18135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 >> v1:172.21.15.6:6789/0 conn(0x7f7c180a18b0 legacy=0x7f7c18135a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 monclient: found mon.b 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1809da80 con 0x7f7c1806c6b0 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient: _finish_auth 0 2024-09-16T05:53:26.331 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 5 monclient: authenticate success, global_id 184135 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 10 monclient: _renew_subs 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c18139160 con 0x7f7c1806c6b0 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 10 monclient: _renew_subs 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c181371f0 con 0x7f7c1806c6b0 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c08001d30 con 0x7f7c1806c6b0 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7c08003a70 con 0x7f7c1806c6b0 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.332 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c167fc640 4 set_mon_vals no callback set 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 10 monclient: dump: 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.333 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f7c08003c90 con 0x7f7c1806c6b0 2024-09-16T05:53:26.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf8005090 con 0x7f7c1806c6b0 2024-09-16T05:53:26.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f7c08044f90 con 0x7f7c1806c6b0 2024-09-16T05:53:26.334 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.314+0000 7f7c177fe640 10 monclient: get_auth_request con 0x7f7c04036c70 auth_method 0 2024-09-16T05:53:26.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.318+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f7c080155c0 con 0x7f7c1806c6b0 2024-09-16T05:53:26.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.318+0000 7f7c14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.335 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.318+0000 7f7c14ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:26.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.456+0000 7fc737f89640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:53:26.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.456+0000 7fc737f89640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:53:26.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.456+0000 7fc737f89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:26.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.456+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc73009f4e0 con 0x7fc7300a0f20 2024-09-16T05:53:26.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc7267fc640 1 -- 172.21.15.6:0/1481910108 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fc72c019050 con 0x7fc7300a0f20 2024-09-16T05:53:26.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc7267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:53:26.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc7267fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:26.462 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.460+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.460+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=2398577760,config=3541143443,health=2127886669,kv=2046952910,logm=4040126160}) 2024-09-16T05:53:26.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.460+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=9,logm=51} crc {auth=283384054,config=1802735181,health=1476039693,kv=2959870627,logm=643629216}) 2024-09-16T05:53:26.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.460+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.460+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=2398577760,config=3541143443,health=2127886669,kv=2046952910,logm=4040126160}) 2024-09-16T05:53:26.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.460+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=3235260102,config=3541143443,health=2127886669,kv=2441816623,logm=3390871087}) 2024-09-16T05:53:26.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc714036be0 legacy=0x7fc7140390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc737f89640 10 monclient: shutdown 2024-09-16T05:53:26.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc737f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:26.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 >> v1:172.21.15.6:6790/0 conn(0x7fc7300a0f20 legacy=0x7fc7301cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 shutdown_connections 2024-09-16T05:53:26.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 >> 172.21.15.6:0/1481910108 conn(0x7fc7300a37e0 msgr2=0x7fc7300727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:26.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 shutdown_connections 2024-09-16T05:53:26.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:26.460+0000 7fc737f89640 1 -- 172.21.15.6:0/1481910108 wait complete. 2024-09-16T05:53:26.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914244545}) 2024-09-16T05:53:26.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4203491287}) 2024-09-16T05:53:26.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914244545}) 2024-09-16T05:53:26.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=312904786}) 2024-09-16T05:53:26.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850675029}) 2024-09-16T05:53:26.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1112843187}) 2024-09-16T05:53:26.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850675029}) 2024-09-16T05:53:26.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.464+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594814084}) 2024-09-16T05:53:26.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91483864}) 2024-09-16T05:53:26.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1381270606}) 2024-09-16T05:53:26.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91483864}) 2024-09-16T05:53:26.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3929286994}) 2024-09-16T05:53:26.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141934176}) 2024-09-16T05:53:26.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820034373}) 2024-09-16T05:53:26.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141934176}) 2024-09-16T05:53:26.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.468+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3326473941}) 2024-09-16T05:53:26.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114039143}) 2024-09-16T05:53:26.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812833505}) 2024-09-16T05:53:26.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114039143}) 2024-09-16T05:53:26.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606305526}) 2024-09-16T05:53:26.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675566297}) 2024-09-16T05:53:26.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125155024}) 2024-09-16T05:53:26.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675566297}) 2024-09-16T05:53:26.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.472+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983224927}) 2024-09-16T05:53:26.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3533601204}) 2024-09-16T05:53:26.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568546050}) 2024-09-16T05:53:26.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3533601204}) 2024-09-16T05:53:26.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3208115072}) 2024-09-16T05:53:26.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215538716}) 2024-09-16T05:53:26.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1479863315}) 2024-09-16T05:53:26.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215538716}) 2024-09-16T05:53:26.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.476+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149775451}) 2024-09-16T05:53:26.481 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:53:26.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050121950}) 2024-09-16T05:53:26.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119096535}) 2024-09-16T05:53:26.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050121950}) 2024-09-16T05:53:26.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201667731}) 2024-09-16T05:53:26.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800401930}) 2024-09-16T05:53:26.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=721683356}) 2024-09-16T05:53:26.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800401930}) 2024-09-16T05:53:26.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.480+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3475592455}) 2024-09-16T05:53:26.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013150135}) 2024-09-16T05:53:26.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989954238}) 2024-09-16T05:53:26.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013150135}) 2024-09-16T05:53:26.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=534098243}) 2024-09-16T05:53:26.487 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.487 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413636741}) 2024-09-16T05:53:26.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1914578696}) 2024-09-16T05:53:26.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413636741}) 2024-09-16T05:53:26.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.484+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210458628}) 2024-09-16T05:53:26.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453182881}) 2024-09-16T05:53:26.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809735101}) 2024-09-16T05:53:26.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453182881}) 2024-09-16T05:53:26.490 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264046077}) 2024-09-16T05:53:26.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907047214}) 2024-09-16T05:53:26.492 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=923285897}) 2024-09-16T05:53:26.492 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.492 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907047214}) 2024-09-16T05:53:26.492 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=233988980}) 2024-09-16T05:53:26.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175179130}) 2024-09-16T05:53:26.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2381437760}) 2024-09-16T05:53:26.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175179130}) 2024-09-16T05:53:26.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.488+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844236065}) 2024-09-16T05:53:26.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260177321}) 2024-09-16T05:53:26.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1053840709}) 2024-09-16T05:53:26.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260177321}) 2024-09-16T05:53:26.495 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646213922}) 2024-09-16T05:53:26.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368045121}) 2024-09-16T05:53:26.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4189628116}) 2024-09-16T05:53:26.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368045121}) 2024-09-16T05:53:26.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.492+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046650441}) 2024-09-16T05:53:26.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266971619}) 2024-09-16T05:53:26.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1332418819}) 2024-09-16T05:53:26.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266971619}) 2024-09-16T05:53:26.499 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404435659}) 2024-09-16T05:53:26.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346983985}) 2024-09-16T05:53:26.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=232043168}) 2024-09-16T05:53:26.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346983985}) 2024-09-16T05:53:26.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.496+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3801879743}) 2024-09-16T05:53:26.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186489466}) 2024-09-16T05:53:26.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268219002}) 2024-09-16T05:53:26.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186489466}) 2024-09-16T05:53:26.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4089034720}) 2024-09-16T05:53:26.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392343565}) 2024-09-16T05:53:26.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2813764103}) 2024-09-16T05:53:26.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392343565}) 2024-09-16T05:53:26.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.500+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604409363}) 2024-09-16T05:53:26.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3110214844}) 2024-09-16T05:53:26.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4250537129}) 2024-09-16T05:53:26.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3110214844}) 2024-09-16T05:53:26.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251846754}) 2024-09-16T05:53:26.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2286049827}) 2024-09-16T05:53:26.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2920091528}) 2024-09-16T05:53:26.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2286049827}) 2024-09-16T05:53:26.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.504+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2504330361}) 2024-09-16T05:53:26.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251650093}) 2024-09-16T05:53:26.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2087995602}) 2024-09-16T05:53:26.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251650093}) 2024-09-16T05:53:26.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4005547039}) 2024-09-16T05:53:26.512 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1812064824}) 2024-09-16T05:53:26.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=380116036}) 2024-09-16T05:53:26.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1812064824}) 2024-09-16T05:53:26.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.508+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3021535672}) 2024-09-16T05:53:26.514 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.514 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740049523}) 2024-09-16T05:53:26.514 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986690268}) 2024-09-16T05:53:26.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740049523}) 2024-09-16T05:53:26.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1882724782}) 2024-09-16T05:53:26.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554708118}) 2024-09-16T05:53:26.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1582178096}) 2024-09-16T05:53:26.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554708118}) 2024-09-16T05:53:26.517 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.512+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018184247}) 2024-09-16T05:53:26.518 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.518 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2284433641}) 2024-09-16T05:53:26.518 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3071833397}) 2024-09-16T05:53:26.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2284433641}) 2024-09-16T05:53:26.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1154142643}) 2024-09-16T05:53:26.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828077481}) 2024-09-16T05:53:26.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=304356337}) 2024-09-16T05:53:26.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828077481}) 2024-09-16T05:53:26.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.516+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2435109349}) 2024-09-16T05:53:26.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425331481}) 2024-09-16T05:53:26.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915492965}) 2024-09-16T05:53:26.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425331481}) 2024-09-16T05:53:26.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2242521598}) 2024-09-16T05:53:26.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820956563}) 2024-09-16T05:53:26.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2231144332}) 2024-09-16T05:53:26.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820956563}) 2024-09-16T05:53:26.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.520+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2949598593}) 2024-09-16T05:53:26.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886306552}) 2024-09-16T05:53:26.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1037018435}) 2024-09-16T05:53:26.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886306552}) 2024-09-16T05:53:26.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087058224}) 2024-09-16T05:53:26.527 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.527 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106817950}) 2024-09-16T05:53:26.527 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3839201295}) 2024-09-16T05:53:26.527 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.528 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106817950}) 2024-09-16T05:53:26.528 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.524+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807137210}) 2024-09-16T05:53:26.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554264595}) 2024-09-16T05:53:26.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506932894}) 2024-09-16T05:53:26.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554264595}) 2024-09-16T05:53:26.529 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232499996}) 2024-09-16T05:53:26.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649681121}) 2024-09-16T05:53:26.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=787319276}) 2024-09-16T05:53:26.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649681121}) 2024-09-16T05:53:26.531 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2778667809}) 2024-09-16T05:53:26.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826848840}) 2024-09-16T05:53:26.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2298975764}) 2024-09-16T05:53:26.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826848840}) 2024-09-16T05:53:26.533 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.528+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323633610}) 2024-09-16T05:53:26.534 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165901736}) 2024-09-16T05:53:26.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=807674438}) 2024-09-16T05:53:26.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165901736}) 2024-09-16T05:53:26.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2351394624}) 2024-09-16T05:53:26.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194650484}) 2024-09-16T05:53:26.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3025513025}) 2024-09-16T05:53:26.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194650484}) 2024-09-16T05:53:26.537 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.532+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371223712}) 2024-09-16T05:53:26.538 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3195202817}) 2024-09-16T05:53:26.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205471053}) 2024-09-16T05:53:26.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3195202817}) 2024-09-16T05:53:26.539 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=591386046}) 2024-09-16T05:53:26.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551205416}) 2024-09-16T05:53:26.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358956433}) 2024-09-16T05:53:26.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551205416}) 2024-09-16T05:53:26.540 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.536+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745124780}) 2024-09-16T05:53:26.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523980313}) 2024-09-16T05:53:26.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3264198695}) 2024-09-16T05:53:26.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523980313}) 2024-09-16T05:53:26.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788032527}) 2024-09-16T05:53:26.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135546017}) 2024-09-16T05:53:26.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3095726564}) 2024-09-16T05:53:26.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135546017}) 2024-09-16T05:53:26.544 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.540+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352040242}) 2024-09-16T05:53:26.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163738660}) 2024-09-16T05:53:26.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062665759}) 2024-09-16T05:53:26.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163738660}) 2024-09-16T05:53:26.546 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3206458563}) 2024-09-16T05:53:26.547 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411017413}) 2024-09-16T05:53:26.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1983831732}) 2024-09-16T05:53:26.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411017413}) 2024-09-16T05:53:26.548 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.544+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925122356}) 2024-09-16T05:53:26.549 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.549 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285178001}) 2024-09-16T05:53:26.549 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124549381}) 2024-09-16T05:53:26.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285178001}) 2024-09-16T05:53:26.550 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1377281718}) 2024-09-16T05:53:26.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494271462}) 2024-09-16T05:53:26.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=232626306}) 2024-09-16T05:53:26.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494271462}) 2024-09-16T05:53:26.552 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.548+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351667867}) 2024-09-16T05:53:26.553 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2416645369}) 2024-09-16T05:53:26.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1063111234}) 2024-09-16T05:53:26.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2416645369}) 2024-09-16T05:53:26.554 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250331363}) 2024-09-16T05:53:26.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.550+0000 7f7c1e2c1640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-16T05:53:26.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.550+0000 7f7c1e2c1640 10 monclient: _send_command 2 [{"prefix": "auth import"}] 2024-09-16T05:53:26.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.550+0000 7f7c1e2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.555 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.550+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "auth import"} v 0) -- 0x7f7bf8005380 con 0x7f7c1806c6b0 2024-09-16T05:53:26.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090388535}) 2024-09-16T05:53:26.556 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762192312}) 2024-09-16T05:53:26.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090388535}) 2024-09-16T05:53:26.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1356571715}) 2024-09-16T05:53:26.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853189223}) 2024-09-16T05:53:26.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=726170024}) 2024-09-16T05:53:26.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.552+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853189223}) 2024-09-16T05:53:26.558 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=589560101}) 2024-09-16T05:53:26.559 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.559 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844698637}) 2024-09-16T05:53:26.559 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2229716229}) 2024-09-16T05:53:26.559 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844698637}) 2024-09-16T05:53:26.560 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282877502}) 2024-09-16T05:53:26.561 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.561 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922167640}) 2024-09-16T05:53:26.561 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2218479925}) 2024-09-16T05:53:26.561 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.561 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922167640}) 2024-09-16T05:53:26.561 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.556+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458764320}) 2024-09-16T05:53:26.563 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.560+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 auth entities have invalid capabilities (AUTH_BAD_CAPS) 2024-09-16T05:53:26.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.560+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.560+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207484875}) 2024-09-16T05:53:26.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.560+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3963894114}) 2024-09-16T05:53:26.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.560+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.560+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207484875}) 2024-09-16T05:53:26.564 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.560+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1774733750}) 2024-09-16T05:53:26.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.566+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "auth import"}]=0 v30) ==== 59+0+0 (unknown 1863088631 0 0) 0x7f7c08014ec0 con 0x7f7c1806c6b0 2024-09-16T05:53:26.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.566+0000 7f7c14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth import"}] 2024-09-16T05:53:26.571 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.566+0000 7f7c14ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:26.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.568+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.568+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300322439}) 2024-09-16T05:53:26.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.568+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2852751488}) 2024-09-16T05:53:26.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.568+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.568+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300322439}) 2024-09-16T05:53:26.572 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.568+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3390966378}) 2024-09-16T05:53:26.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779629197}) 2024-09-16T05:53:26.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772687102}) 2024-09-16T05:53:26.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779629197}) 2024-09-16T05:53:26.574 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845617748}) 2024-09-16T05:53:26.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772170728}) 2024-09-16T05:53:26.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=534588186}) 2024-09-16T05:53:26.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772170728}) 2024-09-16T05:53:26.576 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.572+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342023163}) 2024-09-16T05:53:26.577 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.577 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1010566580}) 2024-09-16T05:53:26.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1611684017}) 2024-09-16T05:53:26.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1010566580}) 2024-09-16T05:53:26.578 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1579808813}) 2024-09-16T05:53:26.579 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.579 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3919317449}) 2024-09-16T05:53:26.579 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2792628889}) 2024-09-16T05:53:26.579 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.580 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3919317449}) 2024-09-16T05:53:26.580 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295705383}) 2024-09-16T05:53:26.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.576+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473045524}) 2024-09-16T05:53:26.581 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1331438356}) 2024-09-16T05:53:26.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473045524}) 2024-09-16T05:53:26.582 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944112904}) 2024-09-16T05:53:26.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=628592951}) 2024-09-16T05:53:26.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1889986128}) 2024-09-16T05:53:26.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=628592951}) 2024-09-16T05:53:26.583 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641134881}) 2024-09-16T05:53:26.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1987152331}) 2024-09-16T05:53:26.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3523446435}) 2024-09-16T05:53:26.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1987152331}) 2024-09-16T05:53:26.585 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.580+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2296256667}) 2024-09-16T05:53:26.587 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.587 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332278363}) 2024-09-16T05:53:26.587 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4248955344}) 2024-09-16T05:53:26.587 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.587 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332278363}) 2024-09-16T05:53:26.587 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4160211567}) 2024-09-16T05:53:26.588 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711420313}) 2024-09-16T05:53:26.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1396620280}) 2024-09-16T05:53:26.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711420313}) 2024-09-16T05:53:26.589 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.584+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2673425417}) 2024-09-16T05:53:26.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.588+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.591 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.588+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=86,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=3172371506,mdsmap=3917278932,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3374435251}) 2024-09-16T05:53:26.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.588+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=84,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1395064774,mdsmap=3140514009,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3536190986}) 2024-09-16T05:53:26.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.588+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.588+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=86,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=3172371506,mdsmap=3917278932,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=3374435251}) 2024-09-16T05:53:26.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.588+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=3993917704,mdsmap=1832746017,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=1730094979}) 2024-09-16T05:53:26.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.596+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.601 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.596+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgrstat=3056060707,mon_config_key=2880090005,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1375606969}) 2024-09-16T05:53:26.602 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.600+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgrstat=1604549851,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1506556195}) 2024-09-16T05:53:26.602 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.600+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:53:26.602 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.600+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=13,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgrstat=3056060707,mon_config_key=2880090005,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1375606969}) 2024-09-16T05:53:26.602 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:53:26.600+0000 7f850989e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=11,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=1604549851,mon_config_key=782542404,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=324217385}) 2024-09-16T05:53:26.727 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.722+0000 7f7c1e2c1640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-16T05:53:26.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.722+0000 7f7c1e2c1640 10 monclient: _send_command 3 [{"prefix": "auth import"}] 2024-09-16T05:53:26.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.722+0000 7f7c1e2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.728 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.722+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "auth import"} v 0) -- 0x7f7bf80010a0 con 0x7f7c1806c6b0 2024-09-16T05:53:26.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.730+0000 7f7c14ff9640 1 -- 172.21.15.63:0/644205241 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "auth import"}]=0 v31) ==== 59+0+0 (unknown 1531835459 0 0) 0x7f7c0801da70 con 0x7f7c1806c6b0 2024-09-16T05:53:26.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.730+0000 7f7c14ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth import"}] 2024-09-16T05:53:26.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.730+0000 7f7c14ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:53:26.736 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.734+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7c04036c70 legacy=0x7f7c04039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.734+0000 7f7c1e2c1640 10 monclient: shutdown 2024-09-16T05:53:26.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.734+0000 7f7c1e2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:26.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.734+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 >> v1:172.21.15.63:6789/0 conn(0x7f7c1806c6b0 legacy=0x7f7c18134c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.734+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 shutdown_connections 2024-09-16T05:53:26.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.734+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 >> 172.21.15.63:0/644205241 conn(0x7f7c180a64a0 msgr2=0x7f7c180a4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:26.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.734+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 shutdown_connections 2024-09-16T05:53:26.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.734+0000 7f7c1e2c1640 1 -- 172.21.15.63:0/644205241 wait complete. 2024-09-16T05:53:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph health 2024-09-16T05:53:26.752 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep AUTH_BAD_CAP 2024-09-16T05:53:26.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient: get_monmap_and_config 2024-09-16T05:53:26.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient: build_initial_monmap 2024-09-16T05:53:26.861 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient: monmap: 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:53:26.862591+0000 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:53:26.862591+0000 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:26.862 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 1 Processor -- start 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 1 -- start start 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient: init 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient(hunting): picked mon.noname-c con 0x7f164809f790 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient(hunting): picked mon.noname-b con 0x7f16480a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient(hunting): picked mon.noname-a con 0x7f164813b0e0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16480a14d0 con 0x7f164813b0e0 2024-09-16T05:53:26.865 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f164809e5f0 con 0x7f164809f790 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16480a0c30 con 0x7f16480a0280 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ecca640 10 monclient(hunting): authenticate will time out at 2063.639635s 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ca66640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f164809f790 0x7f164809fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48922/0 (socket says 172.21.15.63:48922) 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.858+0000 7f164ca66640 1 -- 172.21.15.63:0/1474031288 learned_addr learned my addr 172.21.15.63:0/1474031288 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 502097815 0 0) 0x7f164809e5f0 con 0x7f164809f790 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.866 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient(hunting): my global_id is 184138 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1630003530 con 0x7f164809f790 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 828039363 0 0) 0x7f1630003530 con 0x7f164809f790 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 >> v1:172.21.15.6:6790/0 conn(0x7f16480a0280 legacy=0x7f16481399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 >> v1:172.21.15.6:6789/0 conn(0x7f164813b0e0 legacy=0x7f164813d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: _finish_hunting 0 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 monclient: found mon.noname-c 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.867 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16480a06b0 con 0x7f164809f790 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: _finish_auth 0 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: _check_auth_tickets 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f163c001620 con 0x7f164809f790 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: dump: 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.868 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 5 monclient: authenticate success, global_id 184138 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f163c001980 con 0x7f164809f790 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 1 -- 172.21.15.63:0/1474031288 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f163c0048b0 con 0x7f164809f790 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f16477fe640 10 monclient: dump: 2024-09-16T05:53:26.869 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 4 set_mon_vals no callback set 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient: shutdown 2024-09-16T05:53:26.870 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:26.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- 172.21.15.63:0/1474031288 >> v1:172.21.15.63:6789/0 conn(0x7f164809f790 legacy=0x7f164809fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- 172.21.15.63:0/1474031288 shutdown_connections 2024-09-16T05:53:26.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- 172.21.15.63:0/1474031288 >> 172.21.15.63:0/1474031288 conn(0x7f16480a1890 msgr2=0x7f16480a1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:26.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- 172.21.15.63:0/1474031288 shutdown_connections 2024-09-16T05:53:26.871 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- 172.21.15.63:0/1474031288 wait complete. 2024-09-16T05:53:26.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient: build_initial_monmap 2024-09-16T05:53:26.874 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient: monmap: 2024-09-16T05:53:26.874 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:26.874 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 Processor -- start 2024-09-16T05:53:26.875 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- start start 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient: init 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient(hunting): picked mon.noname-a con 0x7f164809f790 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient(hunting): picked mon.noname-c con 0x7f16480a0280 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient(hunting): picked mon.noname-b con 0x7f164813b0e0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1648121560 con 0x7f164809f790 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1648120390 con 0x7f164813b0e0 2024-09-16T05:53:26.876 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f164809db70 con 0x7f16480a0280 2024-09-16T05:53:26.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:26.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164ecca640 10 monclient(hunting): authenticate will time out at 2063.643681s 2024-09-16T05:53:26.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164d267640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f164813b0e0 0x7f16481d3c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48934/0 (socket says 172.21.15.63:48934) 2024-09-16T05:53:26.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.862+0000 7f164d267640 1 -- 172.21.15.63:0/11945520 learned_addr learned my addr 172.21.15.63:0/11945520 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:26.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 154300605 0 0) 0x7f1648120390 con 0x7f164813b0e0 2024-09-16T05:53:26.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): my global_id is 184141 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1634003530 con 0x7f164813b0e0 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1388910144 0 0) 0x7f1634003530 con 0x7f164813b0e0 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1648120390 con 0x7f164813b0e0 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1640001880 con 0x7f164813b0e0 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:26.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): dump: 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:26.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3370342753 0 0) 0x7f164809db70 con 0x7f16480a0280 2024-09-16T05:53:26.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): my global_id is 164357 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1634003530 con 0x7f16480a0280 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912042180 0 0) 0x7f1648121560 con 0x7f164809f790 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient(hunting): my global_id is 174261 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f164809db70 con 0x7f164809f790 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2489724483 0 0) 0x7f1648120390 con 0x7f164813b0e0 2024-09-16T05:53:26.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 >> v1:172.21.15.6:6790/0 conn(0x7f16480a0280 legacy=0x7f16481d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 >> v1:172.21.15.6:6789/0 conn(0x7f164809f790 legacy=0x7f16481cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient: _finish_hunting 0 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 monclient: found mon.b 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1648121b20 con 0x7f164813b0e0 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient: _finish_auth 0 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient: _check_auth_tickets 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 5 monclient: authenticate success, global_id 184141 2024-09-16T05:53:26.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 10 monclient: _renew_subs 2024-09-16T05:53:26.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1648121280 con 0x7f164813b0e0 2024-09-16T05:53:26.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 10 monclient: _renew_subs 2024-09-16T05:53:26.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1648120110 con 0x7f164813b0e0 2024-09-16T05:53:26.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1640003620 con 0x7f164813b0e0 2024-09-16T05:53:26.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:26.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1640003970 con 0x7f164813b0e0 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f1646ffd640 4 set_mon_vals no callback set 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 10 monclient: dump: 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:26.887 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1640003b10 con 0x7f164813b0e0 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1628005090 con 0x7f164813b0e0 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f1640044da0 con 0x7f164813b0e0 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.866+0000 7f164ca66640 10 monclient: get_auth_request con 0x7f1634036d50 auth_method 0 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.870+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1640015370 con 0x7f164813b0e0 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.870+0000 7f16457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:26.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:26.870+0000 7f16457fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:27.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.070+0000 7f164ecca640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-16T05:53:27.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.070+0000 7f164ecca640 10 monclient: _send_command 2 [{"prefix": "health"}] 2024-09-16T05:53:27.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.070+0000 7f164ecca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.070+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "health"} v 0) -- 0x7f1628005b90 con 0x7f164813b0e0 2024-09-16T05:53:27.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.070+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "health"}]=0 v0) ==== 54+0+99 (unknown 2555570491 0 746166484) 0x7f1640014c70 con 0x7f164813b0e0 2024-09-16T05:53:27.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.070+0000 7f16457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health"}] 2024-09-16T05:53:27.074 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.070+0000 7f16457fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:27.206 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.202+0000 7f164ecca640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-16T05:53:27.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.202+0000 7f164ecca640 10 monclient: _send_command 3 [{"prefix": "health"}] 2024-09-16T05:53:27.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.202+0000 7f164ecca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.202+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "health"} v 0) -- 0x7f1628005d60 con 0x7f164813b0e0 2024-09-16T05:53:27.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.202+0000 7f16457fa640 1 -- 172.21.15.63:0/11945520 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "health"}]=0 v0) ==== 54+0+99 (unknown 2555570491 0 746166484) 0x7f1640018fc0 con 0x7f164813b0e0 2024-09-16T05:53:27.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.202+0000 7f16457fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health"}] 2024-09-16T05:53:27.207 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.202+0000 7f16457fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:53:27.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.206+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1634036d50 legacy=0x7f1634039210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.206+0000 7f164ecca640 10 monclient: shutdown 2024-09-16T05:53:27.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.206+0000 7f164ecca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:27.210 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.206+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 >> v1:172.21.15.63:6789/0 conn(0x7f164813b0e0 legacy=0x7f16481d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.206+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 shutdown_connections 2024-09-16T05:53:27.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.206+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 >> 172.21.15.63:0/11945520 conn(0x7f16480a1890 msgr2=0x7f16480a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:27.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.206+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 shutdown_connections 2024-09-16T05:53:27.211 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.206+0000 7f164ecca640 1 -- 172.21.15.63:0/11945520 wait complete. 2024-09-16T05:53:27.225 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph health detail 2024-09-16T05:53:27.225 INFO:tasks.workunit.client.0.smithi063.stderr:+ grep client.bad 2024-09-16T05:53:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.334+0000 7fe83f2df640 10 monclient: get_monmap_and_config 2024-09-16T05:53:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.334+0000 7fe83f2df640 10 monclient: build_initial_monmap 2024-09-16T05:53:27.338 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.334+0000 7fe83f2df640 10 monclient: monmap: 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:53:27.339639+0000 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:53:27.339639+0000 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:27.339 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.334+0000 7fe83f2df640 1 Processor -- start 2024-09-16T05:53:27.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.334+0000 7fe83f2df640 1 -- start start 2024-09-16T05:53:27.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.334+0000 7fe83f2df640 10 monclient: init 2024-09-16T05:53:27.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:27.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:53:27.346 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient(hunting): picked mon.noname-c con 0x7fe8381338b0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient(hunting): picked mon.noname-b con 0x7fe838136790 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient(hunting): picked mon.noname-a con 0x7fe83813a2a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83809e0d0 con 0x7fe83813a2a0 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8381343a0 con 0x7fe8381338b0 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83809dd80 con 0x7fe838136790 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient(hunting): authenticate will time out at 2064.116700s 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83d07b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe8381338b0 0x7fe838133c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48942/0 (socket says 172.21.15.63:48942) 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83d07b640 1 -- 172.21.15.63:0/105790953 learned_addr learned my addr 172.21.15.63:0/105790953 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412761390 0 0) 0x7fe8381343a0 con 0x7fe8381338b0 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.347 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient(hunting): my global_id is 184144 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe824003530 con 0x7fe8381338b0 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1168606579 0 0) 0x7fe824003530 con 0x7fe8381338b0 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 >> v1:172.21.15.6:6790/0 conn(0x7fe838136790 legacy=0x7fe838138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 >> v1:172.21.15.6:6789/0 conn(0x7fe83813a2a0 legacy=0x7fe83813c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 monclient: found mon.noname-c 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83809bd40 con 0x7fe8381338b0 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: _finish_auth 0 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:53:27.348 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe834001620 con 0x7fe8381338b0 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: dump: 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.349 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 5 monclient: authenticate success, global_id 184144 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe834001900 con 0x7fe8381338b0 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 1 -- 172.21.15.63:0/105790953 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe834001d10 con 0x7fe8381338b0 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe82ffff640 10 monclient: dump: 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.350 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 4 set_mon_vals no callback set 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient: shutdown 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- 172.21.15.63:0/105790953 >> v1:172.21.15.63:6789/0 conn(0x7fe8381338b0 legacy=0x7fe838133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- 172.21.15.63:0/105790953 shutdown_connections 2024-09-16T05:53:27.351 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- 172.21.15.63:0/105790953 >> 172.21.15.63:0/105790953 conn(0x7fe8380a1890 msgr2=0x7fe8380a1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- 172.21.15.63:0/105790953 shutdown_connections 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- 172.21.15.63:0/105790953 wait complete. 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient: build_initial_monmap 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient: monmap: 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:27.352 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 Processor -- start 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 1 -- start start 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient: init 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.338+0000 7fe83f2df640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient(hunting): picked mon.noname-a con 0x7fe8381338b0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient(hunting): picked mon.noname-c con 0x7fe838136790 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient(hunting): picked mon.noname-b con 0x7fe83813a2a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83811bbb0 con 0x7fe8381338b0 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8381cc270 con 0x7fe83813a2a0 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe838134f00 con 0x7fe838136790 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:27.353 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient(hunting): authenticate will time out at 2064.120115s 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83d87c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe83813a2a0 0x7fe8381d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48946/0 (socket says 172.21.15.63:48946) 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83d87c640 1 -- 172.21.15.63:0/3469633449 learned_addr learned my addr 172.21.15.63:0/3469633449 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249896112 0 0) 0x7fe8381cc270 con 0x7fe83813a2a0 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): my global_id is 184147 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe81c003560 con 0x7fe83813a2a0 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 563784934 0 0) 0x7fe81c003560 con 0x7fe83813a2a0 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8381cc270 con 0x7fe83813a2a0 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe830001cd0 con 0x7fe83813a2a0 2024-09-16T05:53:27.354 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): dump: 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:27.355 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545901720 0 0) 0x7fe83811bbb0 con 0x7fe8381338b0 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): my global_id is 174267 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe81c003560 con 0x7fe8381338b0 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1797707379 0 0) 0x7fe838134f00 con 0x7fe838136790 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient(hunting): my global_id is 164363 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe83811bbb0 con 0x7fe838136790 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3568408955 0 0) 0x7fe8381cc270 con 0x7fe83813a2a0 2024-09-16T05:53:27.356 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 >> v1:172.21.15.6:6790/0 conn(0x7fe838136790 legacy=0x7fe8381cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 >> v1:172.21.15.6:6789/0 conn(0x7fe8381338b0 legacy=0x7fe8381cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 monclient: found mon.b 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe838136230 con 0x7fe83813a2a0 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient: _finish_auth 0 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 5 monclient: authenticate success, global_id 184147 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient: _renew_subs 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.357 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe83811b8d0 con 0x7fe83813a2a0 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient: _renew_subs 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe838139bf0 con 0x7fe83813a2a0 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe830001ff0 con 0x7fe83813a2a0 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe830003e70 con 0x7fe83813a2a0 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82f7fe640 4 set_mon_vals no callback set 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 10 monclient: dump: 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.358 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe818005090 con 0x7fe83813a2a0 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe830004090 con 0x7fe83813a2a0 2024-09-16T05:53:27.359 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fe8300452f0 con 0x7fe83813a2a0 2024-09-16T05:53:27.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.342+0000 7fe83d07b640 10 monclient: get_auth_request con 0x7fe81c036d30 auth_method 0 2024-09-16T05:53:27.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.346+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe8300158c0 con 0x7fe83813a2a0 2024-09-16T05:53:27.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.346+0000 7fe82dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.360 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.346+0000 7fe82dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:27.482 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:53:27.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:53:27.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.574+0000 7fe83f2df640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-16T05:53:27.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.574+0000 7fe83f2df640 10 monclient: _send_command 2 [{"prefix": "health", "detail": "detail"}] 2024-09-16T05:53:27.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.574+0000 7fe83f2df640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.574+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "health", "detail": "detail"} v 0) -- 0x7fe818005380 con 0x7fe83813a2a0 2024-09-16T05:53:27.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.574+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) ==== 74+0+1041 (unknown 501213350 0 2456344155) 0x7fe8300151c0 con 0x7fe83813a2a0 2024-09-16T05:53:27.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.574+0000 7fe82dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "detail": "detail"}] 2024-09-16T05:53:27.581 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.574+0000 7fe82dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient: get_monmap_and_config 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient: build_initial_monmap 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient: monmap: 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:27.616479+0000 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:27.616479+0000 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.615 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:27.616 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:27.616 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:27.616 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 1 Processor -- start 2024-09-16T05:53:27.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 1 -- start start 2024-09-16T05:53:27.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient: init 2024-09-16T05:53:27.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:27.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:53:27.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd41337e0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:27.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd4133bc0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:27.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd4133fa0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd409b580 con 0x7f8dd41337e0 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd409aa30 con 0x7f8dd4133bc0 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd4073670 con 0x7f8dd4133fa0 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8ddc054640 10 monclient(hunting): authenticate will time out at 2069.423296s 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd9df0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8dd41337e0 0x7f8dd409bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53178/0 (socket says 172.21.15.6:53178) 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd9df0640 1 -- 172.21.15.6:0/235402140 learned_addr learned my addr 172.21.15.6:0/235402140 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4089144828 0 0) 0x7f8dd4073670 con 0x7f8dd4133fa0 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 10 monclient(hunting): my global_id is 164366 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc0003530 con 0x7f8dd4133fa0 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119920469 0 0) 0x7f8dd409b580 con 0x7f8dd41337e0 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 10 monclient(hunting): my global_id is 174270 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.612+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd4073670 con 0x7f8dd41337e0 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1451491846 0 0) 0x7f8dc0003530 con 0x7f8dd4133fa0 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 >> v1:172.21.15.63:6789/0 conn(0x7f8dd4133bc0 legacy=0x7f8dd413b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 >> v1:172.21.15.6:6789/0 conn(0x7f8dd41337e0 legacy=0x7f8dd409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: _finish_hunting 0 2024-09-16T05:53:27.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 1 monclient: found mon.noname-b 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd4143760 con 0x7f8dd4133fa0 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: _finish_auth 0 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: _check_auth_tickets 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dcc0019e0 con 0x7f8dd4133fa0 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: dump: 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.626 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 5 monclient: authenticate success, global_id 164366 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8dcc001c40 con 0x7f8dd4133fa0 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 1 -- 172.21.15.6:0/235402140 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dcc002050 con 0x7f8dd4133fa0 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:27.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 4 set_mon_vals no callback set 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd8dee640 10 monclient: dump: 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.628 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: shutdown 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- 172.21.15.6:0/235402140 >> v1:172.21.15.6:6790/0 conn(0x7f8dd4133fa0 legacy=0x7f8dd413f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- 172.21.15.6:0/235402140 shutdown_connections 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- 172.21.15.6:0/235402140 >> 172.21.15.6:0/235402140 conn(0x7f8dd40a2760 msgr2=0x7f8dd40a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- 172.21.15.6:0/235402140 shutdown_connections 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- 172.21.15.6:0/235402140 wait complete. 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: build_initial_monmap 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: monmap: 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:27.629 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 Processor -- start 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- start start 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: init 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd41337e0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:27.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd4133bc0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd4133fa0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd413a700 con 0x7f8dd4133bc0 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd413bbb0 con 0x7f8dd4133fa0 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd41434c0 con 0x7f8dd41337e0 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient(hunting): authenticate will time out at 2069.426470s 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd9df0640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8dd41337e0 0x7f8dd41cb8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34228/0 (socket says 172.21.15.6:34228) 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd95ef640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8dd4133bc0 0x7f8dd41d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53190/0 (socket says 172.21.15.6:53190) 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dd9df0640 1 -- 172.21.15.6:0/181451634 learned_addr learned my addr 172.21.15.6:0/181451634 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2771484833 0 0) 0x7f8dd413a700 con 0x7f8dd4133bc0 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): my global_id is 174273 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8db8003530 con 0x7f8dd4133bc0 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 785860518 0 0) 0x7f8dd41434c0 con 0x7f8dd41337e0 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): my global_id is 164369 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd413a700 con 0x7f8dd41337e0 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 458303203 0 0) 0x7f8db8003530 con 0x7f8dd4133bc0 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dd41434c0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dc40018c0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:27.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): dump: 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:27.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2232552959 0 0) 0x7f8dd413a700 con 0x7f8dd41337e0 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8db8003530 con 0x7f8dd41337e0 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dd0001840 con 0x7f8dd41337e0 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): dump: 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.634 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2109836960 0 0) 0x7f8dd413bbb0 con 0x7f8dd4133fa0 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient(hunting): my global_id is 184153 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd413a700 con 0x7f8dd4133fa0 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3660530205 0 0) 0x7f8dd41434c0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 >> v1:172.21.15.6:6790/0 conn(0x7f8dd41337e0 legacy=0x7f8dd41cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 >> v1:172.21.15.63:6789/0 conn(0x7f8dd4133fa0 legacy=0x7f8dd41d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 monclient: found mon.a 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:27.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd4142fb0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient: _finish_auth 0 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 5 monclient: authenticate success, global_id 174273 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: _renew_subs 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd413b8d0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: _renew_subs 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8dd413b5f0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8dc4001cb0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8dc4003af0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dcbfff640 4 set_mon_vals no callback set 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.616+0000 7f8dca7fc640 10 monclient: dump: 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.638 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.639 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.620+0000 7f8ddc054640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.620+0000 7f8ddc054640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.620+0000 7f8ddc054640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:27.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.620+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dbc005090 con 0x7f8dd4133bc0 2024-09-16T05:53:27.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.620+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8dc4003c90 con 0x7f8dd4133bc0 2024-09-16T05:53:27.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.620+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f8dc4044e10 con 0x7f8dd4133bc0 2024-09-16T05:53:27.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.620+0000 7f8dd9df0640 10 monclient: get_auth_request con 0x7f8db8036bf0 auth_method 0 2024-09-16T05:53:27.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.624+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8dc40153e0 con 0x7f8dd4133bc0 2024-09-16T05:53:27.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.624+0000 7f8dca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.624+0000 7f8dca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:27.734 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.730+0000 7fe83f2df640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-16T05:53:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.730+0000 7fe83f2df640 10 monclient: _send_command 3 [{"prefix": "health", "detail": "detail"}] 2024-09-16T05:53:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.730+0000 7fe83f2df640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.730+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "health", "detail": "detail"} v 0) -- 0x7fe8180057b0 con 0x7fe83813a2a0 2024-09-16T05:53:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.730+0000 7fe82dffb640 1 -- 172.21.15.63:0/3469633449 <== mon.1 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) ==== 74+0+1041 (unknown 501213350 0 2456344155) 0x7fe830019510 con 0x7fe83813a2a0 2024-09-16T05:53:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe82dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health", "detail": "detail"}] 2024-09-16T05:53:27.735 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe82dffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:53:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe81c036d30 legacy=0x7fe81c0391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe83f2df640 10 monclient: shutdown 2024-09-16T05:53:27.737 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe83f2df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 >> v1:172.21.15.63:6789/0 conn(0x7fe83813a2a0 legacy=0x7fe8381d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 shutdown_connections 2024-09-16T05:53:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 >> 172.21.15.63:0/3469633449 conn(0x7fe8380a1890 msgr2=0x7fe8380a0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 shutdown_connections 2024-09-16T05:53:27.738 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.734+0000 7fe83f2df640 1 -- 172.21.15.63:0/3469633449 wait complete. 2024-09-16T05:53:27.751 INFO:tasks.workunit.client.0.smithi063.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-16T05:53:27.751 INFO:tasks.workunit.client.0.smithi063.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-16T05:53:27.751 INFO:tasks.workunit.client.0.smithi063.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-16T05:53:27.751 INFO:tasks.workunit.client.0.smithi063.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-16T05:53:27.751 INFO:tasks.workunit.client.0.smithi063.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-16T05:53:27.751 INFO:tasks.workunit.client.0.smithi063.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-16T05:53:27.751 INFO:tasks.workunit.client.0.smithi063.stderr:+ ceph auth rm client.bad 2024-09-16T05:53:27.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:53:27.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:53:27.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:27.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8dbc005380 con 0x7f8dd4133bc0 2024-09-16T05:53:27.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8dca7fc640 1 -- 172.21.15.6:0/181451634 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 3326364185) 0x7f8dc4019090 con 0x7f8dd4133bc0 2024-09-16T05:53:27.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8dca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:53:27.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8dca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:27.844 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:53:27.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8db8036bf0 legacy=0x7f8db80390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 10 monclient: shutdown 2024-09-16T05:53:27.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:27.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 >> v1:172.21.15.6:6789/0 conn(0x7f8dd4133bc0 legacy=0x7f8dd41d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 shutdown_connections 2024-09-16T05:53:27.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 >> 172.21.15.6:0/181451634 conn(0x7f8dd40a2760 msgr2=0x7f8dd4142540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:27.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 shutdown_connections 2024-09-16T05:53:27.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.840+0000 7f8ddc054640 1 -- 172.21.15.6:0/181451634 wait complete. 2024-09-16T05:53:27.859 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:53:27.859 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T05:53:27.860 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:53:27.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:53:27.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient: get_monmap_and_config 2024-09-16T05:53:27.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient: build_initial_monmap 2024-09-16T05:53:27.864 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient: monmap: 2024-09-16T05:53:27.864 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:27.865 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.865 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:53:27.865394+0000 2024-09-16T05:53:27.865 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:53:27.865394+0000 2024-09-16T05:53:27.865 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:27.865 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.865 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:27.866 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:27.866 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:27.866 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.878 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 1 Processor -- start 2024-09-16T05:53:27.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 1 -- start start 2024-09-16T05:53:27.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient: init 2024-09-16T05:53:27.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:27.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:53:27.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient(hunting): picked mon.noname-c con 0x7f418c0a3a90 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:27.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient(hunting): picked mon.noname-b con 0x7f418c0a1ba0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:27.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient(hunting): picked mon.noname-a con 0x7f418c13ccb0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:27.879 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c0a0920 con 0x7f418c13ccb0 2024-09-16T05:53:27.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c0a0ee0 con 0x7f418c0a3a90 2024-09-16T05:53:27.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c0a6910 con 0x7f418c0a1ba0 2024-09-16T05:53:27.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:27.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 10 monclient(hunting): authenticate will time out at 2064.643063s 2024-09-16T05:53:27.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418a59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f418c0a3a90 0x7f418c0a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48948/0 (socket says 172.21.15.63:48948) 2024-09-16T05:53:27.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418a59c640 1 -- 172.21.15.63:0/2149240130 learned_addr learned my addr 172.21.15.63:0/2149240130 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:27.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1435775235 0 0) 0x7f418c0a0ee0 con 0x7f418c0a3a90 2024-09-16T05:53:27.880 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient(hunting): my global_id is 184156 2024-09-16T05:53:27.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4174003530 con 0x7f418c0a3a90 2024-09-16T05:53:27.881 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4189d9b640 1 -- 172.21.15.63:0/2149240130 >> v1:172.21.15.6:6790/0 conn(0x7f418c0a1ba0 legacy=0x7f418c0a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T05:53:27.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4189d9b640 1 -- 172.21.15.63:0/2149240130 >> v1:172.21.15.6:6790/0 conn(0x7f418c0a1ba0 legacy=0x7f418c0a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T05:53:27.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4189d9b640 1 --1- 172.21.15.63:0/2149240130 >> v1:172.21.15.6:6790/0 conn(0x7f418c0a1ba0 0x7f418c0a4580 :-1 s=OPENED pgs=1319 cs=1 l=1).handle_message read tag failed 2024-09-16T05:53:27.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4189d9b640 1 --1- 172.21.15.63:0/2149240130 >> v1:172.21.15.6:6790/0 conn(0x7f418c0a1ba0 0x7f418c0a4580 :-1 s=OPENED pgs=1319 cs=1 l=1).fault on lossy channel, failing 2024-09-16T05:53:27.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6790/0 2024-09-16T05:53:27.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3580998101 0 0) 0x7f4174003530 con 0x7f418c0a3a90 2024-09-16T05:53:27.882 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 >> v1:172.21.15.6:6790/0 conn(0x7f418c0a1ba0 legacy=0x7f418c0a4580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:27.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 >> v1:172.21.15.6:6789/0 conn(0x7f418c13ccb0 legacy=0x7f418c13f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient: _finish_hunting 0 2024-09-16T05:53:27.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 1 monclient: found mon.noname-c 2024-09-16T05:53:27.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:27.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.63:6789/0 2024-09-16T05:53:27.883 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418c1437f0 con 0x7f418c0a3a90 2024-09-16T05:53:27.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient: _finish_auth 0 2024-09-16T05:53:27.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient: _check_auth_tickets 2024-09-16T05:53:27.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:27.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f417c001620 con 0x7f418c0a3a90 2024-09-16T05:53:27.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 5 monclient: authenticate success, global_id 184156 2024-09-16T05:53:27.884 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:27.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:27.885 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f418959a640 10 monclient: dump: 2024-09-16T05:53:27.885 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.885 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.885 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.862+0000 7f4190b74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:27.886 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 <== mon.1 v1:172.21.15.63:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f417c001880 con 0x7f418c0a3a90 2024-09-16T05:53:27.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f418959a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:27.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f418959a640 1 -- 172.21.15.63:0/2149240130 <== mon.1 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f417c001bd0 con 0x7f418c0a3a90 2024-09-16T05:53:27.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:27.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f418959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 4 set_mon_vals no callback set 2024-09-16T05:53:27.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f418959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:27.887 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f418959a640 10 monclient: dump: 2024-09-16T05:53:27.887 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: shutdown 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:27.888 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- 172.21.15.63:0/2149240130 >> v1:172.21.15.63:6789/0 conn(0x7f418c0a3a90 legacy=0x7f418c0a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- 172.21.15.63:0/2149240130 shutdown_connections 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- 172.21.15.63:0/2149240130 >> 172.21.15.63:0/2149240130 conn(0x7f418c0a5b90 msgr2=0x7f418c0a5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- 172.21.15.63:0/2149240130 shutdown_connections 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- 172.21.15.63:0/2149240130 wait complete. 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: build_initial_monmap 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: monmap: 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 0 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 0.000000 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:created 0.000000 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.889 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 Processor -- start 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- start start 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: init 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient(hunting): picked mon.noname-a con 0x7f418c0a1ba0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient(hunting): picked mon.noname-c con 0x7f418c0a3a90 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient(hunting): picked mon.noname-b con 0x7f418c13ccb0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c13c420 con 0x7f418c0a1ba0 2024-09-16T05:53:27.890 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c13cad0 con 0x7f418c13ccb0 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c1402e0 con 0x7f418c0a3a90 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient(hunting): authenticate will time out at 2064.646067s 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f418ad9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f418c13ccb0 0x7f418c1d4280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48952/0 (socket says 172.21.15.63:48952) 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f418ad9d640 1 -- 172.21.15.63:0/4007420911 learned_addr learned my addr 172.21.15.63:0/4007420911 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3735893021 0 0) 0x7f418c1402e0 con 0x7f418c0a3a90 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): my global_id is 164372 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f416c003530 con 0x7f418c0a3a90 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2879951188 0 0) 0x7f418c0a0920 con 0x7f418c13ccb0 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.891 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): my global_id is 184159 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f418c1402e0 con 0x7f418c13ccb0 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 237897251 0 0) 0x7f418c13c420 con 0x7f418c0a1ba0 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): my global_id is 174279 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f418c0a0920 con 0x7f418c0a1ba0 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2054206435 0 0) 0x7f416c003530 con 0x7f418c0a3a90 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f418c13c420 con 0x7f418c0a3a90 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4180001820 con 0x7f418c0a3a90 2024-09-16T05:53:27.892 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): dump: 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.893 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 27433733 0 0) 0x7f418c1402e0 con 0x7f418c13ccb0 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f416c003530 con 0x7f418c13ccb0 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f4184001bc0 con 0x7f418c13ccb0 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): dump: 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.894 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4085815683 0 0) 0x7f418c0a0920 con 0x7f418c0a1ba0 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f418c1402e0 con 0x7f418c0a1ba0 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f417c001520 con 0x7f418c0a1ba0 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient(hunting): dump: 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.895 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3558469020 0 0) 0x7f418c13c420 con 0x7f418c0a3a90 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 >> v1:172.21.15.63:6789/0 conn(0x7f418c13ccb0 legacy=0x7f418c1d4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 >> v1:172.21.15.6:6789/0 conn(0x7f418c0a1ba0 legacy=0x7f418c136ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient: _finish_hunting 0 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 monclient: found mon.c 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:27.896 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418c0a0280 con 0x7f418c0a3a90 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient: _finish_auth 0 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient: _check_auth_tickets 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 5 monclient: authenticate success, global_id 164372 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: _renew_subs 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f418c09c520 con 0x7f418c0a3a90 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: _renew_subs 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f418c09ad30 con 0x7f418c0a3a90 2024-09-16T05:53:27.897 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4180003600 con 0x7f418c0a3a90 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f41800039d0 con 0x7f418c0a3a90 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417bfff640 4 set_mon_vals no callback set 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.866+0000 7f417affd640 10 monclient: dump: 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:epoch 1 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.898 INFO:tasks.workunit.client.0.smithi063.stderr:election_strategy: 1 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.870+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f41800039d0 con 0x7f418c0a3a90 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.870+0000 7f4190b74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.870+0000 7f4190b74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.870+0000 7f4190b74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.870+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4170005090 con 0x7f418c0a3a90 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.870+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f4180044b20 con 0x7f418c0a3a90 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.870+0000 7f418a59c640 10 monclient: get_auth_request con 0x7f416c036d60 auth_method 0 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.874+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f418c09c520 con 0x7f418c0a3a90 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.874+0000 7f417affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:27.899 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:27.874+0000 7f417affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:27.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: get_monmap_and_config 2024-09-16T05:53:27.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: build_initial_monmap 2024-09-16T05:53:27.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: monmap: 2024-09-16T05:53:27.977 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:27.979110+0000 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:27.979110+0000 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:27.978 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 Processor -- start 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- start start 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: init 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient(hunting): picked mon.noname-b con 0x7fe1e409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient(hunting): picked mon.noname-c con 0x7fe1e409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient(hunting): picked mon.noname-a con 0x7fe1e40a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e40a1370 con 0x7fe1e40a0da0 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e409ead0 con 0x7fe1e409e230 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e409edb0 con 0x7fe1e409aa30 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient(hunting): authenticate will time out at 2069.785904s 2024-09-16T05:53:27.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ea783640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe1e40a0da0 0x7fe1e413d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53198/0 (socket says 172.21.15.6:53198) 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ea783640 1 -- 172.21.15.6:0/1753620336 learned_addr learned my addr 172.21.15.6:0/1753620336 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2506307775 0 0) 0x7fe1e409edb0 con 0x7fe1e409aa30 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient(hunting): my global_id is 164375 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1d0003530 con 0x7fe1e409aa30 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773325017 0 0) 0x7fe1e40a1370 con 0x7fe1e40a0da0 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient(hunting): _negotiate 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient(hunting): my global_id is 174282 2024-09-16T05:53:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1e409edb0 con 0x7fe1e40a0da0 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 777829761 0 0) 0x7fe1e409edb0 con 0x7fe1e40a0da0 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 >> v1:172.21.15.6:6790/0 conn(0x7fe1e409aa30 legacy=0x7fe1e409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 >> v1:172.21.15.63:6789/0 conn(0x7fe1e409e230 legacy=0x7fe1e41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: _finish_hunting 0 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 monclient: found mon.noname-a 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e409dd50 con 0x7fe1e40a0da0 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: _finish_auth 0 2024-09-16T05:53:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: _check_auth_tickets 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe1dc001850 con 0x7fe1e40a0da0 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 5 monclient: authenticate success, global_id 174282 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: dump: 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.990 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1dc001ac0 con 0x7fe1e40a0da0 2024-09-16T05:53:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:27.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 1 -- 172.21.15.6:0/1753620336 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe1dc001ed0 con 0x7fe1e40a0da0 2024-09-16T05:53:27.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:27.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:27.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1e8f80640 10 monclient: dump: 2024-09-16T05:53:27.991 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:27.992 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:27.992 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.992 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:27.992 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:27.992 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.993 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:27.993 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:27.993 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:27.993 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:27.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 4 set_mon_vals no callback set 2024-09-16T05:53:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: shutdown 2024-09-16T05:53:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/1753620336 >> v1:172.21.15.6:6789/0 conn(0x7fe1e40a0da0 legacy=0x7fe1e413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/1753620336 shutdown_connections 2024-09-16T05:53:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/1753620336 >> 172.21.15.6:0/1753620336 conn(0x7fe1e40a1650 msgr2=0x7fe1e40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/1753620336 shutdown_connections 2024-09-16T05:53:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/1753620336 wait complete. 2024-09-16T05:53:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: build_initial_monmap 2024-09-16T05:53:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.976+0000 7fe1ec1e6640 10 monclient: monmap: 2024-09-16T05:53:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:27.997 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 1 Processor -- start 2024-09-16T05:53:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 1 -- start start 2024-09-16T05:53:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient: init 2024-09-16T05:53:27.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:27.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:53:27.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient(hunting): picked mon.noname-c con 0x7fe1e409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:27.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient(hunting): picked mon.noname-a con 0x7fe1e409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:27.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient(hunting): picked mon.noname-b con 0x7fe1e40a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e41343a0 con 0x7fe1e409e230 2024-09-16T05:53:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e4132760 con 0x7fe1e40a0da0 2024-09-16T05:53:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e406e6d0 con 0x7fe1e409aa30 2024-09-16T05:53:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient(hunting): authenticate will time out at 2069.788813s 2024-09-16T05:53:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1e9781640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe1e409e230 0x7fe1e41d47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53200/0 (socket says 172.21.15.6:53200) 2024-09-16T05:53:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1e9f82640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe1e409aa30 0x7fe1e4133c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34250/0 (socket says 172.21.15.6:34250) 2024-09-16T05:53:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1e9781640 1 -- 172.21.15.6:0/885982852 learned_addr learned my addr 172.21.15.6:0/885982852 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2863753058 0 0) 0x7fe1e41343a0 con 0x7fe1e409e230 2024-09-16T05:53:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): my global_id is 174285 2024-09-16T05:53:28.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1c8003530 con 0x7fe1e409e230 2024-09-16T05:53:28.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 53488687 0 0) 0x7fe1e406e6d0 con 0x7fe1e409aa30 2024-09-16T05:53:28.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): my global_id is 164378 2024-09-16T05:53:28.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1e41343a0 con 0x7fe1e409aa30 2024-09-16T05:53:28.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3070742458 0 0) 0x7fe1c8003530 con 0x7fe1e409e230 2024-09-16T05:53:28.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1e406e6d0 con 0x7fe1e409e230 2024-09-16T05:53:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe1d4001940 con 0x7fe1e409e230 2024-09-16T05:53:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): dump: 2024-09-16T05:53:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.005 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.005 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.005 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:28.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2161463161 0 0) 0x7fe1e41343a0 con 0x7fe1e409aa30 2024-09-16T05:53:28.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1c8003530 con 0x7fe1e409aa30 2024-09-16T05:53:28.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe1e00018a0 con 0x7fe1e409aa30 2024-09-16T05:53:28.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:28.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:28.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): dump: 2024-09-16T05:53:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606035640 0 0) 0x7fe1e4132760 con 0x7fe1e40a0da0 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): my global_id is 184165 2024-09-16T05:53:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1e41343a0 con 0x7fe1e40a0da0 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4126459120 0 0) 0x7fe1e406e6d0 con 0x7fe1e409e230 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 >> v1:172.21.15.6:6790/0 conn(0x7fe1e409aa30 legacy=0x7fe1e4133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 >> v1:172.21.15.63:6789/0 conn(0x7fe1e40a0da0 legacy=0x7fe1e41d7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 monclient: found mon.a 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e40740c0 con 0x7fe1e409e230 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: _finish_auth 0 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:53:28.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 5 monclient: authenticate success, global_id 174285 2024-09-16T05:53:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient: _renew_subs 2024-09-16T05:53:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e4130b60 con 0x7fe1e409e230 2024-09-16T05:53:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient: _renew_subs 2024-09-16T05:53:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1e4134780 con 0x7fe1e409e230 2024-09-16T05:53:28.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1d4001cd0 con 0x7fe1e409e230 2024-09-16T05:53:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe1d4003af0 con 0x7fe1e409e230 2024-09-16T05:53:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1dbfff640 4 set_mon_vals no callback set 2024-09-16T05:53:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: dump: 2024-09-16T05:53:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.013 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.013 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:28.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1cc0065d0 con 0x7fe1cc002870 2024-09-16T05:53:28.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe1d4003c90 con 0x7fe1e409e230 2024-09-16T05:53:28.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fe1d4044da0 con 0x7fe1e409e230 2024-09-16T05:53:28.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3339186351 0 0) 0x7fe1cc0065d0 con 0x7fe1cc002870 2024-09-16T05:53:28.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient(hunting): my global_id is 174288 2024-09-16T05:53:28.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1e406e6d0 con 0x7fe1cc002870 2024-09-16T05:53:28.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1ea783640 10 monclient: get_auth_request con 0x7fe1c8036bb0 auth_method 0 2024-09-16T05:53:28.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1550703070 0 0) 0x7fe1e406e6d0 con 0x7fe1cc002870 2024-09-16T05:53:28.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1cc0065d0 con 0x7fe1cc002870 2024-09-16T05:53:28.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe1e00037a0 con 0x7fe1cc002870 2024-09-16T05:53:28.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:28.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: dump: 2024-09-16T05:53:28.016 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.016 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.016 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.017 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.017 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.017 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.017 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.017 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.017 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.018 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3182848334 0 0) 0x7fe1cc0065d0 con 0x7fe1cc002870 2024-09-16T05:53:28.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1cc00a160 con 0x7fe1cc002870 2024-09-16T05:53:28.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe1e0005790 con 0x7fe1cc002870 2024-09-16T05:53:28.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:28.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:28.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 >> v1:172.21.15.6:6789/0 conn(0x7fe1cc002870 legacy=0x7fe1cc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.980+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 >> v1:172.21.15.6:6789/0 conn(0x7fe1cc002870 legacy=0x7fe1cc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1cc009e30 con 0x7fe1cc0090f0 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786467267 0 0) 0x7fe1cc009e30 con 0x7fe1cc0090f0 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient(hunting): my global_id is 174291 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1cc0065d0 con 0x7fe1cc0090f0 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3000261438 0 0) 0x7fe1cc0065d0 con 0x7fe1cc0090f0 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1cc009e30 con 0x7fe1cc0090f0 2024-09-16T05:53:28.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe1e0003dd0 con 0x7fe1cc0090f0 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient: dump: 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1829907813 0 0) 0x7fe1cc009e30 con 0x7fe1cc0090f0 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1cc00c3a0 con 0x7fe1cc0090f0 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4298 (unknown 0 0 950847371) 0x7fe1e0004000 con 0x7fe1cc0090f0 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 >> v1:172.21.15.6:6789/0 conn(0x7fe1cc0090f0 legacy=0x7fe1cc009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1da7fc640 1 -- 172.21.15.6:0/885982852 >> v1:172.21.15.6:6789/0 conn(0x7fe1cc0090f0 legacy=0x7fe1cc009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:28.023 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 80, 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 110662, 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.024 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.025 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:53:28.027 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.028 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.029 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:28.030 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.031 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.032 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.033 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.034 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:53:28.035 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe1c8036bb0 legacy=0x7fe1c8039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 10 monclient: shutdown 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 >> v1:172.21.15.6:6789/0 conn(0x7fe1e409e230 legacy=0x7fe1e41d47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ea783640 1 -- 172.21.15.6:0/885982852 reap_dead start 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ea783640 1 -- 172.21.15.6:0/885982852 reap_dead start 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 shutdown_connections 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 >> 172.21.15.6:0/885982852 conn(0x7fe1e40a1650 msgr2=0x7fe1e40a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 shutdown_connections 2024-09-16T05:53:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:27.988+0000 7fe1ec1e6640 1 -- 172.21.15.6:0/885982852 wait complete. 2024-09-16T05:53:28.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:53:28.038 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:53:28.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:53:28.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.094+0000 7f4190b74640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T05:53:28.097 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.094+0000 7f4190b74640 10 monclient: _send_command 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T05:53:28.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.094+0000 7f4190b74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:28.098 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.094+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) -- 0x7f4170005380 con 0x7f418c0a3a90 2024-09-16T05:53:28.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.102+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v32) ==== 79+0+0 (unknown 693141373 0 0) 0x7f4180015150 con 0x7f418c0a3a90 2024-09-16T05:53:28.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.102+0000 7f417affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T05:53:28.105 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.102+0000 7f417affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient: get_monmap_and_config 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient: build_initial_monmap 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient: monmap: 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:28.159600+0000 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:28.159600+0000 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:28.158 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:28.159 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 1 Processor -- start 2024-09-16T05:53:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 1 -- start start 2024-09-16T05:53:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient: init 2024-09-16T05:53:28.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:53:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2b6809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2b6809bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2b681397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b680a1350 con 0x7f2b6809bf20 2024-09-16T05:53:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b680a10d0 con 0x7f2b681397f0 2024-09-16T05:53:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b6809d940 con 0x7f2b6809bb40 2024-09-16T05:53:28.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient(hunting): authenticate will time out at 2069.966340s 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b66ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2b6809bf20 0x7f2b6809c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53238/0 (socket says 172.21.15.6:53238) 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b66ffd640 1 -- 172.21.15.6:0/596990888 learned_addr learned my addr 172.21.15.6:0/596990888 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b677fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2b6809bb40 0x7f2b6809e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:34262/0 (socket says 172.21.15.6:34262) 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4222769776 0 0) 0x7f2b680a1350 con 0x7f2b6809bf20 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): my global_id is 174294 2024-09-16T05:53:28.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b54003530 con 0x7f2b6809bf20 2024-09-16T05:53:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2504102938 0 0) 0x7f2b6809d940 con 0x7f2b6809bb40 2024-09-16T05:53:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): my global_id is 164381 2024-09-16T05:53:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b680a1350 con 0x7f2b6809bb40 2024-09-16T05:53:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461271793 0 0) 0x7f2b680a10d0 con 0x7f2b681397f0 2024-09-16T05:53:28.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient(hunting): my global_id is 184168 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b6809d940 con 0x7f2b681397f0 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2924136673 0 0) 0x7f2b54003530 con 0x7f2b6809bf20 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 >> v1:172.21.15.6:6790/0 conn(0x7f2b6809bb40 legacy=0x7f2b6809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 >> v1:172.21.15.63:6789/0 conn(0x7f2b681397f0 legacy=0x7f2b6813bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: _finish_hunting 0 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 monclient: found mon.noname-a 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6809dc20 con 0x7f2b6809bf20 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: _finish_auth 0 2024-09-16T05:53:28.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: _check_auth_tickets 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2b580019a0 con 0x7f2b6809bf20 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: dump: 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.172 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 5 monclient: authenticate success, global_id 174294 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b58001c50 con 0x7f2b6809bf20 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 1 -- 172.21.15.6:0/596990888 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2b58002020 con 0x7f2b6809bf20 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b6e1d0640 4 set_mon_vals no callback set 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.156+0000 7f2b667fc640 10 monclient: dump: 2024-09-16T05:53:28.173 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: shutdown 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/596990888 >> v1:172.21.15.6:6789/0 conn(0x7f2b6809bf20 legacy=0x7f2b6809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/596990888 shutdown_connections 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/596990888 >> 172.21.15.6:0/596990888 conn(0x7f2b680a1650 msgr2=0x7f2b680a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/596990888 shutdown_connections 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/596990888 wait complete. 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: build_initial_monmap 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: monmap: 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.175 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 Processor -- start 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- start start 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: init 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2b6809bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2b6809bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2b681397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b6811e410 con 0x7f2b6809bb40 2024-09-16T05:53:28.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b6811bfc0 con 0x7f2b681397f0 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b6811e6f0 con 0x7f2b6809bf20 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient(hunting): authenticate will time out at 2069.970182s 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b677fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2b6809bb40 0x7f2b6811b1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53246/0 (socket says 172.21.15.6:53246) 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b677fe640 1 -- 172.21.15.6:0/3262314736 learned_addr learned my addr 172.21.15.6:0/3262314736 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513648281 0 0) 0x7f2b6811e410 con 0x7f2b6809bb40 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): my global_id is 174297 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b48003530 con 0x7f2b6809bb40 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3860648830 0 0) 0x7f2b6811e6f0 con 0x7f2b6809bf20 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): my global_id is 164384 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b6811e410 con 0x7f2b6809bf20 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 221505871 0 0) 0x7f2b6811bfc0 con 0x7f2b681397f0 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): my global_id is 184171 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b6811e6f0 con 0x7f2b681397f0 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 81888528 0 0) 0x7f2b48003530 con 0x7f2b6809bb40 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b6811bfc0 con 0x7f2b6809bb40 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2b500018c0 con 0x7f2b6809bb40 2024-09-16T05:53:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:53:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): dump: 2024-09-16T05:53:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2414939730 0 0) 0x7f2b6811e410 con 0x7f2b6809bf20 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b48003530 con 0x7f2b6809bf20 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2b58001fa0 con 0x7f2b6809bf20 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:28.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient(hunting): dump: 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 827704911 0 0) 0x7f2b6811bfc0 con 0x7f2b6809bb40 2024-09-16T05:53:28.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 >> v1:172.21.15.6:6790/0 conn(0x7f2b6809bf20 legacy=0x7f2b6811b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 >> v1:172.21.15.63:6789/0 conn(0x7f2b681397f0 legacy=0x7f2b681d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 monclient: found mon.a 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b680a4040 con 0x7f2b6809bb40 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient: _finish_auth 0 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 5 monclient: authenticate success, global_id 174297 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: _renew_subs 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b680a3af0 con 0x7f2b6809bb40 2024-09-16T05:53:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: _renew_subs 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b6812ff70 con 0x7f2b6809bb40 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b50001ca0 con 0x7f2b6809bb40 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2b50004c40 con 0x7f2b6809bb40 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b65ffb640 4 set_mon_vals no callback set 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b5ffff640 10 monclient: dump: 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.183 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.160+0000 7f2b6e1d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2b4c0065d0 con 0x7f2b4c002870 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2b5000f9e0 con 0x7f2b6809bb40 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f2b50044d40 con 0x7f2b6809bb40 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3937121259 0 0) 0x7f2b4c0065d0 con 0x7f2b4c002870 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient(hunting): my global_id is 164387 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b6811bfc0 con 0x7f2b4c002870 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b67fff640 10 monclient: get_auth_request con 0x7f2b48036aa0 auth_method 0 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 527975233 0 0) 0x7f2b6811bfc0 con 0x7f2b4c002870 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b4c0065d0 con 0x7f2b4c002870 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2b58004ec0 con 0x7f2b4c002870 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient: dump: 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2837394259 0 0) 0x7f2b4c0065d0 con 0x7f2b4c002870 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b4c00a160 con 0x7f2b4c002870 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2b58007480 con 0x7f2b4c002870 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 >> v1:172.21.15.6:6790/0 conn(0x7f2b4c002870 legacy=0x7f2b4c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.164+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 >> v1:172.21.15.6:6790/0 conn(0x7f2b4c002870 legacy=0x7f2b4c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.168+0000 7f2b6e1d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.168+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2b4c009e50 con 0x7f2b4c009110 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.168+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805021178 0 0) 0x7f2b4c009e50 con 0x7f2b4c009110 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.168+0000 7f2b5ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.168+0000 7f2b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.168+0000 7f2b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.168+0000 7f2b5ffff640 10 monclient(hunting): my global_id is 164390 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.168+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b4c0065d0 con 0x7f2b4c009110 2024-09-16T05:53:28.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3350466724 0 0) 0x7f2b4c0065d0 con 0x7f2b4c009110 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b4c009e50 con 0x7f2b4c009110 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2b58005420 con 0x7f2b4c009110 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 10 monclient: dump: 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.189 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1338418386 0 0) 0x7f2b4c009e50 con 0x7f2b4c009110 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b4c00c3e0 con 0x7f2b4c009110 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 1071114644) 0x7f2b58007630 con 0x7f2b4c009110 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 >> v1:172.21.15.6:6790/0 conn(0x7f2b4c009110 legacy=0x7f2b4c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b5ffff640 1 -- 172.21.15.6:0/3262314736 >> v1:172.21.15.6:6790/0 conn(0x7f2b4c009110 legacy=0x7f2b4c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:28.192 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:53:28.192 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 80, 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 219133, 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:28.193 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.194 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.195 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:53:28.196 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.197 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.201 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.202 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.203 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:53:28.204 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2b48036aa0 legacy=0x7f2b48038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b6e1d0640 10 monclient: shutdown 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b6e1d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 >> v1:172.21.15.6:6789/0 conn(0x7f2b6809bb40 legacy=0x7f2b6811b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b67fff640 1 -- 172.21.15.6:0/3262314736 reap_dead start 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b67fff640 1 -- 172.21.15.6:0/3262314736 reap_dead start 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 shutdown_connections 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 >> 172.21.15.6:0/3262314736 conn(0x7f2b680a1650 msgr2=0x7f2b68133b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 shutdown_connections 2024-09-16T05:53:28.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.172+0000 7f2b6e1d0640 1 -- 172.21.15.6:0/3262314736 wait complete. 2024-09-16T05:53:28.207 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:53:28.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:53:28.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:53:28.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.258+0000 7f4190b74640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T05:53:28.261 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.258+0000 7f4190b74640 10 monclient: _send_command 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T05:53:28.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.258+0000 7f4190b74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:28.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.258+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) -- 0x7f4170002220 con 0x7f418c0a3a90 2024-09-16T05:53:28.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.258+0000 7f417affd640 1 -- 172.21.15.63:0/4007420911 <== mon.2 v1:172.21.15.6:6790/0 11 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v32) ==== 79+0+0 (unknown 693141373 0 0) 0x7f418001d600 con 0x7f418c0a3a90 2024-09-16T05:53:28.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.258+0000 7f417affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T05:53:28.262 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.258+0000 7f417affd640 10 monclient: _finish_command 3 = system:0 2024-09-16T05:53:28.264 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.262+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 >> v1:172.21.15.63:6812/3643207211 conn(0x7f416c036d60 legacy=0x7f416c039220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.262+0000 7f4190b74640 10 monclient: shutdown 2024-09-16T05:53:28.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.262+0000 7f4190b74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:28.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.262+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 >> v1:172.21.15.6:6790/0 conn(0x7f418c0a3a90 legacy=0x7f418c1d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.262+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 shutdown_connections 2024-09-16T05:53:28.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.262+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 >> 172.21.15.63:0/4007420911 conn(0x7f418c0a5b90 msgr2=0x7f418c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:28.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.262+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 shutdown_connections 2024-09-16T05:53:28.265 INFO:tasks.workunit.client.0.smithi063.stderr:2024-09-16T05:53:28.262+0000 7f4190b74640 1 -- 172.21.15.63:0/4007420911 wait complete. 2024-09-16T05:53:28.279 INFO:tasks.workunit.client.0.smithi063.stderr:+ expect 'ceph auth health | grep AUTH_BAD_CAP' 1 2024-09-16T05:53:28.279 INFO:tasks.workunit.client.0.smithi063.stderr:+ cmd='ceph auth health | grep AUTH_BAD_CAP' 2024-09-16T05:53:28.279 INFO:tasks.workunit.client.0.smithi063.stderr:+ expected_ret=1 2024-09-16T05:53:28.280 INFO:tasks.workunit.client.0.smithi063.stdout:ceph auth health | grep AUTH_BAD_CAP 2024-09-16T05:53:28.280 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo ceph auth health '|' grep AUTH_BAD_CAP 2024-09-16T05:53:28.280 INFO:tasks.workunit.client.0.smithi063.stderr:+ eval ceph auth health '|' grep AUTH_BAD_CAP 2024-09-16T05:53:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient: get_monmap_and_config 2024-09-16T05:53:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient: build_initial_monmap 2024-09-16T05:53:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient: monmap: 2024-09-16T05:53:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:28.323911+0000 2024-09-16T05:53:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:28.323911+0000 2024-09-16T05:53:28.322 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:28.323 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:28.323 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 1 Processor -- start 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 1 -- start start 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient: init 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient(hunting): picked mon.noname-c con 0x7fc27c0a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient(hunting): picked mon.noname-a con 0x7fc27c0a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient(hunting): picked mon.noname-b con 0x7fc27c0a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc27c06e520 con 0x7fc27c0a4290 2024-09-16T05:53:28.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc27c0a0780 con 0x7fc27c0a37a0 2024-09-16T05:53:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc27c0a4c70 con 0x7fc27c0a1a00 2024-09-16T05:53:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 10 monclient(hunting): authenticate will time out at 2070.130997s 2024-09-16T05:53:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc273fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc27c0a4290 0x7fc27c13bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53254/0 (socket says 172.21.15.6:53254) 2024-09-16T05:53:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc273fff640 1 -- 172.21.15.6:0/2424497399 learned_addr learned my addr 172.21.15.6:0/2424497399 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3861320664 0 0) 0x7fc27c06e520 con 0x7fc27c0a4290 2024-09-16T05:53:28.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): my global_id is 174300 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc268003530 con 0x7fc27c0a4290 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925419293 0 0) 0x7fc27c0a4c70 con 0x7fc27c0a1a00 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): my global_id is 164393 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc27c06e520 con 0x7fc27c0a1a00 2024-09-16T05:53:28.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906109897 0 0) 0x7fc27c0a0780 con 0x7fc27c0a37a0 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient(hunting): my global_id is 184174 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc27c0a4c70 con 0x7fc27c0a37a0 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3685071403 0 0) 0x7fc27c06e520 con 0x7fc27c0a1a00 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 >> v1:172.21.15.63:6789/0 conn(0x7fc27c0a37a0 legacy=0x7fc27c0a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 >> v1:172.21.15.6:6789/0 conn(0x7fc27c0a4290 legacy=0x7fc27c13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient: _finish_hunting 0 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 monclient: found mon.noname-b 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:53:28.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27c0a65f0 con 0x7fc27c0a1a00 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient: _finish_auth 0 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient: _check_auth_tickets 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 5 monclient: authenticate success, global_id 164393 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc274001980 con 0x7fc27c0a1a00 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc27affd640 10 monclient: dump: 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.334 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.320+0000 7fc281ebf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc274001c60 con 0x7fc27c0a1a00 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27affd640 1 -- 172.21.15.6:0/2424497399 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc274002070 con 0x7fc27c0a1a00 2024-09-16T05:53:28.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 4 set_mon_vals no callback set 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27affd640 10 monclient: dump: 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.336 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: shutdown 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- 172.21.15.6:0/2424497399 >> v1:172.21.15.6:6790/0 conn(0x7fc27c0a1a00 legacy=0x7fc27c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- 172.21.15.6:0/2424497399 shutdown_connections 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- 172.21.15.6:0/2424497399 >> 172.21.15.6:0/2424497399 conn(0x7fc27c0a58a0 msgr2=0x7fc27c0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- 172.21.15.6:0/2424497399 shutdown_connections 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- 172.21.15.6:0/2424497399 wait complete. 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: build_initial_monmap 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: monmap: 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:28.337 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 Processor -- start 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- start start 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: init 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:53:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient(hunting): picked mon.noname-a con 0x7fc27c0a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient(hunting): picked mon.noname-b con 0x7fc27c0a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient(hunting): picked mon.noname-c con 0x7fc27c0a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc27c133690 con 0x7fc27c0a1a00 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc27c135980 con 0x7fc27c0a37a0 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc27c1350e0 con 0x7fc27c0a4290 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient(hunting): authenticate will time out at 2070.134198s 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc27c0a1a00 0x7fc27c136eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53256/0 (socket says 172.21.15.6:53256) 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27b7fe640 1 -- 172.21.15.6:0/830456003 learned_addr learned my addr 172.21.15.6:0/830456003 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:28.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4098063766 0 0) 0x7fc27c1350e0 con 0x7fc27c0a4290 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): my global_id is 164396 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc25c003530 con 0x7fc27c0a4290 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324921488 0 0) 0x7fc27c133690 con 0x7fc27c0a1a00 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): my global_id is 174303 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc27c1350e0 con 0x7fc27c0a1a00 2024-09-16T05:53:28.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1957252397 0 0) 0x7fc27c135980 con 0x7fc27c0a37a0 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): my global_id is 184177 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc27c133690 con 0x7fc27c0a37a0 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1441634095 0 0) 0x7fc25c003530 con 0x7fc27c0a4290 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc27c135980 con 0x7fc27c0a4290 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc274001880 con 0x7fc27c0a4290 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): dump: 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.341 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 427860703 0 0) 0x7fc27c1350e0 con 0x7fc27c0a1a00 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc25c003530 con 0x7fc27c0a1a00 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc26c001880 con 0x7fc27c0a1a00 2024-09-16T05:53:28.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient(hunting): dump: 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1537973122 0 0) 0x7fc27c135980 con 0x7fc27c0a4290 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 >> v1:172.21.15.63:6789/0 conn(0x7fc27c0a37a0 legacy=0x7fc27c1d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 >> v1:172.21.15.6:6789/0 conn(0x7fc27c0a1a00 legacy=0x7fc27c136eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 monclient: found mon.c 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27c1375c0 con 0x7fc27c0a4290 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient: _finish_auth 0 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 5 monclient: authenticate success, global_id 164396 2024-09-16T05:53:28.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: _renew_subs 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc27c134430 con 0x7fc27c0a4290 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: _renew_subs 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc27c1c89f0 con 0x7fc27c0a4290 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc274001c60 con 0x7fc27c0a4290 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc274003ed0 con 0x7fc27c0a4290 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc27a7fc640 4 set_mon_vals no callback set 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc278ff9640 10 monclient: dump: 2024-09-16T05:53:28.345 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.324+0000 7fc281ebf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc2600065d0 con 0x7fc260002870 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc27400ec70 con 0x7fc27c0a4290 2024-09-16T05:53:28.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fc274043e20 con 0x7fc27c0a4290 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877597062 0 0) 0x7fc2600065d0 con 0x7fc260002870 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient(hunting): my global_id is 184180 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc27c135980 con 0x7fc260002870 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc273fff640 10 monclient: get_auth_request con 0x7fc25c036cc0 auth_method 0 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2810774099 0 0) 0x7fc27c135980 con 0x7fc260002870 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2600065d0 con 0x7fc260002870 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc26c0037a0 con 0x7fc260002870 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:28.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient: dump: 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1918674272 0 0) 0x7fc2600065d0 con 0x7fc260002870 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc26000a160 con 0x7fc260002870 2024-09-16T05:53:28.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc26c0057d0 con 0x7fc260002870 2024-09-16T05:53:28.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:28.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:28.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 >> v1:172.21.15.63:6789/0 conn(0x7fc260002870 legacy=0x7fc260004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.328+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 >> v1:172.21.15.63:6789/0 conn(0x7fc260002870 legacy=0x7fc260004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc260009e30 con 0x7fc2600090f0 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 510669688 0 0) 0x7fc260009e30 con 0x7fc2600090f0 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient(hunting): my global_id is 184183 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2600065d0 con 0x7fc2600090f0 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1805659514 0 0) 0x7fc2600065d0 con 0x7fc2600090f0 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc260009e30 con 0x7fc2600090f0 2024-09-16T05:53:28.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc26c003eb0 con 0x7fc2600090f0 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient: dump: 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:28.351 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3283292496 0 0) 0x7fc260009e30 con 0x7fc2600090f0 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc26000c3a0 con 0x7fc2600090f0 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3960 (unknown 0 0 806649428) 0x7fc26c005980 con 0x7fc2600090f0 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 >> v1:172.21.15.63:6789/0 conn(0x7fc2600090f0 legacy=0x7fc260009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc278ff9640 1 -- 172.21.15.6:0/830456003 >> v1:172.21.15.63:6789/0 conn(0x7fc2600090f0 legacy=0x7fc260009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:53:28.352 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 80, 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 3031, 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 2, 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:53:28.353 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.355 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:53:28.356 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:28.357 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:28.358 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.359 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:28.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:28.363 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:28.364 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:28.364 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:28.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:28.364 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:53:28.364 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:53:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc25c036cc0 legacy=0x7fc25c039180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 10 monclient: shutdown 2024-09-16T05:53:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 >> v1:172.21.15.6:6790/0 conn(0x7fc27c0a4290 legacy=0x7fc27c1d3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc27bfff640 1 -- 172.21.15.6:0/830456003 reap_dead start 2024-09-16T05:53:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc27bfff640 1 -- 172.21.15.6:0/830456003 reap_dead start 2024-09-16T05:53:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 shutdown_connections 2024-09-16T05:53:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 >> 172.21.15.6:0/830456003 conn(0x7fc27c0a58a0 msgr2=0x7fc27c0a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 shutdown_connections 2024-09-16T05:53:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:28.336+0000 7fc281ebf640 1 -- 172.21.15.6:0/830456003 wait complete. 2024-09-16T05:53:28.366 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:53:28.366 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T05:53:28.366 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T05:53:28.367 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T05:53:28.367 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:53:28.367 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T05:53:28.367 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:53:28.379 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T05:53:28.379 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T05:53:28.379 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T05:53:28.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:53:28.620 INFO:tasks.workunit.client.0.smithi063.stderr:+ ret=1 2024-09-16T05:53:28.620 INFO:tasks.workunit.client.0.smithi063.stderr:+ [[ 1 -ne 1 ]] 2024-09-16T05:53:28.620 INFO:tasks.workunit.client.0.smithi063.stderr:+ return 0 2024-09-16T05:53:28.620 INFO:tasks.workunit.client.0.smithi063.stdout:OK 2024-09-16T05:53:28.621 INFO:tasks.workunit.client.0.smithi063.stderr:+ echo OK 2024-09-16T05:53:28.621 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:53:28.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:53:28.634 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2024-09-16T05:53:28.634 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T05:53:28.967 DEBUG:teuthology.parallel:result is None 2024-09-16T05:53:28.967 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T05:53:28.980 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T05:53:28.980 DEBUG:teuthology.orchestra.run.smithi063:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T05:53:29.040 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T05:53:29.041 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T05:53:29.049 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T05:53:43.380 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T05:53:43.381 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T05:53:43.381 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T05:53:43.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:53:43.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient: get_monmap_and_config 2024-09-16T05:53:43.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient: build_initial_monmap 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient: monmap: 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:43.516115+0000 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:43.516115+0000 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:43.515 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:43.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 1 Processor -- start 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 1 -- start start 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient: init 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient(hunting): picked mon.noname-c con 0x7f077009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient(hunting): picked mon.noname-a con 0x7f077009e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient(hunting): picked mon.noname-b con 0x7f07700a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07700a1370 con 0x7f077009e230 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f077009ead0 con 0x7f077009aa30 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f077009edb0 con 0x7f07700a0da0 2024-09-16T05:53:43.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:43.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f0776a90640 10 monclient(hunting): authenticate will time out at 2085.322958s 2024-09-16T05:53:43.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f076ffff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f07701399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:43.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f077502d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f07700a0da0 0x7f077013d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35534/0 (socket says 172.21.15.6:35534) 2024-09-16T05:53:43.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.512+0000 7f077502d640 1 -- 172.21.15.6:0/3513214755 learned_addr learned my addr 172.21.15.6:0/3513214755 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:43.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:43.716+0000 7f076ffff640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f07701399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:44.120+0000 7f076ffff640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f07701399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:44.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:44.924+0000 7f076ffff640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f07701399d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:46.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 10 monclient(hunting): tick 2024-09-16T05:53:46.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T05:53:46.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T05:53:46.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T05:53:46.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6790/0 conn(0x7f07700a0da0 legacy=0x7f077013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.63:6789/0 conn(0x7f077009aa30 legacy=0x7f077009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f07701399d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:53:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:53:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f07540036d0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f077009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:46.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f077009e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 1 -- 172.21.15.6:0/3513214755 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f077009dd50 con 0x7f077009e230 2024-09-16T05:53:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f077502d640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f075400cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 1 -- 172.21.15.6:0/3513214755 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07700a1370 con 0x7f077009aa30 2024-09-16T05:53:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 1 -- 172.21.15.6:0/3513214755 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07540103b0 con 0x7f07540036d0 2024-09-16T05:53:46.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.512+0000 7f076e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:46.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:46.716+0000 7f077502d640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f075400cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:47.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:47.120+0000 7f077502d640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f075400cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:47.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:47.924+0000 7f077502d640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f075400cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2771252331 0 0) 0x7f07540103b0 con 0x7f07540036d0 2024-09-16T05:53:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient(hunting): my global_id is 164405 2024-09-16T05:53:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0750003530 con 0x7f07540036d0 2024-09-16T05:53:48.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 675626038 0 0) 0x7f0750003530 con 0x7f07540036d0 2024-09-16T05:53:48.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.63:6789/0 conn(0x7f077009aa30 legacy=0x7f07540093e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f075400cbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:53:48.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 monclient: found mon.noname-b 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0754011660 con 0x7f07540036d0 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0760001560 con 0x7f07540036d0 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f0776a90640 5 monclient: authenticate success, global_id 164405 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f0776a90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: dump: 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f0776a90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07600017c0 con 0x7f07540036d0 2024-09-16T05:53:48.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 1 -- 172.21.15.6:0/3513214755 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0760001c10 con 0x7f07540036d0 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f076f7fe640 10 monclient: dump: 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.436 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.437 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f0776a90640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:48.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.428+0000 7f0776a90640 4 set_mon_vals no callback set 2024-09-16T05:53:48.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient: shutdown 2024-09-16T05:53:48.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:48.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- 172.21.15.6:0/3513214755 >> v1:172.21.15.6:6790/0 conn(0x7f07540036d0 legacy=0x7f0754005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- 172.21.15.6:0/3513214755 shutdown_connections 2024-09-16T05:53:48.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- 172.21.15.6:0/3513214755 >> 172.21.15.6:0/3513214755 conn(0x7f07700a1650 msgr2=0x7f07700a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- 172.21.15.6:0/3513214755 shutdown_connections 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- 172.21.15.6:0/3513214755 wait complete. 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient: build_initial_monmap 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient: monmap: 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 Processor -- start 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- start start 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient: init 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient(hunting): picked mon.noname-b con 0x7f077009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient(hunting): picked mon.noname-a con 0x7f077009e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient(hunting): picked mon.noname-c con 0x7f0770134de0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0770122090 con 0x7f077009e230 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07701353d0 con 0x7f077009aa30 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f077006eba0 con 0x7f0770134de0 2024-09-16T05:53:48.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f0776a90640 10 monclient(hunting): authenticate will time out at 2090.243675s 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f076ffff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f07701ce000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f077502d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0770134de0 0x7f07701d1730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35548/0 (socket says 172.21.15.6:35548) 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.432+0000 7f077502d640 1 -- 172.21.15.6:0/3147163842 learned_addr learned my addr 172.21.15.6:0/3147163842 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685546823 0 0) 0x7f077006eba0 con 0x7f0770134de0 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient(hunting): my global_id is 164408 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0754005220 con 0x7f0770134de0 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2261064219 0 0) 0x7f0754005220 con 0x7f0770134de0 2024-09-16T05:53:48.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f077006eba0 con 0x7f0770134de0 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0768001840 con 0x7f0770134de0 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient(hunting): dump: 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.444 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2176869807 0 0) 0x7f077006eba0 con 0x7f0770134de0 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 >> v1:172.21.15.63:6789/0 conn(0x7f077009aa30 legacy=0x7f07701346d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 >> v1:172.21.15.6:6789/0 conn(0x7f077009e230 legacy=0x7f07701ce000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient: _finish_hunting 0 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 monclient: found mon.c 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0770121ad0 con 0x7f0770134de0 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient: _finish_auth 0 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient: _check_auth_tickets 2024-09-16T05:53:48.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 5 monclient: authenticate success, global_id 164408 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 10 monclient: _renew_subs 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0770121db0 con 0x7f0770134de0 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 10 monclient: _renew_subs 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f07701c8f00 con 0x7f0770134de0 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07680047f0 con 0x7f0770134de0 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0768004b40 con 0x7f0770134de0 2024-09-16T05:53:48.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076e7fc640 4 set_mon_vals no callback set 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 10 monclient: dump: 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.447 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f0768004d60 con 0x7f0770134de0 2024-09-16T05:53:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0748005090 con 0x7f0770134de0 2024-09-16T05:53:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f0768044f50 con 0x7f0770134de0 2024-09-16T05:53:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.436+0000 7f077482c640 10 monclient: get_auth_request con 0x7f07540372e0 auth_method 0 2024-09-16T05:53:48.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.440+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f07680154c0 con 0x7f0770134de0 2024-09-16T05:53:48.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.440+0000 7f076d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:48.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.440+0000 7f076d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.676+0000 7f0776a90640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.676+0000 7f0776a90640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.676+0000 7f0776a90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.676+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0748005380 con 0x7f0770134de0 2024-09-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.676+0000 7f076d7fa640 1 -- 172.21.15.6:0/3147163842 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1391 (unknown 254746657 0 359818795) 0x7f0768019170 con 0x7f0770134de0 2024-09-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.676+0000 7f076d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.676+0000 7f076d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:48.682 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:53:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.680+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 >> v1:172.21.15.63:6812/3643207211 conn(0x7f07540372e0 legacy=0x7f07540397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.680+0000 7f0776a90640 10 monclient: shutdown 2024-09-16T05:53:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.680+0000 7f0776a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.680+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 >> v1:172.21.15.6:6790/0 conn(0x7f0770134de0 legacy=0x7f07701d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.680+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 shutdown_connections 2024-09-16T05:53:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.680+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 >> 172.21.15.6:0/3147163842 conn(0x7f07700a1650 msgr2=0x7f07700a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.680+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 shutdown_connections 2024-09-16T05:53:48.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.680+0000 7f0776a90640 1 -- 172.21.15.6:0/3147163842 wait complete. 2024-09-16T05:53:48.701 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T05:53:48.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T05:53:48.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:53:48.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient: get_monmap_and_config 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient: build_initial_monmap 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient: monmap: 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:48.823479+0000 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:48.823479+0000 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:48.822 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:48.823 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 1 Processor -- start 2024-09-16T05:53:48.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 1 -- start start 2024-09-16T05:53:48.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient: init 2024-09-16T05:53:48.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:48.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient(hunting): picked mon.noname-c con 0x7feb040a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient(hunting): picked mon.noname-a con 0x7feb040a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient(hunting): picked mon.noname-b con 0x7feb040a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb0406e520 con 0x7feb040a4290 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb040a0780 con 0x7feb040a37a0 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb040a4c70 con 0x7feb040a1a00 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 10 monclient(hunting): authenticate will time out at 2090.630781s 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb08f60640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7feb040a4290 legacy=0x7feb0413bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb09f62640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7feb040a1a00 0x7feb0413f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35558/0 (socket says 172.21.15.6:35558) 2024-09-16T05:53:48.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb09f62640 1 -- 172.21.15.6:0/102959963 learned_addr learned my addr 172.21.15.6:0/102959963 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3388836618 0 0) 0x7feb040a4c70 con 0x7feb040a1a00 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient(hunting): my global_id is 164411 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaf0003530 con 0x7feb040a1a00 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3503687194 0 0) 0x7feb040a0780 con 0x7feb040a37a0 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient(hunting): my global_id is 184186 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb040a4c70 con 0x7feb040a37a0 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2674072701 0 0) 0x7feaf0003530 con 0x7feb040a1a00 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 >> v1:172.21.15.63:6789/0 conn(0x7feb040a37a0 legacy=0x7feb040a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 >> v1:172.21.15.6:6789/0 conn(0x7feb040a4290 legacy=0x7feb0413bab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient: _finish_hunting 0 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 monclient: found mon.noname-b 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb040a65f0 con 0x7feb040a1a00 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient: _finish_auth 0 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient: _check_auth_tickets 2024-09-16T05:53:48.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 5 monclient: authenticate success, global_id 164411 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feafc0019a0 con 0x7feb040a1a00 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feafbfff640 10 monclient: dump: 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.830 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.820+0000 7feb0b9c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feafc001cc0 con 0x7feb040a1a00 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feafbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 4 set_mon_vals no callback set 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feafbfff640 1 -- 172.21.15.6:0/102959963 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feafc002090 con 0x7feb040a1a00 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feafbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feafbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feafbfff640 10 monclient: dump: 2024-09-16T05:53:48.831 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: shutdown 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- 172.21.15.6:0/102959963 >> v1:172.21.15.6:6790/0 conn(0x7feb040a1a00 legacy=0x7feb0413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- 172.21.15.6:0/102959963 shutdown_connections 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- 172.21.15.6:0/102959963 >> 172.21.15.6:0/102959963 conn(0x7feb040a58a0 msgr2=0x7feb040a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- 172.21.15.6:0/102959963 shutdown_connections 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- 172.21.15.6:0/102959963 wait complete. 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: build_initial_monmap 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: monmap: 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.833 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:48.834 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:48.834 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:48.834 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 Processor -- start 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- start start 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: init 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient(hunting): picked mon.noname-c con 0x7feb040a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient(hunting): picked mon.noname-a con 0x7feb040a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient(hunting): picked mon.noname-b con 0x7feb040a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb04133640 con 0x7feb040a37a0 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb04135930 con 0x7feb040a4290 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb04135090 con 0x7feb040a1a00 2024-09-16T05:53:48.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient(hunting): authenticate will time out at 2090.634304s 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb08f60640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7feb040a37a0 legacy=0x7feb041d05c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb09761640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7feb040a1a00 0x7feb04136e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35566/0 (socket says 172.21.15.6:35566) 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb09761640 1 -- 172.21.15.6:0/269713369 learned_addr learned my addr 172.21.15.6:0/269713369 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1161103066 0 0) 0x7feb04135090 con 0x7feb040a1a00 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): my global_id is 164414 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feae8003530 con 0x7feb040a1a00 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1256184135 0 0) 0x7feae8003530 con 0x7feb040a1a00 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb04135090 con 0x7feb040a1a00 2024-09-16T05:53:48.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feb00001980 con 0x7feb040a1a00 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): dump: 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.841 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1262899357 0 0) 0x7feb04135930 con 0x7feb040a4290 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient(hunting): my global_id is 184189 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feae8003530 con 0x7feb040a4290 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2815429420 0 0) 0x7feb04135090 con 0x7feb040a1a00 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 >> v1:172.21.15.63:6789/0 conn(0x7feb040a4290 legacy=0x7feb041d3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 >> v1:172.21.15.6:6789/0 conn(0x7feb040a37a0 legacy=0x7feb041d05c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 monclient: found mon.c 2024-09-16T05:53:48.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb041343c0 con 0x7feb040a1a00 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient: _finish_auth 0 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 5 monclient: authenticate success, global_id 164414 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: _renew_subs 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb04137570 con 0x7feb040a1a00 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: _renew_subs 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feb041cbef0 con 0x7feb040a1a00 2024-09-16T05:53:48.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb00001ca0 con 0x7feb040a1a00 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feaf9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feafb7fe640 4 set_mon_vals no callback set 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.824+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feae40065d0 con 0x7feae4002870 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feb00003ab0 con 0x7feb040a1a00 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient: dump: 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.844 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2730217771 0 0) 0x7feae40065d0 con 0x7feae4002870 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient(hunting): my global_id is 164417 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb04135090 con 0x7feae4002870 2024-09-16T05:53:48.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1170022203 0 0) 0x7feb04135090 con 0x7feae4002870 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feae40065d0 con 0x7feae4002870 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7feb00003c90 con 0x7feb040a1a00 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feaf4003ef0 con 0x7feae4002870 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient: dump: 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.846 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7feb00044f40 con 0x7feb040a1a00 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3607161514 0 0) 0x7feae40065d0 con 0x7feae4002870 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feae400a160 con 0x7feae4002870 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feb09f62640 10 monclient: get_auth_request con 0x7feae8036e60 auth_method 0 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7feaf4003e50 con 0x7feae4002870 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 >> v1:172.21.15.6:6790/0 conn(0x7feae4002870 legacy=0x7feae4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.828+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 >> v1:172.21.15.6:6790/0 conn(0x7feae4002870 legacy=0x7feae4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.832+0000 7feb0b9c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.832+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feae4009e60 con 0x7feae4009120 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.832+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596157028 0 0) 0x7feae4009e60 con 0x7feae4009120 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.832+0000 7feaf9ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.832+0000 7feaf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.832+0000 7feaf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.832+0000 7feaf9ffb640 10 monclient(hunting): my global_id is 164420 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.832+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feae40065d0 con 0x7feae4009120 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 802111416 0 0) 0x7feae40065d0 con 0x7feae4009120 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feae4009e60 con 0x7feae4009120 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7feaf40047a0 con 0x7feae4009120 2024-09-16T05:53:48.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 10 monclient: dump: 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2824021123 0 0) 0x7feae4009e60 con 0x7feae4009120 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feae400c3e0 con 0x7feae4009120 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4285 (unknown 0 0 94404178) 0x7feaf40049d0 con 0x7feae4009120 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 >> v1:172.21.15.6:6790/0 conn(0x7feae4009120 legacy=0x7feae40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feaf9ffb640 1 -- 172.21.15.6:0/269713369 >> v1:172.21.15.6:6790/0 conn(0x7feae4009120 legacy=0x7feae40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:48.850 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 82, 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 239797, 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:48.851 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:53:48.852 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:48.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:53:48.854 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:48.855 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:48.856 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:48.857 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:48.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.859 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:48.860 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:48.861 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:48.862 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:53:48.863 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:53:48.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 >> v1:172.21.15.63:6812/3643207211 conn(0x7feae8036e60 legacy=0x7feae8039320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb0b9c5640 10 monclient: shutdown 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb0b9c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 >> v1:172.21.15.6:6790/0 conn(0x7feb040a1a00 legacy=0x7feb04136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb09f62640 1 -- 172.21.15.6:0/269713369 reap_dead start 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb09f62640 1 -- 172.21.15.6:0/269713369 reap_dead start 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 shutdown_connections 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 >> 172.21.15.6:0/269713369 conn(0x7feb040a58a0 msgr2=0x7feb040a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 shutdown_connections 2024-09-16T05:53:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.836+0000 7feb0b9c5640 1 -- 172.21.15.6:0/269713369 wait complete. 2024-09-16T05:53:48.866 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:53:48.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T05:53:48.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:53:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient: get_monmap_and_config 2024-09-16T05:53:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient: build_initial_monmap 2024-09-16T05:53:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient: monmap: 2024-09-16T05:53:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:53:48.984663+0000 2024-09-16T05:53:48.983 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:53:48.984663+0000 2024-09-16T05:53:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:53:48.984 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:53:48.984 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 1 Processor -- start 2024-09-16T05:53:48.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 1 -- start start 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient: init 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient(hunting): picked mon.noname-b con 0x7f04380a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient(hunting): picked mon.noname-a con 0x7f04380a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient(hunting): picked mon.noname-c con 0x7f04380a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f043806e520 con 0x7f04380a4290 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04380a0780 con 0x7f04380a1a00 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04380a4c70 con 0x7f04380a37a0 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.980+0000 7f043df04640 10 monclient(hunting): authenticate will time out at 2090.791957s 2024-09-16T05:53:48.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f0436ffd640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f04380a4290 legacy=0x7f043813bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04377fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f04380a37a0 0x7f04380a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35596/0 (socket says 172.21.15.6:35596) 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04377fe640 1 -- 172.21.15.6:0/3461167258 learned_addr learned my addr 172.21.15.6:0/3461167258 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1267508307 0 0) 0x7f04380a4c70 con 0x7f04380a37a0 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient(hunting): _negotiate 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient(hunting): my global_id is 164423 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0420003530 con 0x7f04380a37a0 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3016786994 0 0) 0x7f0420003530 con 0x7f04380a37a0 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 >> v1:172.21.15.63:6789/0 conn(0x7f04380a1a00 legacy=0x7f043813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 >> v1:172.21.15.6:6789/0 conn(0x7f04380a4290 legacy=0x7f043813bab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T05:53:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: _finish_hunting 0 2024-09-16T05:53:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 monclient: found mon.noname-b 2024-09-16T05:53:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:53:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04380a65f0 con 0x7f04380a37a0 2024-09-16T05:53:48.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: _finish_auth 0 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: _check_auth_tickets 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0428001620 con 0x7f04380a37a0 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: dump: 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.997 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 5 monclient: authenticate success, global_id 164423 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0428001900 con 0x7f04380a37a0 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 1 -- 172.21.15.6:0/3461167258 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0428001cd0 con 0x7f04380a37a0 2024-09-16T05:53:48.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: get_monmap_and_config success 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 4 set_mon_vals no callback set 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f04367fc640 10 monclient: dump: 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:48.999 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:49.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: shutdown 2024-09-16T05:53:49.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:49.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- 172.21.15.6:0/3461167258 >> v1:172.21.15.6:6790/0 conn(0x7f04380a37a0 legacy=0x7f04380a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:49.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- 172.21.15.6:0/3461167258 shutdown_connections 2024-09-16T05:53:49.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- 172.21.15.6:0/3461167258 >> 172.21.15.6:0/3461167258 conn(0x7f04380a58a0 msgr2=0x7f04380a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:49.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- 172.21.15.6:0/3461167258 shutdown_connections 2024-09-16T05:53:49.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- 172.21.15.6:0/3461167258 wait complete. 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: build_initial_monmap 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: monmap: 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:49.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 Processor -- start 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- start start 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: init 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: _reopen_session rank -1 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient(hunting): picked mon.noname-b con 0x7f04380a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient(hunting): picked mon.noname-c con 0x7f04380a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient(hunting): picked mon.noname-a con 0x7f04380a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04380a0100 con 0x7f04380a4290 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0438134270 con 0x7f04380a1a00 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0438133710 con 0x7f04380a37a0 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient(hunting): _renew_subs 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient(hunting): authenticate will time out at 2090.795127s 2024-09-16T05:53:49.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f0437fff640 1 -- >> v1:172.21.15.6:6789/0 conn(0x7f04380a4290 legacy=0x7f04381d3cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.6:6789/0 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f0436ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f04380a37a0 0x7f04381d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35610/0 (socket says 172.21.15.6:35610) 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f0436ffd640 1 -- 172.21.15.6:0/2329430983 learned_addr learned my addr 172.21.15.6:0/2329430983 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1851857419 0 0) 0x7f0438133710 con 0x7f04380a37a0 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient(hunting): my global_id is 164426 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0424003530 con 0x7f04380a37a0 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2789855748 0 0) 0x7f0424003530 con 0x7f04380a37a0 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0438133710 con 0x7f04380a37a0 2024-09-16T05:53:49.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f042c004160 con 0x7f04380a37a0 2024-09-16T05:53:49.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:53:49.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:53:49.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient(hunting): dump: 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:49.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 904908529 0 0) 0x7f0438133710 con 0x7f04380a37a0 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 >> v1:172.21.15.63:6789/0 conn(0x7f04380a1a00 legacy=0x7f0438136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 >> v1:172.21.15.6:6789/0 conn(0x7f04380a4290 legacy=0x7f04381d3cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 monclient: found mon.c 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0438135720 con 0x7f04380a37a0 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient: _finish_auth 0 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f041ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:53:49.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 5 monclient: authenticate success, global_id 164426 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: _renew_subs 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.984+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0438135a00 con 0x7f04380a37a0 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f043df04640 10 monclient: _renew_subs 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f043df04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f04381344b0 con 0x7f04380a37a0 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f042c0043c0 con 0x7f04380a37a0 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f042c004710 con 0x7f04380a37a0 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: dump: 2024-09-16T05:53:49.007 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f0435ffb640 4 set_mon_vals no callback set 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f043df04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f04180065d0 con 0x7f0418002870 2024-09-16T05:53:49.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f042c0048b0 con 0x7f04380a37a0 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f042c044d70 con 0x7f04380a37a0 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f0437fff640 10 monclient: get_auth_request con 0x7f04240369d0 auth_method 0 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1971284726 0 0) 0x7f04180065d0 con 0x7f0418002870 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient(hunting): my global_id is 184198 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0438133710 con 0x7f0418002870 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1256677050 0 0) 0x7f0438133710 con 0x7f0418002870 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04180065d0 con 0x7f0418002870 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0428001890 con 0x7f0418002870 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:49.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: dump: 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2689466896 0 0) 0x7f04180065d0 con 0x7f0418002870 2024-09-16T05:53:49.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f041800a160 con 0x7f0418002870 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0428001610 con 0x7f0418002870 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 >> v1:172.21.15.63:6789/0 conn(0x7f0418002870 legacy=0x7f0418004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.988+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 >> v1:172.21.15.63:6789/0 conn(0x7f0418002870 legacy=0x7f0418004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f043df04640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0418009e50 con 0x7f0418009110 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1156505456 0 0) 0x7f0418009e50 con 0x7f0418009110 2024-09-16T05:53:49.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient(hunting): my global_id is 184201 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04180065d0 con 0x7f0418009110 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2429150091 0 0) 0x7f04180065d0 con 0x7f0418009110 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0418009e50 con 0x7f0418009110 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f0428004b40 con 0x7f0418009110 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient: dump: 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:53:49.013 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2914713052 0 0) 0x7f0418009e50 con 0x7f0418009110 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f041800c3e0 con 0x7f0418009110 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3952 (unknown 0 0 3898537045) 0x7f0428005f80 con 0x7f0418009110 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:53:49.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 >> v1:172.21.15.63:6789/0 conn(0x7f0418009110 legacy=0x7f04180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:48.996+0000 7f041ffff640 1 -- 172.21.15.6:0/2329430983 >> v1:172.21.15.63:6789/0 conn(0x7f0418009110 legacy=0x7f04180095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 82, 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 23695, 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:53:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:53:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:53:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:53:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:53:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:53:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:53:49.026 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 >> v1:172.21.15.63:6812/3643207211 conn(0x7f04240369d0 legacy=0x7f0424038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f043df04640 10 monclient: shutdown 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f043df04640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 >> v1:172.21.15.6:6790/0 conn(0x7f04380a37a0 legacy=0x7f04381d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f0437fff640 1 -- 172.21.15.6:0/2329430983 reap_dead start 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f0437fff640 1 -- 172.21.15.6:0/2329430983 reap_dead start 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 shutdown_connections 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 >> 172.21.15.6:0/2329430983 conn(0x7f04380a58a0 msgr2=0x7f04380a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 shutdown_connections 2024-09-16T05:53:49.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:53:49.000+0000 7f043df04640 1 -- 172.21.15.6:0/2329430983 wait complete. 2024-09-16T05:53:49.029 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T05:53:49.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T05:55:19.029 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T05:55:19.030 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T05:55:19.030 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T05:55:19.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T05:55:19.036 INFO:tasks.ceph.mon.a:Started 2024-09-16T05:55:19.036 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T05:55:19.037 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T05:55:19.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T05:55:34.037 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T05:55:34.038 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T05:55:34.039 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:55:34.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:55:34.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.167+0000 7f6021707640 10 monclient: get_monmap_and_config 2024-09-16T05:55:34.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.167+0000 7f6021707640 10 monclient: build_initial_monmap 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.167+0000 7f6021707640 10 monclient: monmap: 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:34.173034+0000 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:34.173034+0000 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:34.172 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.167+0000 7f6021707640 1 Processor -- start 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.167+0000 7f6021707640 1 -- start start 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.167+0000 7f6021707640 10 monclient: init 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 10 monclient(hunting): picked mon.noname-b con 0x7f601c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 10 monclient(hunting): picked mon.noname-c con 0x7f601c09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 10 monclient(hunting): picked mon.noname-a con 0x7f601c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c0a1370 con 0x7f601c0a0da0 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c09ead0 con 0x7f601c09e230 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c09edb0 con 0x7f601c09aa30 2024-09-16T05:55:34.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:34.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f6021707640 10 monclient(hunting): authenticate will time out at 2195.980183s 2024-09-16T05:55:34.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f601affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f601c09aa30 0x7f601c09ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39450/0 (socket says 172.21.15.6:39450) 2024-09-16T05:55:34.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f601affd640 1 -- 172.21.15.6:0/339986660 learned_addr learned my addr 172.21.15.6:0/339986660 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:34.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:34.171+0000 7f601b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f601c0a0da0 0x7f601c13d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39132/0 (socket says 172.21.15.6:39132) 2024-09-16T05:55:35.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094642986 0 0) 0x7f601c0a1370 con 0x7f601c0a0da0 2024-09-16T05:55:35.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient(hunting): my global_id is 194097 2024-09-16T05:55:35.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6004003530 con 0x7f601c0a0da0 2024-09-16T05:55:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1892996139 0 0) 0x7f6004003530 con 0x7f601c0a0da0 2024-09-16T05:55:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 >> v1:172.21.15.6:6790/0 conn(0x7f601c09aa30 legacy=0x7f601c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 >> v1:172.21.15.63:6789/0 conn(0x7f601c09e230 legacy=0x7f601c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:55:35.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 monclient: found mon.noname-a 2024-09-16T05:55:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c09dd50 con 0x7f601c0a0da0 2024-09-16T05:55:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: _finish_auth 0 2024-09-16T05:55:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:55:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f60140019a0 con 0x7f601c0a0da0 2024-09-16T05:55:35.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:35.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: dump: 2024-09-16T05:55:35.286 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.286 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.286 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.286 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.287 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 5 monclient: authenticate success, global_id 194097 2024-09-16T05:55:35.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:35.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6014001c00 con 0x7f601c0a0da0 2024-09-16T05:55:35.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:35.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 1 -- 172.21.15.6:0/339986660 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6014001f90 con 0x7f601c0a0da0 2024-09-16T05:55:35.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:35.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 4 set_mon_vals no callback set 2024-09-16T05:55:35.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6019ffb640 10 monclient: dump: 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.289 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 10 monclient: shutdown 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 1 -- 172.21.15.6:0/339986660 >> v1:172.21.15.6:6789/0 conn(0x7f601c0a0da0 legacy=0x7f601c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 1 -- 172.21.15.6:0/339986660 shutdown_connections 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 1 -- 172.21.15.6:0/339986660 >> 172.21.15.6:0/339986660 conn(0x7f601c0a1650 msgr2=0x7f601c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 1 -- 172.21.15.6:0/339986660 shutdown_connections 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.279+0000 7f6021707640 1 -- 172.21.15.6:0/339986660 wait complete. 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: build_initial_monmap 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: monmap: 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:35.294 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 1 Processor -- start 2024-09-16T05:55:35.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 1 -- start start 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: init 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient(hunting): picked mon.noname-c con 0x7f601c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient(hunting): picked mon.noname-b con 0x7f601c09e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient(hunting): picked mon.noname-a con 0x7f601c0a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c136e40 con 0x7f601c0a0da0 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c137120 con 0x7f601c09e230 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c1cc4f0 con 0x7f601c09aa30 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:35.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f601affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f601c09aa30 0x7f601c1cb6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39454/0 (socket says 172.21.15.6:39454) 2024-09-16T05:55:35.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f601affd640 1 -- 172.21.15.6:0/1317871845 learned_addr learned my addr 172.21.15.6:0/1317871845 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:35.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient(hunting): authenticate will time out at 2197.094270s 2024-09-16T05:55:35.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1908358190 0 0) 0x7f601c136e40 con 0x7f601c0a0da0 2024-09-16T05:55:35.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient(hunting): my global_id is 194100 2024-09-16T05:55:35.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6008003530 con 0x7f601c0a0da0 2024-09-16T05:55:35.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 204079801 0 0) 0x7f6008003530 con 0x7f601c0a0da0 2024-09-16T05:55:35.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f601c136e40 con 0x7f601c0a0da0 2024-09-16T05:55:35.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f60140018a0 con 0x7f601c0a0da0 2024-09-16T05:55:35.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:35.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:35.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient(hunting): dump: 2024-09-16T05:55:35.299 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.299 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.299 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.300 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.300 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.300 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.300 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.300 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.301 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.301 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:35.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3497126718 0 0) 0x7f601c136e40 con 0x7f601c0a0da0 2024-09-16T05:55:35.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 >> v1:172.21.15.6:6790/0 conn(0x7f601c09aa30 legacy=0x7f601c1cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 >> v1:172.21.15.63:6789/0 conn(0x7f601c09e230 legacy=0x7f601c1cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient: _finish_hunting 0 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 monclient: found mon.a 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c06ea00 con 0x7f601c0a0da0 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient: _finish_auth 0 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient: _check_auth_tickets 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 5 monclient: authenticate success, global_id 194100 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: _renew_subs 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f601c135a50 con 0x7f601c0a0da0 2024-09-16T05:55:35.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: _renew_subs 2024-09-16T05:55:35.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f601c136580 con 0x7f601c0a0da0 2024-09-16T05:55:35.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6014001b40 con 0x7f601c0a0da0 2024-09-16T05:55:35.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:35.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6014005060 con 0x7f601c0a0da0 2024-09-16T05:55:35.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:35.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f60037fe640 10 monclient: dump: 2024-09-16T05:55:35.305 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.305 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.306 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.306 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.306 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.306 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.306 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.306 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.306 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.306 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.283+0000 7f6018ff9640 4 set_mon_vals no callback set 2024-09-16T05:55:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.287+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f6014005280 con 0x7f601c0a0da0 2024-09-16T05:55:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.287+0000 7f6021707640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.287+0000 7f6021707640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.287+0000 7f6021707640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.287+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc005090 con 0x7f601c0a0da0 2024-09-16T05:55:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.287+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f6014045290 con 0x7f601c0a0da0 2024-09-16T05:55:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.287+0000 7f601affd640 10 monclient: get_auth_request con 0x7f6008036a10 auth_method 0 2024-09-16T05:55:35.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.291+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f6014015920 con 0x7f601c0a0da0 2024-09-16T05:55:35.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.291+0000 7f60037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:35.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.291+0000 7f60037fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.523+0000 7f6021707640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:55:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.523+0000 7f6021707640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:55:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.523+0000 7f6021707640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.523+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5ffc005380 con 0x7f601c0a0da0 2024-09-16T05:55:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.523+0000 7f60037fe640 1 -- 172.21.15.6:0/1317871845 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2043464132) 0x7f6014015220 con 0x7f601c0a0da0 2024-09-16T05:55:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.523+0000 7f60037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:55:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.523+0000 7f60037fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:55:35.528 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:55:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.527+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6008036a10 legacy=0x7f6008038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.527+0000 7f6021707640 10 monclient: shutdown 2024-09-16T05:55:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.527+0000 7f6021707640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.527+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 >> v1:172.21.15.6:6789/0 conn(0x7f601c0a0da0 legacy=0x7f601c1d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.527+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 shutdown_connections 2024-09-16T05:55:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.527+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 >> 172.21.15.6:0/1317871845 conn(0x7f601c0a1650 msgr2=0x7f601c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.527+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 shutdown_connections 2024-09-16T05:55:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.527+0000 7f6021707640 1 -- 172.21.15.6:0/1317871845 wait complete. 2024-09-16T05:55:35.546 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:55:35.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T05:55:35.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T05:55:35.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient: get_monmap_and_config 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient: build_initial_monmap 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient: monmap: 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:35.667290+0000 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:35.667290+0000 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:35.666 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:35.667 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 Processor -- start 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- start start 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient: init 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient(hunting): picked mon.noname-c con 0x7efe1009bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient(hunting): picked mon.noname-a con 0x7efe1009bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient(hunting): picked mon.noname-b con 0x7efe101397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe100a1350 con 0x7efe1009bf20 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe100a10d0 con 0x7efe1009bb40 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe1009d940 con 0x7efe101397f0 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient(hunting): authenticate will time out at 2197.474022s 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe16d26640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7efe101397f0 0x7efe1013bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39456/0 (socket says 172.21.15.6:39456) 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe16d26640 1 -- 172.21.15.6:0/2990348124 learned_addr learned my addr 172.21.15.6:0/2990348124 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2160905835 0 0) 0x7efe1009d940 con 0x7efe101397f0 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient(hunting): my global_id is 164429 2024-09-16T05:55:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdfc003530 con 0x7efe101397f0 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2536739109 0 0) 0x7efe100a1350 con 0x7efe1009bf20 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient(hunting): my global_id is 194103 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe1009d940 con 0x7efe1009bf20 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2920397622 0 0) 0x7efe1009d940 con 0x7efe1009bf20 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 >> v1:172.21.15.6:6790/0 conn(0x7efe101397f0 legacy=0x7efe1013bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 >> v1:172.21.15.63:6789/0 conn(0x7efe1009bb40 legacy=0x7efe1009e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: _finish_hunting 0 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 monclient: found mon.noname-a 2024-09-16T05:55:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1009dc20 con 0x7efe1009bf20 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: _finish_auth 0 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: _check_auth_tickets 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 5 monclient: authenticate success, global_id 194103 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe00001580 con 0x7efe1009bf20 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: dump: 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe00001870 con 0x7efe1009bf20 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 1 -- 172.21.15.6:0/2990348124 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe00001c80 con 0x7efe1009bf20 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe15523640 10 monclient: dump: 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:35.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 4 set_mon_vals no callback set 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 10 monclient: shutdown 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- 172.21.15.6:0/2990348124 >> v1:172.21.15.6:6789/0 conn(0x7efe1009bf20 legacy=0x7efe1009c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- 172.21.15.6:0/2990348124 shutdown_connections 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- 172.21.15.6:0/2990348124 >> 172.21.15.6:0/2990348124 conn(0x7efe100a1650 msgr2=0x7efe100a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- 172.21.15.6:0/2990348124 shutdown_connections 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.663+0000 7efe18789640 1 -- 172.21.15.6:0/2990348124 wait complete. 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: build_initial_monmap 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: monmap: 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 1 Processor -- start 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 1 -- start start 2024-09-16T05:55:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: init 2024-09-16T05:55:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient(hunting): picked mon.noname-c con 0x7efe1009bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient(hunting): picked mon.noname-b con 0x7efe1009bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient(hunting): picked mon.noname-a con 0x7efe101397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe100740c0 con 0x7efe101397f0 2024-09-16T05:55:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe1011c570 con 0x7efe1009bf20 2024-09-16T05:55:35.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe10072ab0 con 0x7efe1009bb40 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient(hunting): authenticate will time out at 2197.477602s 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe16525640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7efe1009bb40 0x7efe101c9760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39470/0 (socket says 172.21.15.6:39470) 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe16d26640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efe101397f0 0x7efe101d1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39166/0 (socket says 172.21.15.6:39166) 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe16525640 1 -- 172.21.15.6:0/904296606 learned_addr learned my addr 172.21.15.6:0/904296606 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3807095504 0 0) 0x7efe08001600 con 0x7efe101397f0 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): my global_id is 194106 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdf4003560 con 0x7efe101397f0 2024-09-16T05:55:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3595731107 0 0) 0x7efe10072ab0 con 0x7efe1009bb40 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): my global_id is 164432 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe08001600 con 0x7efe1009bb40 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1050699438 0 0) 0x7efdf4003560 con 0x7efe101397f0 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe10072ab0 con 0x7efe101397f0 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe08001e00 con 0x7efe101397f0 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): dump: 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.685 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3648891135 0 0) 0x7efe08001600 con 0x7efe1009bb40 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdf4003560 con 0x7efe1009bb40 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe0c0018f0 con 0x7efe1009bb40 2024-09-16T05:55:35.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): dump: 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059958379 0 0) 0x7efe1011c570 con 0x7efe1009bf20 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient(hunting): my global_id is 184207 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe08001600 con 0x7efe1009bf20 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 366687606 0 0) 0x7efe10072ab0 con 0x7efe101397f0 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 >> v1:172.21.15.6:6790/0 conn(0x7efe1009bb40 legacy=0x7efe101c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 >> v1:172.21.15.63:6789/0 conn(0x7efe1009bf20 legacy=0x7efe101ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:55:35.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 monclient: found mon.a 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe100a3c50 con 0x7efe101397f0 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient: _finish_auth 0 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 5 monclient: authenticate success, global_id 194106 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: _renew_subs 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe1013c9b0 con 0x7efe101397f0 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: _renew_subs 2024-09-16T05:55:35.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efe1011c290 con 0x7efe101397f0 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe08004e20 con 0x7efe101397f0 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe08005170 con 0x7efe101397f0 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe14d22640 4 set_mon_vals no callback set 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 10 monclient: dump: 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.690 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efdf80065d0 con 0x7efdf8002870 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7efe0800ff10 con 0x7efe101397f0 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.667+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7efe08045360 con 0x7efe101397f0 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1039078603 0 0) 0x7efdf80065d0 con 0x7efdf8002870 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient(hunting): my global_id is 194109 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe10072ab0 con 0x7efdf8002870 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2963778810 0 0) 0x7efe10072ab0 con 0x7efdf8002870 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdf80065d0 con 0x7efdf8002870 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe0c0037c0 con 0x7efdf8002870 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient: dump: 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.692 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe15d24640 10 monclient: get_auth_request con 0x7efdf4036c20 auth_method 0 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1947775781 0 0) 0x7efdf80065d0 con 0x7efdf8002870 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efdf800a160 con 0x7efdf8002870 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efe0c0057b0 con 0x7efdf8002870 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:35.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 >> v1:172.21.15.6:6789/0 conn(0x7efdf8002870 legacy=0x7efdf8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.671+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 >> v1:172.21.15.6:6789/0 conn(0x7efdf8002870 legacy=0x7efdf8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe18789640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efdf8009e10 con 0x7efdf80090f0 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 910795467 0 0) 0x7efdf8009e10 con 0x7efdf80090f0 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient(hunting): my global_id is 194112 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdf80065d0 con 0x7efdf80090f0 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3094033663 0 0) 0x7efdf80065d0 con 0x7efdf80090f0 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdf8009e10 con 0x7efdf80090f0 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7efe0c003cf0 con 0x7efdf80090f0 2024-09-16T05:55:35.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient: dump: 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1010902672 0 0) 0x7efdf8009e10 con 0x7efdf80090f0 2024-09-16T05:55:35.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efdf800c390 con 0x7efdf80090f0 2024-09-16T05:55:35.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3963 (unknown 0 0 3668092425) 0x7efe0c005960 con 0x7efdf80090f0 2024-09-16T05:55:35.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:55:35.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:55:35.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 >> v1:172.21.15.6:6789/0 conn(0x7efdf80090f0 legacy=0x7efdf8009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.675+0000 7efe06ffd640 1 -- 172.21.15.6:0/904296606 >> v1:172.21.15.6:6789/0 conn(0x7efdf80090f0 legacy=0x7efdf8009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:55:35.696 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:35.696 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:55:35.696 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 90, 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 16559, 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:55:35.697 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:55:35.698 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.699 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:55:35.700 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:35.701 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:35.702 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:55:35.703 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:35.704 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.705 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:55:35.706 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.707 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.708 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:55:35.708 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:35.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 >> v1:172.21.15.63:6812/3643207211 conn(0x7efdf4036c20 legacy=0x7efdf40390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe18789640 10 monclient: shutdown 2024-09-16T05:55:35.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe18789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:35.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 >> v1:172.21.15.6:6789/0 conn(0x7efe101397f0 legacy=0x7efe101d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe16d26640 1 -- 172.21.15.6:0/904296606 reap_dead start 2024-09-16T05:55:35.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe16d26640 1 -- 172.21.15.6:0/904296606 reap_dead start 2024-09-16T05:55:35.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 shutdown_connections 2024-09-16T05:55:35.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 >> 172.21.15.6:0/904296606 conn(0x7efe100a1650 msgr2=0x7efe10138ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:35.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 shutdown_connections 2024-09-16T05:55:35.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.679+0000 7efe18789640 1 -- 172.21.15.6:0/904296606 wait complete. 2024-09-16T05:55:35.710 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:55:35.710 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T05:55:35.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient: get_monmap_and_config 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient: build_initial_monmap 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient: monmap: 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:35.827554+0000 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:35.827554+0000 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:35.827 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 1 Processor -- start 2024-09-16T05:55:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 1 -- start start 2024-09-16T05:55:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient: init 2024-09-16T05:55:35.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ec09bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ec09bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ec139780 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec09d8d0 con 0x7fe2ec139780 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec0a1010 con 0x7fe2ec09bb40 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec0a07c0 con 0x7fe2ec09bf20 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 10 monclient(hunting): authenticate will time out at 2197.634924s 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f0faa640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe2ec139780 0x7fe2ec13bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39186/0 (socket says 172.21.15.6:39186) 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2e3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe2ec09bf20 0x7fe2ec09c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39474/0 (socket says 172.21.15.6:39474) 2024-09-16T05:55:35.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f0faa640 1 -- 172.21.15.6:0/1545521392 learned_addr learned my addr 172.21.15.6:0/1545521392 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1407332483 0 0) 0x7fe2ec09d8d0 con 0x7fe2ec139780 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): my global_id is 194115 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d8003530 con 0x7fe2ec139780 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582218138 0 0) 0x7fe2ec0a07c0 con 0x7fe2ec09bf20 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): my global_id is 164435 2024-09-16T05:55:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec09d8d0 con 0x7fe2ec09bf20 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3774198548 0 0) 0x7fe2ec0a1010 con 0x7fe2ec09bb40 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient(hunting): my global_id is 184210 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec0a07c0 con 0x7fe2ec09bb40 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3911989304 0 0) 0x7fe2d8003530 con 0x7fe2ec139780 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 >> v1:172.21.15.6:6790/0 conn(0x7fe2ec09bf20 legacy=0x7fe2ec09c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 >> v1:172.21.15.63:6789/0 conn(0x7fe2ec09bb40 legacy=0x7fe2ec09e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:35.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ec1402c0 con 0x7fe2ec139780 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 5 monclient: authenticate success, global_id 194115 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2e4001920 con 0x7fe2ec139780 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2eb7fe640 10 monclient: dump: 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.833 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.823+0000 7fe2f2a0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2e4001b80 con 0x7fe2ec139780 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2eb7fe640 1 -- 172.21.15.6:0/1545521392 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2e4001f10 con 0x7fe2ec139780 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 4 set_mon_vals no callback set 2024-09-16T05:55:35.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2eb7fe640 10 monclient: dump: 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: shutdown 2024-09-16T05:55:35.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:35.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/1545521392 >> v1:172.21.15.6:6789/0 conn(0x7fe2ec139780 legacy=0x7fe2ec13bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/1545521392 shutdown_connections 2024-09-16T05:55:35.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/1545521392 >> 172.21.15.6:0/1545521392 conn(0x7fe2ec0a1650 msgr2=0x7fe2ec0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:35.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/1545521392 shutdown_connections 2024-09-16T05:55:35.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/1545521392 wait complete. 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: build_initial_monmap 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: monmap: 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 Processor -- start 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- start start 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: init 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ec09bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ec09bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ec139780 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec133d90 con 0x7fe2ec09bf20 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec0740c0 con 0x7fe2ec139780 2024-09-16T05:55:35.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec11bd10 con 0x7fe2ec09bb40 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient(hunting): authenticate will time out at 2197.638122s 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e3fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe2ec09bf20 0x7fe2ec1d0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39194/0 (socket says 172.21.15.6:39194) 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e3fff640 1 -- 172.21.15.6:0/3304133508 learned_addr learned my addr 172.21.15.6:0/3304133508 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786035293 0 0) 0x7fe2ec133d90 con 0x7fe2ec09bf20 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): my global_id is 194118 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2cc003560 con 0x7fe2ec09bf20 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 17693972 0 0) 0x7fe2ec11bd10 con 0x7fe2ec09bb40 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): my global_id is 164438 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec133d90 con 0x7fe2ec09bb40 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3226883094 0 0) 0x7fe2cc003560 con 0x7fe2ec09bf20 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2ec11bd10 con 0x7fe2ec09bf20 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2d4001a20 con 0x7fe2ec09bf20 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): dump: 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3440161038 0 0) 0x7fe2ec133d90 con 0x7fe2ec09bb40 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2cc003560 con 0x7fe2ec09bb40 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2dc0018c0 con 0x7fe2ec09bb40 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): dump: 2024-09-16T05:55:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009966360 0 0) 0x7fe2ec1402c0 con 0x7fe2ec139780 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient(hunting): my global_id is 184213 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec133d90 con 0x7fe2ec139780 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3613906745 0 0) 0x7fe2ec11bd10 con 0x7fe2ec09bf20 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 >> v1:172.21.15.6:6790/0 conn(0x7fe2ec09bb40 legacy=0x7fe2ec133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 >> v1:172.21.15.63:6789/0 conn(0x7fe2ec139780 legacy=0x7fe2ec1d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient: _finish_hunting 0 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 monclient: found mon.a 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ec1cc1f0 con 0x7fe2ec09bf20 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient: _finish_auth 0 2024-09-16T05:55:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 5 monclient: authenticate success, global_id 194118 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: _renew_subs 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2ec0a2950 con 0x7fe2ec09bf20 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: _renew_subs 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2ec11ba30 con 0x7fe2ec09bf20 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2d4001cc0 con 0x7fe2ec09bf20 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2d4004c40 con 0x7fe2ec09bf20 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2ea7fc640 4 set_mon_vals no callback set 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2e97fa640 10 monclient: dump: 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.827+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe2d00065d0 con 0x7fe2d0002870 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe2d400f9e0 con 0x7fe2ec09bf20 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fe2d4044fe0 con 0x7fe2ec09bf20 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 239084079 0 0) 0x7fe2d00065d0 con 0x7fe2d0002870 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient(hunting): my global_id is 164441 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec11bd10 con 0x7fe2d0002870 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2f0faa640 10 monclient: get_auth_request con 0x7fe2cc036b40 auth_method 0 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3143782572 0 0) 0x7fe2ec11bd10 con 0x7fe2d0002870 2024-09-16T05:55:35.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2d00065d0 con 0x7fe2d0002870 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2dc003620 con 0x7fe2d0002870 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient: dump: 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.853 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1636355947 0 0) 0x7fe2d00065d0 con 0x7fe2d0002870 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2d000a160 con 0x7fe2d0002870 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe2dc004650 con 0x7fe2d0002870 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 >> v1:172.21.15.6:6790/0 conn(0x7fe2d0002870 legacy=0x7fe2d0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.831+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 >> v1:172.21.15.6:6790/0 conn(0x7fe2d0002870 legacy=0x7fe2d0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.835+0000 7fe2f2a0d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.835+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe2d0009e50 con 0x7fe2d0009110 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.835+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864865953 0 0) 0x7fe2d0009e50 con 0x7fe2d0009110 2024-09-16T05:55:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.835+0000 7fe2e97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.835+0000 7fe2e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.835+0000 7fe2e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.835+0000 7fe2e97fa640 10 monclient(hunting): my global_id is 164444 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.835+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d00065d0 con 0x7fe2d0009110 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1763241863 0 0) 0x7fe2d00065d0 con 0x7fe2d0009110 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2d0009e50 con 0x7fe2d0009110 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe2dc003d90 con 0x7fe2d0009110 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 10 monclient: dump: 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2322246566 0 0) 0x7fe2d0009e50 con 0x7fe2d0009110 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe2d000c3e0 con 0x7fe2d0009110 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 2562229337) 0x7fe2dc003fc0 con 0x7fe2d0009110 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:55:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 >> v1:172.21.15.6:6790/0 conn(0x7fe2d0009110 legacy=0x7fe2d00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2e97fa640 1 -- 172.21.15.6:0/3304133508 >> v1:172.21.15.6:6790/0 conn(0x7fe2d0009110 legacy=0x7fe2d00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 90, 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 346799, 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:55:35.857 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:35.858 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:35.859 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:55:35.860 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:55:35.861 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.862 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.863 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:35.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:55:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:55:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:55:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:55:35.869 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe2cc036b40 legacy=0x7fe2cc039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f2a0d640 10 monclient: shutdown 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f2a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 >> v1:172.21.15.6:6789/0 conn(0x7fe2ec09bf20 legacy=0x7fe2ec1d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f0faa640 1 -- 172.21.15.6:0/3304133508 reap_dead start 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f0faa640 1 -- 172.21.15.6:0/3304133508 reap_dead start 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 shutdown_connections 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 >> 172.21.15.6:0/3304133508 conn(0x7fe2ec0a1650 msgr2=0x7fe2ec138d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 shutdown_connections 2024-09-16T05:55:35.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.839+0000 7fe2f2a0d640 1 -- 172.21.15.6:0/3304133508 wait complete. 2024-09-16T05:55:35.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:55:35.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T05:55:35.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T05:55:35.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient: get_monmap_and_config 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient: build_initial_monmap 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient: monmap: 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:35.988286+0000 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:35.988286+0000 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:35.987 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 1 Processor -- start 2024-09-16T05:55:35.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 1 -- start start 2024-09-16T05:55:35.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient: init 2024-09-16T05:55:35.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:35.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:35.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c409bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:35.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c409bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c41397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c40a1350 con 0x7ff6c409bf20 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c40a10d0 con 0x7ff6c41397f0 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c409d940 con 0x7ff6c409bb40 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient(hunting): authenticate will time out at 2197.794864s 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c2ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff6c409bb40 0x7ff6c409e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39490/0 (socket says 172.21.15.6:39490) 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c2ffd640 1 -- 172.21.15.6:0/750408092 learned_addr learned my addr 172.21.15.6:0/750408092 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c27fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff6c409bf20 0x7ff6c409c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39208/0 (socket says 172.21.15.6:39208) 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3792175777 0 0) 0x7ff6c409d940 con 0x7ff6c409bb40 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient(hunting): my global_id is 164447 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6ac003530 con 0x7ff6c409bb40 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 494621677 0 0) 0x7ff6c40a1350 con 0x7ff6c409bf20 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient(hunting): my global_id is 194121 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c409d940 con 0x7ff6c409bf20 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2498999971 0 0) 0x7ff6c409d940 con 0x7ff6c409bf20 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 >> v1:172.21.15.6:6790/0 conn(0x7ff6c409bb40 legacy=0x7ff6c409e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 >> v1:172.21.15.63:6789/0 conn(0x7ff6c41397f0 legacy=0x7ff6c413bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 monclient: found mon.noname-a 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c409dc20 con 0x7ff6c409bf20 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: _finish_auth 0 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 5 monclient: authenticate success, global_id 194121 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff6b8001620 con 0x7ff6c409bf20 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:35.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: dump: 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff6b8001810 con 0x7ff6c409bf20 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 1 -- 172.21.15.6:0/750408092 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff6b8001c20 con 0x7ff6c409bf20 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c9994640 4 set_mon_vals no callback set 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.983+0000 7ff6c1ffb640 10 monclient: dump: 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.000 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: shutdown 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- 172.21.15.6:0/750408092 >> v1:172.21.15.6:6789/0 conn(0x7ff6c409bf20 legacy=0x7ff6c409c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- 172.21.15.6:0/750408092 shutdown_connections 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- 172.21.15.6:0/750408092 >> 172.21.15.6:0/750408092 conn(0x7ff6c40a1650 msgr2=0x7ff6c40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- 172.21.15.6:0/750408092 shutdown_connections 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- 172.21.15.6:0/750408092 wait complete. 2024-09-16T05:55:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: build_initial_monmap 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: monmap: 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 Processor -- start 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- start start 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: init 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c409bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c409bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c41397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c411d4b0 con 0x7ff6c41397f0 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c40a3e20 con 0x7ff6c409bb40 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c411bfb0 con 0x7ff6c409bf20 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient(hunting): authenticate will time out at 2197.798006s 2024-09-16T05:55:36.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c27fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff6c409bf20 0x7ff6c41d0360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39494/0 (socket says 172.21.15.6:39494) 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c27fc640 1 -- 172.21.15.6:0/2910685440 learned_addr learned my addr 172.21.15.6:0/2910685440 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4102343747 0 0) 0x7ff6c411bfb0 con 0x7ff6c409bf20 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): my global_id is 164450 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6b0003530 con 0x7ff6c409bf20 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1011878883 0 0) 0x7ff6c411d4b0 con 0x7ff6c41397f0 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): my global_id is 194124 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c411bfb0 con 0x7ff6c41397f0 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1095352814 0 0) 0x7ff6b0003530 con 0x7ff6c409bf20 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6c411d4b0 con 0x7ff6c409bf20 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff6b8001c60 con 0x7ff6c409bf20 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): dump: 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.005 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 570359636 0 0) 0x7ff6c411bfb0 con 0x7ff6c41397f0 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6b0003530 con 0x7ff6c41397f0 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff6bc001b40 con 0x7ff6c41397f0 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): dump: 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2885715694 0 0) 0x7ff6c40a3e20 con 0x7ff6c409bb40 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient(hunting): my global_id is 184219 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c411bfb0 con 0x7ff6c409bb40 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1936371956 0 0) 0x7ff6c411d4b0 con 0x7ff6c409bf20 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 >> v1:172.21.15.63:6789/0 conn(0x7ff6c409bb40 legacy=0x7ff6c412eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 >> v1:172.21.15.6:6789/0 conn(0x7ff6c41397f0 legacy=0x7ff6c41d3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient: _finish_hunting 0 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 monclient: found mon.c 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c40a3a60 con 0x7ff6c409bf20 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient: _finish_auth 0 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient: _check_auth_tickets 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:36.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 5 monclient: authenticate success, global_id 164450 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: _renew_subs 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c40740c0 con 0x7ff6c409bf20 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: _renew_subs 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6c411d1d0 con 0x7ff6c409bf20 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff6b8004bc0 con 0x7ff6c409bf20 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff6b8004f10 con 0x7ff6c409bf20 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:36.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c0ff9640 4 set_mon_vals no callback set 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 10 monclient: dump: 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:36.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7ff6b80050b0 con 0x7ff6c409bf20 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff6a40065d0 con 0x7ff6a4002870 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.987+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7ff6b8044e80 con 0x7ff6c409bf20 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6c37fe640 10 monclient: get_auth_request con 0x7ff6b0036c90 auth_method 0 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510786532 0 0) 0x7ff6a40065d0 con 0x7ff6a4002870 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient(hunting): my global_id is 184222 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c411d4b0 con 0x7ff6a4002870 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1967738943 0 0) 0x7ff6c411d4b0 con 0x7ff6a4002870 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6a40065d0 con 0x7ff6a4002870 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff6b4001250 con 0x7ff6a4002870 2024-09-16T05:55:36.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient: dump: 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 867307021 0 0) 0x7ff6a40065d0 con 0x7ff6a4002870 2024-09-16T05:55:36.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6a400a160 con 0x7ff6a4002870 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff6b4001210 con 0x7ff6a4002870 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 >> v1:172.21.15.63:6789/0 conn(0x7ff6a4002870 legacy=0x7ff6a4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.991+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 >> v1:172.21.15.63:6789/0 conn(0x7ff6a4002870 legacy=0x7ff6a4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff6a4009e50 con 0x7ff6a4009110 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4222613858 0 0) 0x7ff6a4009e50 con 0x7ff6a4009110 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient(hunting): my global_id is 184225 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6a40065d0 con 0x7ff6a4009110 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3569623915 0 0) 0x7ff6a40065d0 con 0x7ff6a4009110 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6a4009e50 con 0x7ff6a4009110 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7ff6b4004990 con 0x7ff6a4009110 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient: dump: 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.015 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1097265643 0 0) 0x7ff6a4009e50 con 0x7ff6a4009110 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 --> v1:172.21.15.63:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6a400c3e0 con 0x7ff6a4009110 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 <== mon.1 v1:172.21.15.63:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3962 (unknown 0 0 288609379) 0x7ff6b4004990 con 0x7ff6a4009110 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 >> v1:172.21.15.63:6789/0 conn(0x7ff6a4009110 legacy=0x7ff6a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6abfff640 1 -- 172.21.15.6:0/2910685440 >> v1:172.21.15.63:6789/0 conn(0x7ff6a4009110 legacy=0x7ff6a40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 90, 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 130698, 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-09-16T05:55:36.017 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:36.018 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:36.019 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:36.020 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "22154726-d7f4-4d5e-b601-06ee1f3e7219", 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-16T05:34:04.807479Z", 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-09-16T05:55:36.021 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 1, 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-09-16T05:55:36.022 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:36.023 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:36.024 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789", 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.025 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.63:6789/0", 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.63:6789/0", 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-16T05:55:36.026 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.027 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "squid", 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-09-16T05:55:36.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-16T05:55:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-16T05:55:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-16T05:55:36.029 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-09-16T05:55:36.029 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 >> v1:172.21.15.63:6812/3643207211 conn(0x7ff6b0036c90 legacy=0x7ff6b0039150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 10 monclient: shutdown 2024-09-16T05:55:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 >> v1:172.21.15.6:6790/0 conn(0x7ff6c409bf20 legacy=0x7ff6c41d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c37fe640 1 -- 172.21.15.6:0/2910685440 reap_dead start 2024-09-16T05:55:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c37fe640 1 -- 172.21.15.6:0/2910685440 reap_dead start 2024-09-16T05:55:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 shutdown_connections 2024-09-16T05:55:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 >> 172.21.15.6:0/2910685440 conn(0x7ff6c40a1650 msgr2=0x7ff6c412f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 shutdown_connections 2024-09-16T05:55:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:35.999+0000 7ff6c9994640 1 -- 172.21.15.6:0/2910685440 wait complete. 2024-09-16T05:55:36.031 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T05:55:36.031 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T05:55:36.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T05:55:36.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.139+0000 7f2a5d40b640 10 monclient: get_monmap_and_config 2024-09-16T05:55:36.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.139+0000 7f2a5d40b640 10 monclient: build_initial_monmap 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.139+0000 7f2a5d40b640 10 monclient: monmap: 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:36.146205+0000 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:36.146205+0000 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:36.145 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 Processor -- start 2024-09-16T05:55:36.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- start start 2024-09-16T05:55:36.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient: init 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5809bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient(hunting): picked mon.noname-a con 0x7f2a581397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a580a1350 con 0x7f2a581397f0 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a580a10d0 con 0x7f2a5809bf20 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5809d940 con 0x7f2a5809bb40 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient(hunting): authenticate will time out at 2197.953029s 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a577fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2a581397f0 0x7f2a5813bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39214/0 (socket says 172.21.15.6:39214) 2024-09-16T05:55:36.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a577fe640 1 -- 172.21.15.6:0/331968269 learned_addr learned my addr 172.21.15.6:0/331968269 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4290031373 0 0) 0x7f2a580a1350 con 0x7f2a581397f0 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient(hunting): my global_id is 194127 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a44003530 con 0x7f2a581397f0 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2660571665 0 0) 0x7f2a5809d940 con 0x7f2a5809bb40 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient(hunting): my global_id is 164453 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a580a1350 con 0x7f2a5809bb40 2024-09-16T05:55:36.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2800784493 0 0) 0x7f2a44003530 con 0x7f2a581397f0 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 >> v1:172.21.15.6:6790/0 conn(0x7f2a5809bb40 legacy=0x7f2a5809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 >> v1:172.21.15.63:6789/0 conn(0x7f2a5809bf20 legacy=0x7f2a5809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 monclient: found mon.noname-a 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5809dc20 con 0x7f2a581397f0 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: _finish_auth 0 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a500019a0 con 0x7f2a581397f0 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 5 monclient: authenticate success, global_id 194127 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: dump: 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a50001c00 con 0x7f2a581397f0 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 1 -- 172.21.15.6:0/331968269 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a50001f50 con 0x7f2a581397f0 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 4 set_mon_vals no callback set 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a55ffb640 10 monclient: dump: 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.153 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient: shutdown 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- 172.21.15.6:0/331968269 >> v1:172.21.15.6:6789/0 conn(0x7f2a581397f0 legacy=0x7f2a5813bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- 172.21.15.6:0/331968269 shutdown_connections 2024-09-16T05:55:36.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- 172.21.15.6:0/331968269 >> 172.21.15.6:0/331968269 conn(0x7f2a580a1650 msgr2=0x7f2a580a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- 172.21.15.6:0/331968269 shutdown_connections 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- 172.21.15.6:0/331968269 wait complete. 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient: build_initial_monmap 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient: monmap: 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:36.155 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 Processor -- start 2024-09-16T05:55:36.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 1 -- start start 2024-09-16T05:55:36.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.143+0000 7f2a5d40b640 10 monclient: init 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient(hunting): picked mon.noname-b con 0x7f2a5809bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient(hunting): picked mon.noname-a con 0x7f2a581397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a580722c0 con 0x7f2a581397f0 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a580a3d70 con 0x7f2a5809bf20 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5813cb40 con 0x7f2a5809bb40 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient(hunting): authenticate will time out at 2197.956240s 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a56ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2a5809bb40 0x7f2a581cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39508/0 (socket says 172.21.15.6:39508) 2024-09-16T05:55:36.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a56ffd640 1 -- 172.21.15.6:0/216181047 learned_addr learned my addr 172.21.15.6:0/216181047 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 280341646 0 0) 0x7f2a5813cb40 con 0x7f2a5809bb40 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): my global_id is 164456 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a38003530 con 0x7f2a5809bb40 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2165435464 0 0) 0x7f2a580722c0 con 0x7f2a581397f0 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): my global_id is 194130 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a5813cb40 con 0x7f2a581397f0 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3228225026 0 0) 0x7f2a38003530 con 0x7f2a5809bb40 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a580722c0 con 0x7f2a5809bb40 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a48001a80 con 0x7f2a5809bb40 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): dump: 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3785565142 0 0) 0x7f2a5813cb40 con 0x7f2a581397f0 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a38003530 con 0x7f2a581397f0 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a500018a0 con 0x7f2a581397f0 2024-09-16T05:55:36.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): dump: 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.164 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 521459477 0 0) 0x7f2a580a3d70 con 0x7f2a5809bf20 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient(hunting): my global_id is 184231 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a5813cb40 con 0x7f2a5809bf20 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1709417181 0 0) 0x7f2a580722c0 con 0x7f2a5809bb40 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 >> v1:172.21.15.63:6789/0 conn(0x7f2a5809bf20 legacy=0x7f2a581d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 >> v1:172.21.15.6:6789/0 conn(0x7f2a581397f0 legacy=0x7f2a581d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 monclient: found mon.c 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a580740c0 con 0x7f2a5809bb40 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 5 monclient: authenticate success, global_id 164456 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient: _renew_subs 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a58071610 con 0x7f2a5809bb40 2024-09-16T05:55:36.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient: _renew_subs 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a58070e10 con 0x7f2a5809bb40 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a48001c20 con 0x7f2a5809bb40 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f2a48003ad0 con 0x7f2a5809bb40 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a557fa640 4 set_mon_vals no callback set 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 10 monclient: dump: 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.167 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f2a48003c70 con 0x7f2a5809bb40 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c005090 con 0x7f2a5809bb40 2024-09-16T05:55:36.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f2a48044e80 con 0x7f2a5809bb40 2024-09-16T05:55:36.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.147+0000 7f2a567fc640 10 monclient: get_auth_request con 0x7f2a38036b80 auth_method 0 2024-09-16T05:55:36.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.151+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f2a480154b0 con 0x7f2a5809bb40 2024-09-16T05:55:36.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.151+0000 7f2a4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:36.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.151+0000 7f2a4f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.375+0000 7f2a5d40b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T05:55:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.375+0000 7f2a5d40b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T05:55:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.375+0000 7f2a5d40b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:36.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.375+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f2a3c005b90 con 0x7f2a5809bb40 2024-09-16T05:55:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.379+0000 7f2a4f7fe640 1 -- 172.21.15.6:0/216181047 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f2a48014db0 con 0x7f2a5809bb40 2024-09-16T05:55:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.379+0000 7f2a4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T05:55:36.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.379+0000 7f2a4f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:55:36.384 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.379+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.384 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.379+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=526236450,config=3541143443,health=2482433086,kv=2361287224,logm=3205929716}) 2024-09-16T05:55:36.384 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.379+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=10,logm=50} crc {auth=499931582,config=3541143443,health=3842293808,kv=2392046976,logm=733352175}) 2024-09-16T05:55:36.384 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.379+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.384 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.379+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=526236450,config=3541143443,health=2482433086,kv=2361287224,logm=3205929716}) 2024-09-16T05:55:36.384 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.379+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=747603256,config=3541143443,health=1744959055,kv=2361287224,logm=1281452020}) 2024-09-16T05:55:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.383+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 >> v1:172.21.15.63:6812/3643207211 conn(0x7f2a38036b80 legacy=0x7f2a38039040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.383+0000 7f2a5d40b640 10 monclient: shutdown 2024-09-16T05:55:36.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.383+0000 7f2a5d40b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.383+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 >> v1:172.21.15.6:6790/0 conn(0x7f2a5809bb40 legacy=0x7f2a581cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.383+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 shutdown_connections 2024-09-16T05:55:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.383+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 >> 172.21.15.6:0/216181047 conn(0x7f2a580a1650 msgr2=0x7f2a5809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.383+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 shutdown_connections 2024-09-16T05:55:36.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:36.383+0000 7f2a5d40b640 1 -- 172.21.15.6:0/216181047 wait complete. 2024-09-16T05:55:36.387 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.387 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425503421}) 2024-09-16T05:55:36.387 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2031173926}) 2024-09-16T05:55:36.387 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.387 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425503421}) 2024-09-16T05:55:36.387 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=442306044}) 2024-09-16T05:55:36.389 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.389 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461402311}) 2024-09-16T05:55:36.389 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032095158}) 2024-09-16T05:55:36.389 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.389 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461402311}) 2024-09-16T05:55:36.389 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.383+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188939614}) 2024-09-16T05:55:36.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.387+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.387+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065345184}) 2024-09-16T05:55:36.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.387+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2117456567}) 2024-09-16T05:55:36.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.387+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.387+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065345184}) 2024-09-16T05:55:36.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.387+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=667172773}) 2024-09-16T05:55:36.393 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.393 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714228169}) 2024-09-16T05:55:36.393 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058579794}) 2024-09-16T05:55:36.393 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714228169}) 2024-09-16T05:55:36.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091609271}) 2024-09-16T05:55:36.395 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.395 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563889887}) 2024-09-16T05:55:36.395 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1039853397}) 2024-09-16T05:55:36.396 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.396 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563889887}) 2024-09-16T05:55:36.396 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.391+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522344249}) 2024-09-16T05:55:36.398 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.399 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1469875709}) 2024-09-16T05:55:36.399 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1687455111}) 2024-09-16T05:55:36.399 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.399 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1469875709}) 2024-09-16T05:55:36.399 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2306471137}) 2024-09-16T05:55:36.399 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.400 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333003410}) 2024-09-16T05:55:36.400 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=592515597}) 2024-09-16T05:55:36.400 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.400 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333003410}) 2024-09-16T05:55:36.400 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.395+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511658134}) 2024-09-16T05:55:36.401 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.401 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995675441}) 2024-09-16T05:55:36.402 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T05:55:36.402 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2795913635}) 2024-09-16T05:55:36.403 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.403 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995675441}) 2024-09-16T05:55:36.403 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2179422392}) 2024-09-16T05:55:36.404 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.404 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1290503254}) 2024-09-16T05:55:36.404 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3707643526}) 2024-09-16T05:55:36.404 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.404 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1290503254}) 2024-09-16T05:55:36.404 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.399+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896463034}) 2024-09-16T05:55:36.406 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.406 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366749287}) 2024-09-16T05:55:36.406 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2417199494}) 2024-09-16T05:55:36.406 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.406 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366749287}) 2024-09-16T05:55:36.406 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036288842}) 2024-09-16T05:55:36.408 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.408 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734060565}) 2024-09-16T05:55:36.408 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768975340}) 2024-09-16T05:55:36.408 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.408 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734060565}) 2024-09-16T05:55:36.408 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.403+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=404745860}) 2024-09-16T05:55:36.410 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.410 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704932467}) 2024-09-16T05:55:36.410 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266207612}) 2024-09-16T05:55:36.410 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.410 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704932467}) 2024-09-16T05:55:36.410 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3221960442}) 2024-09-16T05:55:36.412 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.412 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413855027}) 2024-09-16T05:55:36.412 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2443848128}) 2024-09-16T05:55:36.412 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.412 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413855027}) 2024-09-16T05:55:36.412 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.407+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=764409501}) 2024-09-16T05:55:36.414 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.414 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884014163}) 2024-09-16T05:55:36.414 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4287820558}) 2024-09-16T05:55:36.414 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.414 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884014163}) 2024-09-16T05:55:36.415 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582468374}) 2024-09-16T05:55:36.416 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.416 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954814119}) 2024-09-16T05:55:36.416 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2306489607}) 2024-09-16T05:55:36.416 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.416 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954814119}) 2024-09-16T05:55:36.416 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.411+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3680177792}) 2024-09-16T05:55:36.418 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.418 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571538036}) 2024-09-16T05:55:36.418 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2801580345}) 2024-09-16T05:55:36.418 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.418 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571538036}) 2024-09-16T05:55:36.418 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1936003341}) 2024-09-16T05:55:36.420 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.420 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780258576}) 2024-09-16T05:55:36.420 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3378154010}) 2024-09-16T05:55:36.420 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.420 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780258576}) 2024-09-16T05:55:36.421 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.415+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2054350099}) 2024-09-16T05:55:36.422 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934399287}) 2024-09-16T05:55:36.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181846661}) 2024-09-16T05:55:36.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934399287}) 2024-09-16T05:55:36.423 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2425077140}) 2024-09-16T05:55:36.424 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.425 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4113091413}) 2024-09-16T05:55:36.425 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1309127013}) 2024-09-16T05:55:36.425 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.425 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4113091413}) 2024-09-16T05:55:36.425 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.419+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2328686824}) 2024-09-16T05:55:36.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710371747}) 2024-09-16T05:55:36.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1400457000}) 2024-09-16T05:55:36.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710371747}) 2024-09-16T05:55:36.427 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732270838}) 2024-09-16T05:55:36.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835990337}) 2024-09-16T05:55:36.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1138964349}) 2024-09-16T05:55:36.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835990337}) 2024-09-16T05:55:36.429 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.423+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2098096253}) 2024-09-16T05:55:36.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896381242}) 2024-09-16T05:55:36.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129251857}) 2024-09-16T05:55:36.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896381242}) 2024-09-16T05:55:36.431 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3743548308}) 2024-09-16T05:55:36.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370696825}) 2024-09-16T05:55:36.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3801102393}) 2024-09-16T05:55:36.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370696825}) 2024-09-16T05:55:36.433 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.427+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925639384}) 2024-09-16T05:55:36.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.431+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.431+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283143904}) 2024-09-16T05:55:36.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.431+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=520890462}) 2024-09-16T05:55:36.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.431+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.431+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283143904}) 2024-09-16T05:55:36.435 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.431+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898296110}) 2024-09-16T05:55:36.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.431+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.431+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311443765}) 2024-09-16T05:55:36.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703506480}) 2024-09-16T05:55:36.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311443765}) 2024-09-16T05:55:36.437 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221681867}) 2024-09-16T05:55:36.439 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136573954}) 2024-09-16T05:55:36.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1667478027}) 2024-09-16T05:55:36.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136573954}) 2024-09-16T05:55:36.440 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.435+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909240370}) 2024-09-16T05:55:36.441 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.441 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056665156}) 2024-09-16T05:55:36.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777361407}) 2024-09-16T05:55:36.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056665156}) 2024-09-16T05:55:36.442 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906114505}) 2024-09-16T05:55:36.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3990387051}) 2024-09-16T05:55:36.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3281712806}) 2024-09-16T05:55:36.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3990387051}) 2024-09-16T05:55:36.444 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.439+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563193532}) 2024-09-16T05:55:36.445 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.445 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135436148}) 2024-09-16T05:55:36.445 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181666573}) 2024-09-16T05:55:36.446 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.446 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2135436148}) 2024-09-16T05:55:36.446 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3237405239}) 2024-09-16T05:55:36.447 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.447 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499045723}) 2024-09-16T05:55:36.447 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1418500491}) 2024-09-16T05:55:36.447 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.448 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499045723}) 2024-09-16T05:55:36.448 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.443+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790562929}) 2024-09-16T05:55:36.449 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.449 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285083829}) 2024-09-16T05:55:36.450 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2024063028}) 2024-09-16T05:55:36.450 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.450 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285083829}) 2024-09-16T05:55:36.450 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470370576}) 2024-09-16T05:55:36.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77757200}) 2024-09-16T05:55:36.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1152890389}) 2024-09-16T05:55:36.451 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.452 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77757200}) 2024-09-16T05:55:36.452 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890786987}) 2024-09-16T05:55:36.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.447+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786792775}) 2024-09-16T05:55:36.453 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3274945984}) 2024-09-16T05:55:36.454 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.454 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786792775}) 2024-09-16T05:55:36.454 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098381464}) 2024-09-16T05:55:36.456 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.456 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351303918}) 2024-09-16T05:55:36.456 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854607255}) 2024-09-16T05:55:36.456 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.456 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351303918}) 2024-09-16T05:55:36.456 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1350263735}) 2024-09-16T05:55:36.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.451+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217546263}) 2024-09-16T05:55:36.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1446318934}) 2024-09-16T05:55:36.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217546263}) 2024-09-16T05:55:36.457 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663689974}) 2024-09-16T05:55:36.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984209803}) 2024-09-16T05:55:36.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488168456}) 2024-09-16T05:55:36.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984209803}) 2024-09-16T05:55:36.459 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=159397465}) 2024-09-16T05:55:36.460 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.460 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265158688}) 2024-09-16T05:55:36.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905843222}) 2024-09-16T05:55:36.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265158688}) 2024-09-16T05:55:36.461 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.455+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717588705}) 2024-09-16T05:55:36.462 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.462 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083677134}) 2024-09-16T05:55:36.462 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1901257271}) 2024-09-16T05:55:36.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083677134}) 2024-09-16T05:55:36.463 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1791730502}) 2024-09-16T05:55:36.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158678333}) 2024-09-16T05:55:36.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802277813}) 2024-09-16T05:55:36.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.465 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158678333}) 2024-09-16T05:55:36.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.459+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2975422918}) 2024-09-16T05:55:36.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914943769}) 2024-09-16T05:55:36.466 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=178385032}) 2024-09-16T05:55:36.467 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.467 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914943769}) 2024-09-16T05:55:36.467 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=130277365}) 2024-09-16T05:55:36.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555007874}) 2024-09-16T05:55:36.468 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2602747157}) 2024-09-16T05:55:36.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555007874}) 2024-09-16T05:55:36.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.463+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3863239631}) 2024-09-16T05:55:36.470 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.470 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=575992665}) 2024-09-16T05:55:36.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297235919}) 2024-09-16T05:55:36.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=575992665}) 2024-09-16T05:55:36.471 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2874488549}) 2024-09-16T05:55:36.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284142033}) 2024-09-16T05:55:36.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1747211896}) 2024-09-16T05:55:36.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284142033}) 2024-09-16T05:55:36.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.467+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=786913559}) 2024-09-16T05:55:36.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.471+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.471+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238800410}) 2024-09-16T05:55:36.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.471+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2619164156}) 2024-09-16T05:55:36.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.471+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.471+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238800410}) 2024-09-16T05:55:36.475 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.471+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4049374959}) 2024-09-16T05:55:36.477 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.477 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385162045}) 2024-09-16T05:55:36.477 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036061826}) 2024-09-16T05:55:36.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385162045}) 2024-09-16T05:55:36.478 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657711984}) 2024-09-16T05:55:36.479 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.479 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056839890}) 2024-09-16T05:55:36.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3966315328}) 2024-09-16T05:55:36.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056839890}) 2024-09-16T05:55:36.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.475+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966367604}) 2024-09-16T05:55:36.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67378280}) 2024-09-16T05:55:36.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3870677981}) 2024-09-16T05:55:36.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67378280}) 2024-09-16T05:55:36.482 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1731779940}) 2024-09-16T05:55:36.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223632844}) 2024-09-16T05:55:36.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002083872}) 2024-09-16T05:55:36.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223632844}) 2024-09-16T05:55:36.484 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.479+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381630879}) 2024-09-16T05:55:36.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706380665}) 2024-09-16T05:55:36.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775598504}) 2024-09-16T05:55:36.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706380665}) 2024-09-16T05:55:36.486 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2341548893}) 2024-09-16T05:55:36.488 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3328538729}) 2024-09-16T05:55:36.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987250503}) 2024-09-16T05:55:36.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3328538729}) 2024-09-16T05:55:36.489 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.483+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1395919441}) 2024-09-16T05:55:36.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466300241}) 2024-09-16T05:55:36.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094922448}) 2024-09-16T05:55:36.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466300241}) 2024-09-16T05:55:36.491 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1364635528}) 2024-09-16T05:55:36.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701131238}) 2024-09-16T05:55:36.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3277573760}) 2024-09-16T05:55:36.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701131238}) 2024-09-16T05:55:36.493 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.487+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1536862892}) 2024-09-16T05:55:36.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.491+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.491+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954404795}) 2024-09-16T05:55:36.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.491+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3118038540}) 2024-09-16T05:55:36.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.491+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.491+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954404795}) 2024-09-16T05:55:36.496 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.491+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2200946545}) 2024-09-16T05:55:36.497 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747721283}) 2024-09-16T05:55:36.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1402567916}) 2024-09-16T05:55:36.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747721283}) 2024-09-16T05:55:36.498 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276827977}) 2024-09-16T05:55:36.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544392809}) 2024-09-16T05:55:36.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421122288}) 2024-09-16T05:55:36.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544392809}) 2024-09-16T05:55:36.500 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.495+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2234374320}) 2024-09-16T05:55:36.501 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135723670}) 2024-09-16T05:55:36.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745022210}) 2024-09-16T05:55:36.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135723670}) 2024-09-16T05:55:36.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3835781028}) 2024-09-16T05:55:36.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.503 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056423600}) 2024-09-16T05:55:36.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3946686789}) 2024-09-16T05:55:36.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056423600}) 2024-09-16T05:55:36.504 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.499+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=781886207}) 2024-09-16T05:55:36.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303416331}) 2024-09-16T05:55:36.505 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4008845681}) 2024-09-16T05:55:36.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303416331}) 2024-09-16T05:55:36.506 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=891388377}) 2024-09-16T05:55:36.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1072567562}) 2024-09-16T05:55:36.507 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018172372}) 2024-09-16T05:55:36.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1072567562}) 2024-09-16T05:55:36.508 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.503+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2759140543}) 2024-09-16T05:55:36.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548861843}) 2024-09-16T05:55:36.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1644710939}) 2024-09-16T05:55:36.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.509 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548861843}) 2024-09-16T05:55:36.510 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954998815}) 2024-09-16T05:55:36.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825961801}) 2024-09-16T05:55:36.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2677943019}) 2024-09-16T05:55:36.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.511 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825961801}) 2024-09-16T05:55:36.512 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.507+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1679540340}) 2024-09-16T05:55:36.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3452228736}) 2024-09-16T05:55:36.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1674347851}) 2024-09-16T05:55:36.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3452228736}) 2024-09-16T05:55:36.513 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2361880644}) 2024-09-16T05:55:36.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065465684}) 2024-09-16T05:55:36.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1265669841}) 2024-09-16T05:55:36.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.515 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065465684}) 2024-09-16T05:55:36.516 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.511+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3436666534}) 2024-09-16T05:55:36.517 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.517 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970631435}) 2024-09-16T05:55:36.518 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437826731}) 2024-09-16T05:55:36.518 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.518 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970631435}) 2024-09-16T05:55:36.518 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1376134708}) 2024-09-16T05:55:36.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839738519}) 2024-09-16T05:55:36.519 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=442826863}) 2024-09-16T05:55:36.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839738519}) 2024-09-16T05:55:36.520 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.515+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527995949}) 2024-09-16T05:55:36.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2699835072}) 2024-09-16T05:55:36.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1040742278}) 2024-09-16T05:55:36.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2699835072}) 2024-09-16T05:55:36.522 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318536838}) 2024-09-16T05:55:36.523 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737105748}) 2024-09-16T05:55:36.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1347440233}) 2024-09-16T05:55:36.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737105748}) 2024-09-16T05:55:36.524 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.519+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136634705}) 2024-09-16T05:55:36.525 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.523+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.523+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3235901048,mdsmap=3035854586}) 2024-09-16T05:55:36.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.523+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2993851472}) 2024-09-16T05:55:36.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.523+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.523+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3235901048,mdsmap=3035854586}) 2024-09-16T05:55:36.526 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.523+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4081456055}) 2024-09-16T05:55:36.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.531+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.531+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=13,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mdsmap=3140514009,mgr=2296374319,mgr_metadata=2545138219,mgrstat=3131387884,mon_config_key=3702425230,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1248129561}) 2024-09-16T05:55:36.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.531+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=49} crc {mdsmap=3140514009,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2369008096,mon_config_key=1416732161,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1503630732}) 2024-09-16T05:55:36.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.531+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.531+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=13,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mdsmap=3140514009,mgr=2296374319,mgr_metadata=2545138219,mgrstat=3131387884,mon_config_key=3702425230,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=1248129561}) 2024-09-16T05:55:36.536 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.531+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=12,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=51} crc {mdsmap=3140514009,mgr=2296374319,mgr_command_descs=2107673600,mgr_metadata=2545138219,mgrstat=2202297157,mon_config_key=3625351901,monmap=3625042558,osd_metadata=1319166417,osd_pg_creating=1754493876,osd_snap=723727393,osdmap=3208755680}) 2024-09-16T05:55:36.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.539+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T05:55:36.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.539+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=3} crc {osdmap=2889745764}) 2024-09-16T05:55:36.542 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-16T05:55:36.539+0000 7fdc2952b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=2} crc {osdmap=930036694}) 2024-09-16T05:55:37.404 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T05:55:37.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T05:55:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient: get_monmap_and_config 2024-09-16T05:55:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient: build_initial_monmap 2024-09-16T05:55:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient: monmap: 2024-09-16T05:55:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:37.532635+0000 2024-09-16T05:55:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:37.532635+0000 2024-09-16T05:55:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:37.532 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 1 Processor -- start 2024-09-16T05:55:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 1 -- start start 2024-09-16T05:55:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient: init 2024-09-16T05:55:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:55:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient(hunting): picked mon.noname-c con 0x7f335c09aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient(hunting): picked mon.noname-a con 0x7f335c09e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient(hunting): picked mon.noname-b con 0x7f335c0a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f335c0a1370 con 0x7f335c09e230 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f335c09ead0 con 0x7f335c09aa30 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f335c09edb0 con 0x7f335c0a0da0 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3363afa640 10 monclient(hunting): authenticate will time out at 2199.339397s 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3361095640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f335c09e230 0x7f335c1399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39240/0 (socket says 172.21.15.6:39240) 2024-09-16T05:55:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3361095640 1 -- 172.21.15.6:0/1627057109 learned_addr learned my addr 172.21.15.6:0/1627057109 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431574298 0 0) 0x7f335c0a1370 con 0x7f335c09e230 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 10 monclient(hunting): my global_id is 194133 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3348003530 con 0x7f335c09e230 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2073683921 0 0) 0x7f335c09edb0 con 0x7f335c0a0da0 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.527+0000 7f3360894640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient(hunting): my global_id is 164459 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f335c0a1370 con 0x7f335c0a0da0 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3047164051 0 0) 0x7f335c0a1370 con 0x7f335c0a0da0 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 >> v1:172.21.15.63:6789/0 conn(0x7f335c09aa30 legacy=0x7f335c09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 >> v1:172.21.15.6:6789/0 conn(0x7f335c09e230 legacy=0x7f335c1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: _finish_hunting 0 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 monclient: found mon.noname-b 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c09dd50 con 0x7f335c0a0da0 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: _finish_auth 0 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: _check_auth_tickets 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3354001620 con 0x7f335c0a0da0 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 5 monclient: authenticate success, global_id 164459 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: dump: 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3354001810 con 0x7f335c0a0da0 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 4 set_mon_vals no callback set 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 1 -- 172.21.15.6:0/1627057109 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3354001ba0 con 0x7f335c0a0da0 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3360894640 10 monclient: dump: 2024-09-16T05:55:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: shutdown 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- 172.21.15.6:0/1627057109 >> v1:172.21.15.6:6790/0 conn(0x7f335c0a0da0 legacy=0x7f335c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- 172.21.15.6:0/1627057109 shutdown_connections 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- 172.21.15.6:0/1627057109 >> 172.21.15.6:0/1627057109 conn(0x7f335c0a1650 msgr2=0x7f335c0a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- 172.21.15.6:0/1627057109 shutdown_connections 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- 172.21.15.6:0/1627057109 wait complete. 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: build_initial_monmap 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: monmap: 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:37.543 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 Processor -- start 2024-09-16T05:55:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- start start 2024-09-16T05:55:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: init 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient(hunting): picked mon.noname-c con 0x7f335c09aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient(hunting): picked mon.noname-a con 0x7f335c09e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient(hunting): picked mon.noname-b con 0x7f335c0a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f335c135cc0 con 0x7f335c09e230 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f335c132810 con 0x7f335c0a0da0 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f335c133a30 con 0x7f335c09aa30 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient(hunting): authenticate will time out at 2199.342626s 2024-09-16T05:55:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3361095640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f335c09e230 0x7f335c132100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39242/0 (socket says 172.21.15.6:39242) 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3361095640 1 -- 172.21.15.6:0/1115856300 learned_addr learned my addr 172.21.15.6:0/1115856300 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934141251 0 0) 0x7f335c135cc0 con 0x7f335c09e230 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): my global_id is 194136 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3340003530 con 0x7f335c09e230 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4206131602 0 0) 0x7f335c133a30 con 0x7f335c09aa30 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): my global_id is 164462 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f335c135cc0 con 0x7f335c09aa30 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3853142597 0 0) 0x7f3340003530 con 0x7f335c09e230 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f335c133a30 con 0x7f335c09e230 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3350001dc0 con 0x7f335c09e230 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): dump: 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500334513 0 0) 0x7f335c132810 con 0x7f335c0a0da0 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): my global_id is 184237 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3340003530 con 0x7f335c0a0da0 2024-09-16T05:55:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3403091622 0 0) 0x7f335c135cc0 con 0x7f335c09aa30 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f335c132810 con 0x7f335c09aa30 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3358001840 con 0x7f335c09aa30 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient(hunting): dump: 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 220781034 0 0) 0x7f335c133a30 con 0x7f335c09e230 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 >> v1:172.21.15.6:6790/0 conn(0x7f335c09aa30 legacy=0x7f335c1319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 >> v1:172.21.15.63:6789/0 conn(0x7f335c0a0da0 legacy=0x7f335c1d1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 monclient: found mon.a 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c1340f0 con 0x7f335c09e230 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient: _finish_auth 0 2024-09-16T05:55:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 5 monclient: authenticate success, global_id 194136 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: _renew_subs 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f335c06e640 con 0x7f335c09e230 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: _renew_subs 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f335c1352c0 con 0x7f335c09e230 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3350001980 con 0x7f335c09e230 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3350004e20 con 0x7f335c09e230 2024-09-16T05:55:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334ffff640 4 set_mon_vals no callback set 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.531+0000 7f334e7fc640 10 monclient: dump: 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.554 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.535+0000 7f3363afa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.535+0000 7f3363afa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.535+0000 7f3363afa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.535+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3344005090 con 0x7f335c09e230 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.535+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f335000fc80 con 0x7f335c09e230 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.535+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f3350045180 con 0x7f335c09e230 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.535+0000 7f3361896640 10 monclient: get_auth_request con 0x7f3340036ba0 auth_method 0 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.539+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f33500158f0 con 0x7f335c09e230 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.539+0000 7f334e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.539+0000 7f334e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.763+0000 7f3363afa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T05:55:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.763+0000 7f3363afa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T05:55:37.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.763+0000 7f3363afa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.763+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3344005380 con 0x7f335c09e230 2024-09-16T05:55:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.763+0000 7f334e7fc640 1 -- 172.21.15.6:0/1115856300 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1397 (unknown 254746657 0 2126128530) 0x7f33500151f0 con 0x7f335c09e230 2024-09-16T05:55:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.763+0000 7f334e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T05:55:37.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.763+0000 7f334e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:55:37.767 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","modified":"2024-09-16T05:34:04.807479Z","created":"2024-09-16T05:34:04.807479Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T05:55:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.767+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3340036ba0 legacy=0x7f3340039060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.767+0000 7f3363afa640 10 monclient: shutdown 2024-09-16T05:55:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.767+0000 7f3363afa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.767+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 >> v1:172.21.15.6:6789/0 conn(0x7f335c09e230 legacy=0x7f335c132100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.767+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 shutdown_connections 2024-09-16T05:55:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.767+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 >> 172.21.15.6:0/1115856300 conn(0x7f335c0a1650 msgr2=0x7f335c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.767+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 shutdown_connections 2024-09-16T05:55:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.767+0000 7f3363afa640 1 -- 172.21.15.6:0/1115856300 wait complete. 2024-09-16T05:55:37.785 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T05:55:37.785 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T05:55:37.795 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T05:55:37.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:55:37.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.903+0000 7f765d305640 10 monclient: get_monmap_and_config 2024-09-16T05:55:37.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.903+0000 7f765d305640 10 monclient: build_initial_monmap 2024-09-16T05:55:37.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.903+0000 7f765d305640 10 monclient: monmap: 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:37.910028+0000 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:37.910028+0000 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:37.909 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.903+0000 7f765d305640 1 Processor -- start 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- start start 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: init 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): picked mon.noname-b con 0x7f765809aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): picked mon.noname-c con 0x7f765809e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): picked mon.noname-a con 0x7f76580a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76580a1370 con 0x7f76580a0da0 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765809ead0 con 0x7f765809e230 2024-09-16T05:55:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765809edb0 con 0x7f765809aa30 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): authenticate will time out at 2199.716773s 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7656ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f765809aa30 0x7f765809ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39524/0 (socket says 172.21.15.6:39524) 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7656ffd640 1 -- 172.21.15.6:0/1154792485 learned_addr learned my addr 172.21.15.6:0/1154792485 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400326442 0 0) 0x7f76580a1370 con 0x7f76580a0da0 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient(hunting): my global_id is 194139 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7640003530 con 0x7f76580a0da0 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420624265 0 0) 0x7f765809edb0 con 0x7f765809aa30 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient(hunting): my global_id is 164465 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76580a1370 con 0x7f765809aa30 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 711683664 0 0) 0x7f76580a1370 con 0x7f765809aa30 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 >> v1:172.21.15.63:6789/0 conn(0x7f765809e230 legacy=0x7f76581399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 >> v1:172.21.15.6:6789/0 conn(0x7f76580a0da0 legacy=0x7f765813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 monclient: found mon.noname-b 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765809dd50 con 0x7f765809aa30 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: _finish_auth 0 2024-09-16T05:55:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 5 monclient: authenticate success, global_id 164465 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7648001620 con 0x7f765809aa30 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: dump: 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7648001890 con 0x7f765809aa30 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 1 -- 172.21.15.6:0/1154792485 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7648001c20 con 0x7f765809aa30 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 4 set_mon_vals no callback set 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7655ffb640 10 monclient: dump: 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: shutdown 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- 172.21.15.6:0/1154792485 >> v1:172.21.15.6:6790/0 conn(0x7f765809aa30 legacy=0x7f765809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- 172.21.15.6:0/1154792485 shutdown_connections 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- 172.21.15.6:0/1154792485 >> 172.21.15.6:0/1154792485 conn(0x7f76580a1650 msgr2=0x7f76580a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- 172.21.15.6:0/1154792485 shutdown_connections 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- 172.21.15.6:0/1154792485 wait complete. 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: build_initial_monmap 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: monmap: 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:37.918 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:37.919 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:37.919 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 Processor -- start 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- start start 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: init 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): picked mon.noname-c con 0x7f765809e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): picked mon.noname-a con 0x7f76580a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): picked mon.noname-b con 0x7f765813c480 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76580740c0 con 0x7f76580a0da0 2024-09-16T05:55:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765813d0d0 con 0x7f765813c480 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765811c850 con 0x7f765809e230 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f765d305640 10 monclient(hunting): authenticate will time out at 2199.719693s 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7656ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f765809e230 0x7f76581c9910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39540/0 (socket says 172.21.15.6:39540) 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.907+0000 7f7656ffd640 1 -- 172.21.15.6:0/302311696 learned_addr learned my addr 172.21.15.6:0/302311696 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2866395431 0 0) 0x7f765811c850 con 0x7f765809e230 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): my global_id is 164468 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7644003530 con 0x7f765809e230 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1479822411 0 0) 0x7f765809ead0 con 0x7f76580a0da0 2024-09-16T05:55:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): my global_id is 194142 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f765811c850 con 0x7f76580a0da0 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1251916454 0 0) 0x7f765811c850 con 0x7f76580a0da0 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f765809ead0 con 0x7f76580a0da0 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2135388116 0 0) 0x7f7644003530 con 0x7f765809e230 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f765811c850 con 0x7f765809e230 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f764c001880 con 0x7f76580a0da0 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): dump: 2024-09-16T05:55:37.924 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f7648001be0 con 0x7f765809e230 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): dump: 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453650748 0 0) 0x7f765813d0d0 con 0x7f765813c480 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:37.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient(hunting): my global_id is 184243 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7644003530 con 0x7f765813c480 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2320753994 0 0) 0x7f765809ead0 con 0x7f76580a0da0 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 >> v1:172.21.15.6:6790/0 conn(0x7f765809e230 legacy=0x7f76581c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 >> v1:172.21.15.63:6789/0 conn(0x7f765813c480 legacy=0x7f76581d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 monclient: found mon.a 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765811cb30 con 0x7f76580a0da0 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient: _finish_auth 0 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:55:37.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 5 monclient: authenticate success, global_id 194142 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 10 monclient: _renew_subs 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f765813d970 con 0x7f76580a0da0 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 10 monclient: _renew_subs 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f76581ca020 con 0x7f76580a0da0 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f764c003620 con 0x7f76580a0da0 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f764c003970 con 0x7f76580a0da0 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:37.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f76557fa640 4 set_mon_vals no callback set 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 10 monclient: dump: 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:37.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7638005090 con 0x7f76580a0da0 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f764c00e710 con 0x7f76580a0da0 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f764c043ae0 con 0x7f76580a0da0 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.911+0000 7f7656ffd640 10 monclient: get_auth_request con 0x7f7644036bb0 auth_method 0 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.915+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f76580740c0 con 0x7f76580a0da0 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.915+0000 7f763f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:37.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:37.915+0000 7f763f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.067+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7638000ba0 con 0x7f7644036bb0 2024-09-16T05:55:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f763f7fe640 1 -- 172.21.15.6:0/302311696 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77961 (unknown 2965378022 0 50223216) 0x7f7638000ba0 con 0x7f7644036bb0 2024-09-16T05:55:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:55:38.079 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:55:38.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 >> v1:172.21.15.63:6812/3643207211 conn(0x7f7644036bb0 legacy=0x7f7644039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f765d305640 10 monclient: shutdown 2024-09-16T05:55:38.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f765d305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:38.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 >> v1:172.21.15.6:6789/0 conn(0x7f76580a0da0 legacy=0x7f76581ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 shutdown_connections 2024-09-16T05:55:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 >> 172.21.15.6:0/302311696 conn(0x7f76580a1650 msgr2=0x7f765806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 shutdown_connections 2024-09-16T05:55:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.075+0000 7f765d305640 1 -- 172.21.15.6:0/302311696 wait complete. 2024-09-16T05:55:38.095 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":880,"stamp":"2024-09-16T05:55:37.626572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205768,"kb_used_data":18640,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566025272,"statfs":{"total":579820584960,"available":579609878528,"internally_reserved":0,"allocated":19087360,"data_stored":15446128,"data_compressed":1794170,"data_compressed_allocated":2719744,"data_compressed_original":5914624,"omap_allocated":9566,"internal_metadata":191486626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003677"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":288,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485848+0000","last_change":"2024-09-16T05:39:26.074706+0000","last_active":"2024-09-16T05:52:56.485848+0000","last_peered":"2024-09-16T05:52:56.485848+0000","last_clean":"2024-09-16T05:52:56.485848+0000","last_became_active":"2024-09-16T05:39:26.073689+0000","last_became_peered":"2024-09-16T05:39:26.073689+0000","last_unstale":"2024-09-16T05:52:56.485848+0000","last_undegraded":"2024-09-16T05:52:56.485848+0000","last_fullsized":"2024-09-16T05:52:56.485848+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:22.219893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:27.376074+0000","last_change":"2024-09-16T05:39:26.053781+0000","last_active":"2024-09-16T05:52:27.376074+0000","last_peered":"2024-09-16T05:52:27.376074+0000","last_clean":"2024-09-16T05:52:27.376074+0000","last_became_active":"2024-09-16T05:39:26.053397+0000","last_became_peered":"2024-09-16T05:39:26.053397+0000","last_unstale":"2024-09-16T05:52:27.376074+0000","last_undegraded":"2024-09-16T05:52:27.376074+0000","last_fullsized":"2024-09-16T05:52:27.376074+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:58.904858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":294,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.480780+0000","last_change":"2024-09-16T05:39:26.070406+0000","last_active":"2024-09-16T05:52:56.480780+0000","last_peered":"2024-09-16T05:52:56.480780+0000","last_clean":"2024-09-16T05:52:56.480780+0000","last_became_active":"2024-09-16T05:39:26.069114+0000","last_became_peered":"2024-09-16T05:39:26.069114+0000","last_unstale":"2024-09-16T05:52:56.480780+0000","last_undegraded":"2024-09-16T05:52:56.480780+0000","last_fullsized":"2024-09-16T05:52:56.480780+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:19:02.873639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:27.376003+0000","last_change":"2024-09-16T05:39:26.056400+0000","last_active":"2024-09-16T05:52:27.376003+0000","last_peered":"2024-09-16T05:52:27.376003+0000","last_clean":"2024-09-16T05:52:27.376003+0000","last_became_active":"2024-09-16T05:39:26.055883+0000","last_became_peered":"2024-09-16T05:39:26.055883+0000","last_unstale":"2024-09-16T05:52:27.376003+0000","last_undegraded":"2024-09-16T05:52:27.376003+0000","last_fullsized":"2024-09-16T05:52:27.376003+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:38.780469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":224,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:22.364898+0000","last_change":"2024-09-16T05:39:27.057197+0000","last_active":"2024-09-16T05:51:22.364898+0000","last_peered":"2024-09-16T05:51:22.364898+0000","last_clean":"2024-09-16T05:51:22.364898+0000","last_became_active":"2024-09-16T05:39:27.056198+0000","last_became_peered":"2024-09-16T05:39:27.056198+0000","last_unstale":"2024-09-16T05:51:22.364898+0000","last_undegraded":"2024-09-16T05:51:22.364898+0000","last_fullsized":"2024-09-16T05:51:22.364898+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:54.298350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":276,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485909+0000","last_change":"2024-09-16T05:39:26.074867+0000","last_active":"2024-09-16T05:52:56.485909+0000","last_peered":"2024-09-16T05:52:56.485909+0000","last_clean":"2024-09-16T05:52:56.485909+0000","last_became_active":"2024-09-16T05:39:26.073067+0000","last_became_peered":"2024-09-16T05:39:26.073067+0000","last_unstale":"2024-09-16T05:52:56.485909+0000","last_undegraded":"2024-09-16T05:52:56.485909+0000","last_fullsized":"2024-09-16T05:52:56.485909+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:24.660359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":176,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.362499+0000","last_change":"2024-09-16T05:39:26.061744+0000","last_active":"2024-09-16T05:52:26.362499+0000","last_peered":"2024-09-16T05:52:26.362499+0000","last_clean":"2024-09-16T05:52:26.362499+0000","last_became_active":"2024-09-16T05:39:26.061165+0000","last_became_peered":"2024-09-16T05:39:26.061165+0000","last_unstale":"2024-09-16T05:52:26.362499+0000","last_undegraded":"2024-09-16T05:52:26.362499+0000","last_fullsized":"2024-09-16T05:52:26.362499+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:26.707122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356374+0000","last_change":"2024-09-16T05:39:26.053923+0000","last_active":"2024-09-16T05:51:56.356374+0000","last_peered":"2024-09-16T05:51:56.356374+0000","last_clean":"2024-09-16T05:51:56.356374+0000","last_became_active":"2024-09-16T05:39:26.053013+0000","last_became_peered":"2024-09-16T05:39:26.053013+0000","last_unstale":"2024-09-16T05:51:56.356374+0000","last_undegraded":"2024-09-16T05:51:56.356374+0000","last_fullsized":"2024-09-16T05:51:56.356374+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:32.161682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":209,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:37.377442+0000","last_change":"2024-09-16T05:39:26.059457+0000","last_active":"2024-09-16T05:52:37.377442+0000","last_peered":"2024-09-16T05:52:37.377442+0000","last_clean":"2024-09-16T05:52:37.377442+0000","last_became_active":"2024-09-16T05:39:26.058997+0000","last_became_peered":"2024-09-16T05:39:26.058997+0000","last_unstale":"2024-09-16T05:52:37.377442+0000","last_undegraded":"2024-09-16T05:52:37.377442+0000","last_fullsized":"2024-09-16T05:52:37.377442+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:25.563929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":213,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:06.368302+0000","last_change":"2024-09-16T05:39:26.049544+0000","last_active":"2024-09-16T05:53:06.368302+0000","last_peered":"2024-09-16T05:53:06.368302+0000","last_clean":"2024-09-16T05:53:06.368302+0000","last_became_active":"2024-09-16T05:39:26.049092+0000","last_became_peered":"2024-09-16T05:39:26.049092+0000","last_unstale":"2024-09-16T05:53:06.368302+0000","last_undegraded":"2024-09-16T05:53:06.368302+0000","last_fullsized":"2024-09-16T05:53:06.368302+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:29.953136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":226,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:57.376805+0000","last_change":"2024-09-16T05:39:26.057943+0000","last_active":"2024-09-16T05:52:57.376805+0000","last_peered":"2024-09-16T05:52:57.376805+0000","last_clean":"2024-09-16T05:52:57.376805+0000","last_became_active":"2024-09-16T05:39:26.057564+0000","last_became_peered":"2024-09-16T05:39:26.057564+0000","last_unstale":"2024-09-16T05:52:57.376805+0000","last_undegraded":"2024-09-16T05:52:57.376805+0000","last_fullsized":"2024-09-16T05:52:57.376805+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:01:19.835552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356313+0000","last_change":"2024-09-16T05:39:26.060420+0000","last_active":"2024-09-16T05:51:56.356313+0000","last_peered":"2024-09-16T05:51:56.356313+0000","last_clean":"2024-09-16T05:51:56.356313+0000","last_became_active":"2024-09-16T05:39:26.059546+0000","last_became_peered":"2024-09-16T05:39:26.059546+0000","last_unstale":"2024-09-16T05:51:56.356313+0000","last_undegraded":"2024-09-16T05:51:56.356313+0000","last_fullsized":"2024-09-16T05:51:56.356313+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:24.267557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356350+0000","last_change":"2024-09-16T05:39:26.074051+0000","last_active":"2024-09-16T05:51:56.356350+0000","last_peered":"2024-09-16T05:51:56.356350+0000","last_clean":"2024-09-16T05:51:56.356350+0000","last_became_active":"2024-09-16T05:39:26.073016+0000","last_became_peered":"2024-09-16T05:39:26.073016+0000","last_unstale":"2024-09-16T05:51:56.356350+0000","last_undegraded":"2024-09-16T05:51:56.356350+0000","last_fullsized":"2024-09-16T05:51:56.356350+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:30.421445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":289,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485788+0000","last_change":"2024-09-16T05:39:26.077800+0000","last_active":"2024-09-16T05:52:56.485788+0000","last_peered":"2024-09-16T05:52:56.485788+0000","last_clean":"2024-09-16T05:52:56.485788+0000","last_became_active":"2024-09-16T05:39:26.076818+0000","last_became_peered":"2024-09-16T05:39:26.076818+0000","last_unstale":"2024-09-16T05:52:56.485788+0000","last_undegraded":"2024-09-16T05:52:56.485788+0000","last_fullsized":"2024-09-16T05:52:56.485788+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:13:30.887596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":184,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.472790+0000","last_change":"2024-09-16T05:39:27.051741+0000","last_active":"2024-09-16T05:51:56.472790+0000","last_peered":"2024-09-16T05:51:56.472790+0000","last_clean":"2024-09-16T05:51:56.472790+0000","last_became_active":"2024-09-16T05:39:27.050732+0000","last_became_peered":"2024-09-16T05:39:27.050732+0000","last_unstale":"2024-09-16T05:51:56.472790+0000","last_undegraded":"2024-09-16T05:51:56.472790+0000","last_fullsized":"2024-09-16T05:51:56.472790+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:18.147347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:31.369313+0000","last_change":"2024-09-16T05:39:26.053064+0000","last_active":"2024-09-16T05:53:31.369313+0000","last_peered":"2024-09-16T05:53:31.369313+0000","last_clean":"2024-09-16T05:53:31.369313+0000","last_became_active":"2024-09-16T05:39:26.052209+0000","last_became_peered":"2024-09-16T05:39:26.052209+0000","last_unstale":"2024-09-16T05:53:31.369313+0000","last_undegraded":"2024-09-16T05:53:31.369313+0000","last_fullsized":"2024-09-16T05:53:31.369313+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:24.007866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":591,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.366813+0000","last_change":"2024-09-16T05:39:26.061773+0000","last_active":"2024-09-16T05:52:56.366813+0000","last_peered":"2024-09-16T05:52:56.366813+0000","last_clean":"2024-09-16T05:52:56.366813+0000","last_became_active":"2024-09-16T05:39:26.061101+0000","last_became_peered":"2024-09-16T05:39:26.061101+0000","last_unstale":"2024-09-16T05:52:56.366813+0000","last_undegraded":"2024-09-16T05:52:56.366813+0000","last_fullsized":"2024-09-16T05:52:56.366813+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_clean_scrub_stamp":"2024-09-16T05:34:47.359645+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:56:28.349578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":441,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:57.369715+0000","last_change":"2024-09-16T05:39:26.073172+0000","last_active":"2024-09-16T05:51:57.369715+0000","last_peered":"2024-09-16T05:51:57.369715+0000","last_clean":"2024-09-16T05:51:57.369715+0000","last_became_active":"2024-09-16T05:39:26.063463+0000","last_became_peered":"2024-09-16T05:39:26.063463+0000","last_unstale":"2024-09-16T05:51:57.369715+0000","last_undegraded":"2024-09-16T05:51:57.369715+0000","last_fullsized":"2024-09-16T05:51:57.369715+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:08.651405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:01.367585+0000","last_change":"2024-09-16T05:39:26.060982+0000","last_active":"2024-09-16T05:53:01.367585+0000","last_peered":"2024-09-16T05:53:01.367585+0000","last_clean":"2024-09-16T05:53:01.367585+0000","last_became_active":"2024-09-16T05:39:26.060550+0000","last_became_peered":"2024-09-16T05:39:26.060550+0000","last_unstale":"2024-09-16T05:53:01.367585+0000","last_undegraded":"2024-09-16T05:53:01.367585+0000","last_fullsized":"2024-09-16T05:53:01.367585+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:07:51.890578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.472754+0000","last_change":"2024-09-16T05:39:26.059723+0000","last_active":"2024-09-16T05:51:56.472754+0000","last_peered":"2024-09-16T05:51:56.472754+0000","last_clean":"2024-09-16T05:51:56.472754+0000","last_became_active":"2024-09-16T05:39:26.058771+0000","last_became_peered":"2024-09-16T05:39:26.058771+0000","last_unstale":"2024-09-16T05:51:56.472754+0000","last_undegraded":"2024-09-16T05:51:56.472754+0000","last_fullsized":"2024-09-16T05:51:56.472754+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:58:48.900055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":450,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:57.369673+0000","last_change":"2024-09-16T05:39:26.080898+0000","last_active":"2024-09-16T05:51:57.369673+0000","last_peered":"2024-09-16T05:51:57.369673+0000","last_clean":"2024-09-16T05:51:57.369673+0000","last_became_active":"2024-09-16T05:39:26.065589+0000","last_became_peered":"2024-09-16T05:39:26.065589+0000","last_unstale":"2024-09-16T05:51:57.369673+0000","last_undegraded":"2024-09-16T05:51:57.369673+0000","last_fullsized":"2024-09-16T05:51:57.369673+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:54.750367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":430,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356395+0000","last_change":"2024-09-16T05:39:27.057353+0000","last_active":"2024-09-16T05:51:56.356395+0000","last_peered":"2024-09-16T05:51:56.356395+0000","last_clean":"2024-09-16T05:51:56.356395+0000","last_became_active":"2024-09-16T05:39:27.056262+0000","last_became_peered":"2024-09-16T05:39:27.056262+0000","last_unstale":"2024-09-16T05:51:56.356395+0000","last_undegraded":"2024-09-16T05:51:56.356395+0000","last_fullsized":"2024-09-16T05:51:56.356395+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:41.799365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:01.358717+0000","last_change":"2024-09-16T05:39:26.060534+0000","last_active":"2024-09-16T05:52:01.358717+0000","last_peered":"2024-09-16T05:52:01.358717+0000","last_clean":"2024-09-16T05:52:01.358717+0000","last_became_active":"2024-09-16T05:39:26.060029+0000","last_became_peered":"2024-09-16T05:39:26.060029+0000","last_unstale":"2024-09-16T05:52:01.358717+0000","last_undegraded":"2024-09-16T05:52:01.358717+0000","last_fullsized":"2024-09-16T05:52:01.358717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:47.905126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.360460+0000","last_change":"2024-09-16T05:39:26.055632+0000","last_active":"2024-09-16T05:52:26.360460+0000","last_peered":"2024-09-16T05:52:26.360460+0000","last_clean":"2024-09-16T05:52:26.360460+0000","last_became_active":"2024-09-16T05:39:26.054759+0000","last_became_peered":"2024-09-16T05:39:26.054759+0000","last_unstale":"2024-09-16T05:52:26.360460+0000","last_undegraded":"2024-09-16T05:52:26.360460+0000","last_fullsized":"2024-09-16T05:52:26.360460+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:14.840382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.362449+0000","last_change":"2024-09-16T05:39:26.051119+0000","last_active":"2024-09-16T05:52:26.362449+0000","last_peered":"2024-09-16T05:52:26.362449+0000","last_clean":"2024-09-16T05:52:26.362449+0000","last_became_active":"2024-09-16T05:39:26.050738+0000","last_became_peered":"2024-09-16T05:39:26.050738+0000","last_unstale":"2024-09-16T05:52:26.362449+0000","last_undegraded":"2024-09-16T05:52:26.362449+0000","last_fullsized":"2024-09-16T05:52:26.362449+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:37.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607807,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640509,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34328,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337512,"statfs":{"total":96636764160,"available":96601612288,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640509,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640508,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640511,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34328,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337512,"statfs":{"total":96636764160,"available":96601612288,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:55:38.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:55:38.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient: get_monmap_and_config 2024-09-16T05:55:38.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient: build_initial_monmap 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient: monmap: 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:38.209248+0000 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:38.209248+0000 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:38.208 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:38.209 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 1 Processor -- start 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 1 -- start start 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient: init 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e80a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e80a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e80a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e806e520 con 0x7fb1e80a37a0 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e80a0780 con 0x7fb1e80a1a00 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e80a4c70 con 0x7fb1e80a4290 2024-09-16T05:55:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.203+0000 7fb1efd30640 10 monclient(hunting): authenticate will time out at 2200.015954s 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ed2cb640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb1e80a4290 0x7fb1e813bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39542/0 (socket says 172.21.15.6:39542) 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ed2cb640 1 -- 172.21.15.6:0/3176577059 learned_addr learned my addr 172.21.15.6:0/3176577059 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1698997981 0 0) 0x7fb1e80a4c70 con 0x7fb1e80a4290 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient(hunting): my global_id is 164471 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d4003530 con 0x7fb1e80a4290 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720205860 0 0) 0x7fb1e806e520 con 0x7fb1e80a37a0 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient(hunting): my global_id is 194145 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1e80a4c70 con 0x7fb1e80a37a0 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3082593410 0 0) 0x7fb1d4003530 con 0x7fb1e80a4290 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 >> v1:172.21.15.63:6789/0 conn(0x7fb1e80a1a00 legacy=0x7fb1e813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 >> v1:172.21.15.6:6789/0 conn(0x7fb1e80a37a0 legacy=0x7fb1e80a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: _finish_hunting 0 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 monclient: found mon.noname-b 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e80a65f0 con 0x7fb1e80a4290 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: _finish_auth 0 2024-09-16T05:55:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: _check_auth_tickets 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1e40015a0 con 0x7fb1e80a4290 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: dump: 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 5 monclient: authenticate success, global_id 164471 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1e4001800 con 0x7fb1e80a4290 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 1 -- 172.21.15.6:0/3176577059 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1e4001bd0 con 0x7fb1e80a4290 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1ecaca640 10 monclient: dump: 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 4 set_mon_vals no callback set 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: shutdown 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- 172.21.15.6:0/3176577059 >> v1:172.21.15.6:6790/0 conn(0x7fb1e80a4290 legacy=0x7fb1e813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- 172.21.15.6:0/3176577059 shutdown_connections 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- 172.21.15.6:0/3176577059 >> 172.21.15.6:0/3176577059 conn(0x7fb1e80a58a0 msgr2=0x7fb1e80a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- 172.21.15.6:0/3176577059 shutdown_connections 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- 172.21.15.6:0/3176577059 wait complete. 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: build_initial_monmap 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: monmap: 2024-09-16T05:55:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 Processor -- start 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- start start 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: init 2024-09-16T05:55:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e80a1a00 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e80a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e80a4290 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e813de50 con 0x7fb1e80a1a00 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e813f300 con 0x7fb1e80a4290 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e813ed40 con 0x7fb1e80a37a0 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient(hunting): authenticate will time out at 2200.018788s 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1edacc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb1e80a1a00 0x7fb1e81cbf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39264/0 (socket says 172.21.15.6:39264) 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1edacc640 1 -- 172.21.15.6:0/1785391134 learned_addr learned my addr 172.21.15.6:0/1785391134 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2073872369 0 0) 0x7fb1e813ed40 con 0x7fb1e80a37a0 2024-09-16T05:55:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): my global_id is 164474 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1cc003530 con 0x7fb1e80a37a0 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497174611 0 0) 0x7fb1e813de50 con 0x7fb1e80a1a00 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): my global_id is 194148 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1e813ed40 con 0x7fb1e80a1a00 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2326556978 0 0) 0x7fb1e813ed40 con 0x7fb1e80a1a00 2024-09-16T05:55:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1e813de50 con 0x7fb1e80a1a00 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3755909407 0 0) 0x7fb1cc003530 con 0x7fb1e80a37a0 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1e813ed40 con 0x7fb1e80a37a0 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d8001980 con 0x7fb1e80a1a00 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): dump: 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1e4001b50 con 0x7fb1e80a37a0 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): dump: 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.227 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3886482995 0 0) 0x7fb1e813f300 con 0x7fb1e80a4290 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient(hunting): my global_id is 184249 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1cc003530 con 0x7fb1e80a4290 2024-09-16T05:55:38.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3884215132 0 0) 0x7fb1e813de50 con 0x7fb1e80a1a00 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 >> v1:172.21.15.6:6790/0 conn(0x7fb1e80a37a0 legacy=0x7fb1e81d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 >> v1:172.21.15.63:6789/0 conn(0x7fb1e80a4290 legacy=0x7fb1e81d46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 monclient: found mon.a 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e813f020 con 0x7fb1e80a1a00 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient: _finish_auth 0 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 5 monclient: authenticate success, global_id 194148 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: _renew_subs 2024-09-16T05:55:38.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e813dab0 con 0x7fb1e80a1a00 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: _renew_subs 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1e813c660 con 0x7fb1e80a1a00 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1d8001cf0 con 0x7fb1e80a1a00 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fb1d8003a70 con 0x7fb1e80a1a00 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1de7fc640 10 monclient: dump: 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.230 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.207+0000 7fb1dffff640 4 set_mon_vals no callback set 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.211+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fb1d8003c10 con 0x7fb1e80a1a00 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.211+0000 7fb1efd30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.211+0000 7fb1efd30640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.211+0000 7fb1efd30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.211+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1d0005090 con 0x7fb1e80a1a00 2024-09-16T05:55:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.211+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fb1d8044e40 con 0x7fb1e80a1a00 2024-09-16T05:55:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.211+0000 7fb1ed2cb640 10 monclient: get_auth_request con 0x7fb1cc036c50 auth_method 0 2024-09-16T05:55:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.215+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fb1d8015410 con 0x7fb1e80a1a00 2024-09-16T05:55:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.215+0000 7fb1de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.215+0000 7fb1de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:38.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.379+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb1d0000ba0 con 0x7fb1cc036c50 2024-09-16T05:55:38.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.383+0000 7fb1de7fc640 1 -- 172.21.15.6:0/1785391134 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77961 (unknown 2965378022 0 50223216) 0x7fb1d0000ba0 con 0x7fb1cc036c50 2024-09-16T05:55:38.393 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:55:38.394 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:55:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.387+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 >> v1:172.21.15.63:6812/3643207211 conn(0x7fb1cc036c50 legacy=0x7fb1cc039110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.387+0000 7fb1efd30640 10 monclient: shutdown 2024-09-16T05:55:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.387+0000 7fb1efd30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.387+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 >> v1:172.21.15.6:6789/0 conn(0x7fb1e80a1a00 legacy=0x7fb1e81cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.387+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 shutdown_connections 2024-09-16T05:55:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.387+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 >> 172.21.15.6:0/1785391134 conn(0x7fb1e80a58a0 msgr2=0x7fb1e809d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.387+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 shutdown_connections 2024-09-16T05:55:38.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.387+0000 7fb1efd30640 1 -- 172.21.15.6:0/1785391134 wait complete. 2024-09-16T05:55:38.407 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":880,"stamp":"2024-09-16T05:55:37.626572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205768,"kb_used_data":18640,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566025272,"statfs":{"total":579820584960,"available":579609878528,"internally_reserved":0,"allocated":19087360,"data_stored":15446128,"data_compressed":1794170,"data_compressed_allocated":2719744,"data_compressed_original":5914624,"omap_allocated":9566,"internal_metadata":191486626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003677"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":288,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485848+0000","last_change":"2024-09-16T05:39:26.074706+0000","last_active":"2024-09-16T05:52:56.485848+0000","last_peered":"2024-09-16T05:52:56.485848+0000","last_clean":"2024-09-16T05:52:56.485848+0000","last_became_active":"2024-09-16T05:39:26.073689+0000","last_became_peered":"2024-09-16T05:39:26.073689+0000","last_unstale":"2024-09-16T05:52:56.485848+0000","last_undegraded":"2024-09-16T05:52:56.485848+0000","last_fullsized":"2024-09-16T05:52:56.485848+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:22.219893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:27.376074+0000","last_change":"2024-09-16T05:39:26.053781+0000","last_active":"2024-09-16T05:52:27.376074+0000","last_peered":"2024-09-16T05:52:27.376074+0000","last_clean":"2024-09-16T05:52:27.376074+0000","last_became_active":"2024-09-16T05:39:26.053397+0000","last_became_peered":"2024-09-16T05:39:26.053397+0000","last_unstale":"2024-09-16T05:52:27.376074+0000","last_undegraded":"2024-09-16T05:52:27.376074+0000","last_fullsized":"2024-09-16T05:52:27.376074+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:58.904858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":294,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.480780+0000","last_change":"2024-09-16T05:39:26.070406+0000","last_active":"2024-09-16T05:52:56.480780+0000","last_peered":"2024-09-16T05:52:56.480780+0000","last_clean":"2024-09-16T05:52:56.480780+0000","last_became_active":"2024-09-16T05:39:26.069114+0000","last_became_peered":"2024-09-16T05:39:26.069114+0000","last_unstale":"2024-09-16T05:52:56.480780+0000","last_undegraded":"2024-09-16T05:52:56.480780+0000","last_fullsized":"2024-09-16T05:52:56.480780+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:19:02.873639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:27.376003+0000","last_change":"2024-09-16T05:39:26.056400+0000","last_active":"2024-09-16T05:52:27.376003+0000","last_peered":"2024-09-16T05:52:27.376003+0000","last_clean":"2024-09-16T05:52:27.376003+0000","last_became_active":"2024-09-16T05:39:26.055883+0000","last_became_peered":"2024-09-16T05:39:26.055883+0000","last_unstale":"2024-09-16T05:52:27.376003+0000","last_undegraded":"2024-09-16T05:52:27.376003+0000","last_fullsized":"2024-09-16T05:52:27.376003+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:38.780469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":224,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:22.364898+0000","last_change":"2024-09-16T05:39:27.057197+0000","last_active":"2024-09-16T05:51:22.364898+0000","last_peered":"2024-09-16T05:51:22.364898+0000","last_clean":"2024-09-16T05:51:22.364898+0000","last_became_active":"2024-09-16T05:39:27.056198+0000","last_became_peered":"2024-09-16T05:39:27.056198+0000","last_unstale":"2024-09-16T05:51:22.364898+0000","last_undegraded":"2024-09-16T05:51:22.364898+0000","last_fullsized":"2024-09-16T05:51:22.364898+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:54.298350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":276,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485909+0000","last_change":"2024-09-16T05:39:26.074867+0000","last_active":"2024-09-16T05:52:56.485909+0000","last_peered":"2024-09-16T05:52:56.485909+0000","last_clean":"2024-09-16T05:52:56.485909+0000","last_became_active":"2024-09-16T05:39:26.073067+0000","last_became_peered":"2024-09-16T05:39:26.073067+0000","last_unstale":"2024-09-16T05:52:56.485909+0000","last_undegraded":"2024-09-16T05:52:56.485909+0000","last_fullsized":"2024-09-16T05:52:56.485909+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:24.660359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":176,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.362499+0000","last_change":"2024-09-16T05:39:26.061744+0000","last_active":"2024-09-16T05:52:26.362499+0000","last_peered":"2024-09-16T05:52:26.362499+0000","last_clean":"2024-09-16T05:52:26.362499+0000","last_became_active":"2024-09-16T05:39:26.061165+0000","last_became_peered":"2024-09-16T05:39:26.061165+0000","last_unstale":"2024-09-16T05:52:26.362499+0000","last_undegraded":"2024-09-16T05:52:26.362499+0000","last_fullsized":"2024-09-16T05:52:26.362499+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:26.707122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356374+0000","last_change":"2024-09-16T05:39:26.053923+0000","last_active":"2024-09-16T05:51:56.356374+0000","last_peered":"2024-09-16T05:51:56.356374+0000","last_clean":"2024-09-16T05:51:56.356374+0000","last_became_active":"2024-09-16T05:39:26.053013+0000","last_became_peered":"2024-09-16T05:39:26.053013+0000","last_unstale":"2024-09-16T05:51:56.356374+0000","last_undegraded":"2024-09-16T05:51:56.356374+0000","last_fullsized":"2024-09-16T05:51:56.356374+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:32.161682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":209,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:37.377442+0000","last_change":"2024-09-16T05:39:26.059457+0000","last_active":"2024-09-16T05:52:37.377442+0000","last_peered":"2024-09-16T05:52:37.377442+0000","last_clean":"2024-09-16T05:52:37.377442+0000","last_became_active":"2024-09-16T05:39:26.058997+0000","last_became_peered":"2024-09-16T05:39:26.058997+0000","last_unstale":"2024-09-16T05:52:37.377442+0000","last_undegraded":"2024-09-16T05:52:37.377442+0000","last_fullsized":"2024-09-16T05:52:37.377442+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:25.563929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":213,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:06.368302+0000","last_change":"2024-09-16T05:39:26.049544+0000","last_active":"2024-09-16T05:53:06.368302+0000","last_peered":"2024-09-16T05:53:06.368302+0000","last_clean":"2024-09-16T05:53:06.368302+0000","last_became_active":"2024-09-16T05:39:26.049092+0000","last_became_peered":"2024-09-16T05:39:26.049092+0000","last_unstale":"2024-09-16T05:53:06.368302+0000","last_undegraded":"2024-09-16T05:53:06.368302+0000","last_fullsized":"2024-09-16T05:53:06.368302+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:29.953136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":226,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:57.376805+0000","last_change":"2024-09-16T05:39:26.057943+0000","last_active":"2024-09-16T05:52:57.376805+0000","last_peered":"2024-09-16T05:52:57.376805+0000","last_clean":"2024-09-16T05:52:57.376805+0000","last_became_active":"2024-09-16T05:39:26.057564+0000","last_became_peered":"2024-09-16T05:39:26.057564+0000","last_unstale":"2024-09-16T05:52:57.376805+0000","last_undegraded":"2024-09-16T05:52:57.376805+0000","last_fullsized":"2024-09-16T05:52:57.376805+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:01:19.835552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356313+0000","last_change":"2024-09-16T05:39:26.060420+0000","last_active":"2024-09-16T05:51:56.356313+0000","last_peered":"2024-09-16T05:51:56.356313+0000","last_clean":"2024-09-16T05:51:56.356313+0000","last_became_active":"2024-09-16T05:39:26.059546+0000","last_became_peered":"2024-09-16T05:39:26.059546+0000","last_unstale":"2024-09-16T05:51:56.356313+0000","last_undegraded":"2024-09-16T05:51:56.356313+0000","last_fullsized":"2024-09-16T05:51:56.356313+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:24.267557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356350+0000","last_change":"2024-09-16T05:39:26.074051+0000","last_active":"2024-09-16T05:51:56.356350+0000","last_peered":"2024-09-16T05:51:56.356350+0000","last_clean":"2024-09-16T05:51:56.356350+0000","last_became_active":"2024-09-16T05:39:26.073016+0000","last_became_peered":"2024-09-16T05:39:26.073016+0000","last_unstale":"2024-09-16T05:51:56.356350+0000","last_undegraded":"2024-09-16T05:51:56.356350+0000","last_fullsized":"2024-09-16T05:51:56.356350+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:30.421445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":289,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485788+0000","last_change":"2024-09-16T05:39:26.077800+0000","last_active":"2024-09-16T05:52:56.485788+0000","last_peered":"2024-09-16T05:52:56.485788+0000","last_clean":"2024-09-16T05:52:56.485788+0000","last_became_active":"2024-09-16T05:39:26.076818+0000","last_became_peered":"2024-09-16T05:39:26.076818+0000","last_unstale":"2024-09-16T05:52:56.485788+0000","last_undegraded":"2024-09-16T05:52:56.485788+0000","last_fullsized":"2024-09-16T05:52:56.485788+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:13:30.887596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":184,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.472790+0000","last_change":"2024-09-16T05:39:27.051741+0000","last_active":"2024-09-16T05:51:56.472790+0000","last_peered":"2024-09-16T05:51:56.472790+0000","last_clean":"2024-09-16T05:51:56.472790+0000","last_became_active":"2024-09-16T05:39:27.050732+0000","last_became_peered":"2024-09-16T05:39:27.050732+0000","last_unstale":"2024-09-16T05:51:56.472790+0000","last_undegraded":"2024-09-16T05:51:56.472790+0000","last_fullsized":"2024-09-16T05:51:56.472790+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:18.147347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:31.369313+0000","last_change":"2024-09-16T05:39:26.053064+0000","last_active":"2024-09-16T05:53:31.369313+0000","last_peered":"2024-09-16T05:53:31.369313+0000","last_clean":"2024-09-16T05:53:31.369313+0000","last_became_active":"2024-09-16T05:39:26.052209+0000","last_became_peered":"2024-09-16T05:39:26.052209+0000","last_unstale":"2024-09-16T05:53:31.369313+0000","last_undegraded":"2024-09-16T05:53:31.369313+0000","last_fullsized":"2024-09-16T05:53:31.369313+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:24.007866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":591,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.366813+0000","last_change":"2024-09-16T05:39:26.061773+0000","last_active":"2024-09-16T05:52:56.366813+0000","last_peered":"2024-09-16T05:52:56.366813+0000","last_clean":"2024-09-16T05:52:56.366813+0000","last_became_active":"2024-09-16T05:39:26.061101+0000","last_became_peered":"2024-09-16T05:39:26.061101+0000","last_unstale":"2024-09-16T05:52:56.366813+0000","last_undegraded":"2024-09-16T05:52:56.366813+0000","last_fullsized":"2024-09-16T05:52:56.366813+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_clean_scrub_stamp":"2024-09-16T05:34:47.359645+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:56:28.349578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":441,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:57.369715+0000","last_change":"2024-09-16T05:39:26.073172+0000","last_active":"2024-09-16T05:51:57.369715+0000","last_peered":"2024-09-16T05:51:57.369715+0000","last_clean":"2024-09-16T05:51:57.369715+0000","last_became_active":"2024-09-16T05:39:26.063463+0000","last_became_peered":"2024-09-16T05:39:26.063463+0000","last_unstale":"2024-09-16T05:51:57.369715+0000","last_undegraded":"2024-09-16T05:51:57.369715+0000","last_fullsized":"2024-09-16T05:51:57.369715+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:08.651405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:01.367585+0000","last_change":"2024-09-16T05:39:26.060982+0000","last_active":"2024-09-16T05:53:01.367585+0000","last_peered":"2024-09-16T05:53:01.367585+0000","last_clean":"2024-09-16T05:53:01.367585+0000","last_became_active":"2024-09-16T05:39:26.060550+0000","last_became_peered":"2024-09-16T05:39:26.060550+0000","last_unstale":"2024-09-16T05:53:01.367585+0000","last_undegraded":"2024-09-16T05:53:01.367585+0000","last_fullsized":"2024-09-16T05:53:01.367585+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:07:51.890578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.472754+0000","last_change":"2024-09-16T05:39:26.059723+0000","last_active":"2024-09-16T05:51:56.472754+0000","last_peered":"2024-09-16T05:51:56.472754+0000","last_clean":"2024-09-16T05:51:56.472754+0000","last_became_active":"2024-09-16T05:39:26.058771+0000","last_became_peered":"2024-09-16T05:39:26.058771+0000","last_unstale":"2024-09-16T05:51:56.472754+0000","last_undegraded":"2024-09-16T05:51:56.472754+0000","last_fullsized":"2024-09-16T05:51:56.472754+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:58:48.900055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":450,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:57.369673+0000","last_change":"2024-09-16T05:39:26.080898+0000","last_active":"2024-09-16T05:51:57.369673+0000","last_peered":"2024-09-16T05:51:57.369673+0000","last_clean":"2024-09-16T05:51:57.369673+0000","last_became_active":"2024-09-16T05:39:26.065589+0000","last_became_peered":"2024-09-16T05:39:26.065589+0000","last_unstale":"2024-09-16T05:51:57.369673+0000","last_undegraded":"2024-09-16T05:51:57.369673+0000","last_fullsized":"2024-09-16T05:51:57.369673+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:54.750367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":430,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356395+0000","last_change":"2024-09-16T05:39:27.057353+0000","last_active":"2024-09-16T05:51:56.356395+0000","last_peered":"2024-09-16T05:51:56.356395+0000","last_clean":"2024-09-16T05:51:56.356395+0000","last_became_active":"2024-09-16T05:39:27.056262+0000","last_became_peered":"2024-09-16T05:39:27.056262+0000","last_unstale":"2024-09-16T05:51:56.356395+0000","last_undegraded":"2024-09-16T05:51:56.356395+0000","last_fullsized":"2024-09-16T05:51:56.356395+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:41.799365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:01.358717+0000","last_change":"2024-09-16T05:39:26.060534+0000","last_active":"2024-09-16T05:52:01.358717+0000","last_peered":"2024-09-16T05:52:01.358717+0000","last_clean":"2024-09-16T05:52:01.358717+0000","last_became_active":"2024-09-16T05:39:26.060029+0000","last_became_peered":"2024-09-16T05:39:26.060029+0000","last_unstale":"2024-09-16T05:52:01.358717+0000","last_undegraded":"2024-09-16T05:52:01.358717+0000","last_fullsized":"2024-09-16T05:52:01.358717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:47.905126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.360460+0000","last_change":"2024-09-16T05:39:26.055632+0000","last_active":"2024-09-16T05:52:26.360460+0000","last_peered":"2024-09-16T05:52:26.360460+0000","last_clean":"2024-09-16T05:52:26.360460+0000","last_became_active":"2024-09-16T05:39:26.054759+0000","last_became_peered":"2024-09-16T05:39:26.054759+0000","last_unstale":"2024-09-16T05:52:26.360460+0000","last_undegraded":"2024-09-16T05:52:26.360460+0000","last_fullsized":"2024-09-16T05:52:26.360460+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:14.840382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.362449+0000","last_change":"2024-09-16T05:39:26.051119+0000","last_active":"2024-09-16T05:52:26.362449+0000","last_peered":"2024-09-16T05:52:26.362449+0000","last_clean":"2024-09-16T05:52:26.362449+0000","last_became_active":"2024-09-16T05:39:26.050738+0000","last_became_peered":"2024-09-16T05:39:26.050738+0000","last_unstale":"2024-09-16T05:52:26.362449+0000","last_undegraded":"2024-09-16T05:52:26.362449+0000","last_fullsized":"2024-09-16T05:52:26.362449+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:37.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607807,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640509,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34328,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337512,"statfs":{"total":96636764160,"available":96601612288,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640509,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640508,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640511,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34328,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337512,"statfs":{"total":96636764160,"available":96601612288,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:55:38.409 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T05:55:38.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:55:38.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.519+0000 7fcfd8a4a640 10 monclient: get_monmap_and_config 2024-09-16T05:55:38.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.519+0000 7fcfd8a4a640 10 monclient: build_initial_monmap 2024-09-16T05:55:38.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.519+0000 7fcfd8a4a640 10 monclient: monmap: 2024-09-16T05:55:38.525 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:38.525 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.525 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:38.526356+0000 2024-09-16T05:55:38.526 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:38.526356+0000 2024-09-16T05:55:38.526 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:38.526 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.526 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:38.526 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:38.526 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:38.526 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 Processor -- start 2024-09-16T05:55:38.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- start start 2024-09-16T05:55:38.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient: init 2024-09-16T05:55:38.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fcfd409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fcfd409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fcfd40a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd40a1370 con 0x7fcfd409e230 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd409ead0 con 0x7fcfd40a0da0 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd409edb0 con 0x7fcfd409aa30 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient(hunting): authenticate will time out at 2200.333261s 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd1d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcfd409e230 0x7fcfd41399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39270/0 (socket says 172.21.15.6:39270) 2024-09-16T05:55:38.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd1d9b640 1 -- 172.21.15.6:0/468813897 learned_addr learned my addr 172.21.15.6:0/468813897 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:38.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3290314535 0 0) 0x7fcfd40a1370 con 0x7fcfd409e230 2024-09-16T05:55:38.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): my global_id is 194151 2024-09-16T05:55:38.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfc0003530 con 0x7fcfd409e230 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315870609 0 0) 0x7fcfd409edb0 con 0x7fcfd409aa30 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): my global_id is 164477 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfd40a1370 con 0x7fcfd409aa30 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522378998 0 0) 0x7fcfd409ead0 con 0x7fcfd40a0da0 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient(hunting): my global_id is 184252 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfd409edb0 con 0x7fcfd40a0da0 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 212901786 0 0) 0x7fcfc0003530 con 0x7fcfd409e230 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 >> v1:172.21.15.6:6790/0 conn(0x7fcfd409aa30 legacy=0x7fcfd409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 >> v1:172.21.15.63:6789/0 conn(0x7fcfd40a0da0 legacy=0x7fcfd413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: _finish_hunting 0 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 monclient: found mon.noname-a 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd409dd50 con 0x7fcfd409e230 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: _finish_auth 0 2024-09-16T05:55:38.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: _check_auth_tickets 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 5 monclient: authenticate success, global_id 194151 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfc4001620 con 0x7fcfd409e230 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: dump: 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.539 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcfc40018d0 con 0x7fcfd409e230 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 1 -- 172.21.15.6:0/468813897 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfc4001ce0 con 0x7fcfd409e230 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd159a640 10 monclient: dump: 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.541 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 4 set_mon_vals no callback set 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient: shutdown 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/468813897 >> v1:172.21.15.6:6789/0 conn(0x7fcfd409e230 legacy=0x7fcfd41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/468813897 shutdown_connections 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/468813897 >> 172.21.15.6:0/468813897 conn(0x7fcfd40a1650 msgr2=0x7fcfd40a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/468813897 shutdown_connections 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/468813897 wait complete. 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient: build_initial_monmap 2024-09-16T05:55:38.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 10 monclient: monmap: 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 Processor -- start 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.523+0000 7fcfd8a4a640 1 -- start start 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: init 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fcfd409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fcfd409e230 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fcfd40a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd413d090 con 0x7fcfd409e230 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd406e8d0 con 0x7fcfd40a0da0 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd40740c0 con 0x7fcfd409aa30 2024-09-16T05:55:38.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient(hunting): authenticate will time out at 2200.336551s 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fcfd409aa30 0x7fcfd41cb8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39586/0 (socket says 172.21.15.6:39586) 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd259c640 1 -- 172.21.15.6:0/2439844013 learned_addr learned my addr 172.21.15.6:0/2439844013 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3299350356 0 0) 0x7fcfd40740c0 con 0x7fcfd409aa30 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): my global_id is 164480 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfb4003530 con 0x7fcfd409aa30 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769400219 0 0) 0x7fcfd413d090 con 0x7fcfd409e230 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): my global_id is 194154 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfd40740c0 con 0x7fcfd409e230 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 985384989 0 0) 0x7fcfb4003530 con 0x7fcfd409aa30 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfd413d090 con 0x7fcfd409aa30 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfbc001aa0 con 0x7fcfd409aa30 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): dump: 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.546 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1832277017 0 0) 0x7fcfd40740c0 con 0x7fcfd409e230 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfb4003530 con 0x7fcfd409e230 2024-09-16T05:55:38.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfc4001c20 con 0x7fcfd409e230 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): dump: 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.548 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3535879529 0 0) 0x7fcfd406e8d0 con 0x7fcfd40a0da0 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient(hunting): my global_id is 184255 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfd40740c0 con 0x7fcfd40a0da0 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 795076647 0 0) 0x7fcfd413d090 con 0x7fcfd409aa30 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 >> v1:172.21.15.63:6789/0 conn(0x7fcfd40a0da0 legacy=0x7fcfd41d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 >> v1:172.21.15.6:6789/0 conn(0x7fcfd409e230 legacy=0x7fcfd41cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient: _finish_hunting 0 2024-09-16T05:55:38.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 monclient: found mon.c 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd409d770 con 0x7fcfd409aa30 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient: _finish_auth 0 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient: _check_auth_tickets 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 5 monclient: authenticate success, global_id 164480 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: _renew_subs 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd413cdb0 con 0x7fcfd409aa30 2024-09-16T05:55:38.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: _renew_subs 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcfd413d4b0 con 0x7fcfd409aa30 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcfbc003930 con 0x7fcfd409aa30 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fcfbc003c80 con 0x7fcfd409aa30 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 10 monclient: dump: 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.551 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd0d99640 4 set_mon_vals no callback set 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fcfbc003e60 con 0x7fcfd409aa30 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fcfbc044e70 con 0x7fcfd409aa30 2024-09-16T05:55:38.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb8005090 con 0x7fcfd409aa30 2024-09-16T05:55:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.527+0000 7fcfd1d9b640 10 monclient: get_auth_request con 0x7fcfb4036c00 auth_method 0 2024-09-16T05:55:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.531+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fcfbc0154c0 con 0x7fcfd409aa30 2024-09-16T05:55:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.531+0000 7fcfcaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.531+0000 7fcfcaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:38.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.687+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcfb8000ba0 con 0x7fcfb4036c00 2024-09-16T05:55:38.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfcaffd640 1 -- 172.21.15.6:0/2439844013 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77961 (unknown 2965378022 0 50223216) 0x7fcfb8000ba0 con 0x7fcfb4036c00 2024-09-16T05:55:38.697 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:55:38.697 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:55:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 >> v1:172.21.15.63:6812/3643207211 conn(0x7fcfb4036c00 legacy=0x7fcfb40390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfd8a4a640 10 monclient: shutdown 2024-09-16T05:55:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfd8a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 >> v1:172.21.15.6:6790/0 conn(0x7fcfd409aa30 legacy=0x7fcfd41cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 shutdown_connections 2024-09-16T05:55:38.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 >> 172.21.15.6:0/2439844013 conn(0x7fcfd40a1650 msgr2=0x7fcfd4071dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 shutdown_connections 2024-09-16T05:55:38.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.695+0000 7fcfd8a4a640 1 -- 172.21.15.6:0/2439844013 wait complete. 2024-09-16T05:55:38.713 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":880,"stamp":"2024-09-16T05:55:37.626572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205768,"kb_used_data":18640,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566025272,"statfs":{"total":579820584960,"available":579609878528,"internally_reserved":0,"allocated":19087360,"data_stored":15446128,"data_compressed":1794170,"data_compressed_allocated":2719744,"data_compressed_original":5914624,"omap_allocated":9566,"internal_metadata":191486626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003677"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":288,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485848+0000","last_change":"2024-09-16T05:39:26.074706+0000","last_active":"2024-09-16T05:52:56.485848+0000","last_peered":"2024-09-16T05:52:56.485848+0000","last_clean":"2024-09-16T05:52:56.485848+0000","last_became_active":"2024-09-16T05:39:26.073689+0000","last_became_peered":"2024-09-16T05:39:26.073689+0000","last_unstale":"2024-09-16T05:52:56.485848+0000","last_undegraded":"2024-09-16T05:52:56.485848+0000","last_fullsized":"2024-09-16T05:52:56.485848+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:22.219893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:27.376074+0000","last_change":"2024-09-16T05:39:26.053781+0000","last_active":"2024-09-16T05:52:27.376074+0000","last_peered":"2024-09-16T05:52:27.376074+0000","last_clean":"2024-09-16T05:52:27.376074+0000","last_became_active":"2024-09-16T05:39:26.053397+0000","last_became_peered":"2024-09-16T05:39:26.053397+0000","last_unstale":"2024-09-16T05:52:27.376074+0000","last_undegraded":"2024-09-16T05:52:27.376074+0000","last_fullsized":"2024-09-16T05:52:27.376074+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:58.904858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":294,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.480780+0000","last_change":"2024-09-16T05:39:26.070406+0000","last_active":"2024-09-16T05:52:56.480780+0000","last_peered":"2024-09-16T05:52:56.480780+0000","last_clean":"2024-09-16T05:52:56.480780+0000","last_became_active":"2024-09-16T05:39:26.069114+0000","last_became_peered":"2024-09-16T05:39:26.069114+0000","last_unstale":"2024-09-16T05:52:56.480780+0000","last_undegraded":"2024-09-16T05:52:56.480780+0000","last_fullsized":"2024-09-16T05:52:56.480780+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:19:02.873639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:27.376003+0000","last_change":"2024-09-16T05:39:26.056400+0000","last_active":"2024-09-16T05:52:27.376003+0000","last_peered":"2024-09-16T05:52:27.376003+0000","last_clean":"2024-09-16T05:52:27.376003+0000","last_became_active":"2024-09-16T05:39:26.055883+0000","last_became_peered":"2024-09-16T05:39:26.055883+0000","last_unstale":"2024-09-16T05:52:27.376003+0000","last_undegraded":"2024-09-16T05:52:27.376003+0000","last_fullsized":"2024-09-16T05:52:27.376003+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:38.780469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":224,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:22.364898+0000","last_change":"2024-09-16T05:39:27.057197+0000","last_active":"2024-09-16T05:51:22.364898+0000","last_peered":"2024-09-16T05:51:22.364898+0000","last_clean":"2024-09-16T05:51:22.364898+0000","last_became_active":"2024-09-16T05:39:27.056198+0000","last_became_peered":"2024-09-16T05:39:27.056198+0000","last_unstale":"2024-09-16T05:51:22.364898+0000","last_undegraded":"2024-09-16T05:51:22.364898+0000","last_fullsized":"2024-09-16T05:51:22.364898+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:54.298350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":276,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485909+0000","last_change":"2024-09-16T05:39:26.074867+0000","last_active":"2024-09-16T05:52:56.485909+0000","last_peered":"2024-09-16T05:52:56.485909+0000","last_clean":"2024-09-16T05:52:56.485909+0000","last_became_active":"2024-09-16T05:39:26.073067+0000","last_became_peered":"2024-09-16T05:39:26.073067+0000","last_unstale":"2024-09-16T05:52:56.485909+0000","last_undegraded":"2024-09-16T05:52:56.485909+0000","last_fullsized":"2024-09-16T05:52:56.485909+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:03:24.660359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":176,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.362499+0000","last_change":"2024-09-16T05:39:26.061744+0000","last_active":"2024-09-16T05:52:26.362499+0000","last_peered":"2024-09-16T05:52:26.362499+0000","last_clean":"2024-09-16T05:52:26.362499+0000","last_became_active":"2024-09-16T05:39:26.061165+0000","last_became_peered":"2024-09-16T05:39:26.061165+0000","last_unstale":"2024-09-16T05:52:26.362499+0000","last_undegraded":"2024-09-16T05:52:26.362499+0000","last_fullsized":"2024-09-16T05:52:26.362499+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:26.707122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356374+0000","last_change":"2024-09-16T05:39:26.053923+0000","last_active":"2024-09-16T05:51:56.356374+0000","last_peered":"2024-09-16T05:51:56.356374+0000","last_clean":"2024-09-16T05:51:56.356374+0000","last_became_active":"2024-09-16T05:39:26.053013+0000","last_became_peered":"2024-09-16T05:39:26.053013+0000","last_unstale":"2024-09-16T05:51:56.356374+0000","last_undegraded":"2024-09-16T05:51:56.356374+0000","last_fullsized":"2024-09-16T05:51:56.356374+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:32.161682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":209,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:37.377442+0000","last_change":"2024-09-16T05:39:26.059457+0000","last_active":"2024-09-16T05:52:37.377442+0000","last_peered":"2024-09-16T05:52:37.377442+0000","last_clean":"2024-09-16T05:52:37.377442+0000","last_became_active":"2024-09-16T05:39:26.058997+0000","last_became_peered":"2024-09-16T05:39:26.058997+0000","last_unstale":"2024-09-16T05:52:37.377442+0000","last_undegraded":"2024-09-16T05:52:37.377442+0000","last_fullsized":"2024-09-16T05:52:37.377442+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:25.563929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":213,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:06.368302+0000","last_change":"2024-09-16T05:39:26.049544+0000","last_active":"2024-09-16T05:53:06.368302+0000","last_peered":"2024-09-16T05:53:06.368302+0000","last_clean":"2024-09-16T05:53:06.368302+0000","last_became_active":"2024-09-16T05:39:26.049092+0000","last_became_peered":"2024-09-16T05:39:26.049092+0000","last_unstale":"2024-09-16T05:53:06.368302+0000","last_undegraded":"2024-09-16T05:53:06.368302+0000","last_fullsized":"2024-09-16T05:53:06.368302+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:29.953136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":226,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:57.376805+0000","last_change":"2024-09-16T05:39:26.057943+0000","last_active":"2024-09-16T05:52:57.376805+0000","last_peered":"2024-09-16T05:52:57.376805+0000","last_clean":"2024-09-16T05:52:57.376805+0000","last_became_active":"2024-09-16T05:39:26.057564+0000","last_became_peered":"2024-09-16T05:39:26.057564+0000","last_unstale":"2024-09-16T05:52:57.376805+0000","last_undegraded":"2024-09-16T05:52:57.376805+0000","last_fullsized":"2024-09-16T05:52:57.376805+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:01:19.835552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356313+0000","last_change":"2024-09-16T05:39:26.060420+0000","last_active":"2024-09-16T05:51:56.356313+0000","last_peered":"2024-09-16T05:51:56.356313+0000","last_clean":"2024-09-16T05:51:56.356313+0000","last_became_active":"2024-09-16T05:39:26.059546+0000","last_became_peered":"2024-09-16T05:39:26.059546+0000","last_unstale":"2024-09-16T05:51:56.356313+0000","last_undegraded":"2024-09-16T05:51:56.356313+0000","last_fullsized":"2024-09-16T05:51:56.356313+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:24.267557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356350+0000","last_change":"2024-09-16T05:39:26.074051+0000","last_active":"2024-09-16T05:51:56.356350+0000","last_peered":"2024-09-16T05:51:56.356350+0000","last_clean":"2024-09-16T05:51:56.356350+0000","last_became_active":"2024-09-16T05:39:26.073016+0000","last_became_peered":"2024-09-16T05:39:26.073016+0000","last_unstale":"2024-09-16T05:51:56.356350+0000","last_undegraded":"2024-09-16T05:51:56.356350+0000","last_fullsized":"2024-09-16T05:51:56.356350+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:30.421445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":289,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.485788+0000","last_change":"2024-09-16T05:39:26.077800+0000","last_active":"2024-09-16T05:52:56.485788+0000","last_peered":"2024-09-16T05:52:56.485788+0000","last_clean":"2024-09-16T05:52:56.485788+0000","last_became_active":"2024-09-16T05:39:26.076818+0000","last_became_peered":"2024-09-16T05:39:26.076818+0000","last_unstale":"2024-09-16T05:52:56.485788+0000","last_undegraded":"2024-09-16T05:52:56.485788+0000","last_fullsized":"2024-09-16T05:52:56.485788+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:13:30.887596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":184,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.472790+0000","last_change":"2024-09-16T05:39:27.051741+0000","last_active":"2024-09-16T05:51:56.472790+0000","last_peered":"2024-09-16T05:51:56.472790+0000","last_clean":"2024-09-16T05:51:56.472790+0000","last_became_active":"2024-09-16T05:39:27.050732+0000","last_became_peered":"2024-09-16T05:39:27.050732+0000","last_unstale":"2024-09-16T05:51:56.472790+0000","last_undegraded":"2024-09-16T05:51:56.472790+0000","last_fullsized":"2024-09-16T05:51:56.472790+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:18.147347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:31.369313+0000","last_change":"2024-09-16T05:39:26.053064+0000","last_active":"2024-09-16T05:53:31.369313+0000","last_peered":"2024-09-16T05:53:31.369313+0000","last_clean":"2024-09-16T05:53:31.369313+0000","last_became_active":"2024-09-16T05:39:26.052209+0000","last_became_peered":"2024-09-16T05:39:26.052209+0000","last_unstale":"2024-09-16T05:53:31.369313+0000","last_undegraded":"2024-09-16T05:53:31.369313+0000","last_fullsized":"2024-09-16T05:53:31.369313+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:24.007866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":591,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.366813+0000","last_change":"2024-09-16T05:39:26.061773+0000","last_active":"2024-09-16T05:52:56.366813+0000","last_peered":"2024-09-16T05:52:56.366813+0000","last_clean":"2024-09-16T05:52:56.366813+0000","last_became_active":"2024-09-16T05:39:26.061101+0000","last_became_peered":"2024-09-16T05:39:26.061101+0000","last_unstale":"2024-09-16T05:52:56.366813+0000","last_undegraded":"2024-09-16T05:52:56.366813+0000","last_fullsized":"2024-09-16T05:52:56.366813+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_clean_scrub_stamp":"2024-09-16T05:34:47.359645+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:56:28.349578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":441,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:57.369715+0000","last_change":"2024-09-16T05:39:26.073172+0000","last_active":"2024-09-16T05:51:57.369715+0000","last_peered":"2024-09-16T05:51:57.369715+0000","last_clean":"2024-09-16T05:51:57.369715+0000","last_became_active":"2024-09-16T05:39:26.063463+0000","last_became_peered":"2024-09-16T05:39:26.063463+0000","last_unstale":"2024-09-16T05:51:57.369715+0000","last_undegraded":"2024-09-16T05:51:57.369715+0000","last_fullsized":"2024-09-16T05:51:57.369715+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:08.651405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:01.367585+0000","last_change":"2024-09-16T05:39:26.060982+0000","last_active":"2024-09-16T05:53:01.367585+0000","last_peered":"2024-09-16T05:53:01.367585+0000","last_clean":"2024-09-16T05:53:01.367585+0000","last_became_active":"2024-09-16T05:39:26.060550+0000","last_became_peered":"2024-09-16T05:39:26.060550+0000","last_unstale":"2024-09-16T05:53:01.367585+0000","last_undegraded":"2024-09-16T05:53:01.367585+0000","last_fullsized":"2024-09-16T05:53:01.367585+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:07:51.890578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.472754+0000","last_change":"2024-09-16T05:39:26.059723+0000","last_active":"2024-09-16T05:51:56.472754+0000","last_peered":"2024-09-16T05:51:56.472754+0000","last_clean":"2024-09-16T05:51:56.472754+0000","last_became_active":"2024-09-16T05:39:26.058771+0000","last_became_peered":"2024-09-16T05:39:26.058771+0000","last_unstale":"2024-09-16T05:51:56.472754+0000","last_undegraded":"2024-09-16T05:51:56.472754+0000","last_fullsized":"2024-09-16T05:51:56.472754+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:58:48.900055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":450,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:57.369673+0000","last_change":"2024-09-16T05:39:26.080898+0000","last_active":"2024-09-16T05:51:57.369673+0000","last_peered":"2024-09-16T05:51:57.369673+0000","last_clean":"2024-09-16T05:51:57.369673+0000","last_became_active":"2024-09-16T05:39:26.065589+0000","last_became_peered":"2024-09-16T05:39:26.065589+0000","last_unstale":"2024-09-16T05:51:57.369673+0000","last_undegraded":"2024-09-16T05:51:57.369673+0000","last_fullsized":"2024-09-16T05:51:57.369673+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:54.750367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":430,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:56.356395+0000","last_change":"2024-09-16T05:39:27.057353+0000","last_active":"2024-09-16T05:51:56.356395+0000","last_peered":"2024-09-16T05:51:56.356395+0000","last_clean":"2024-09-16T05:51:56.356395+0000","last_became_active":"2024-09-16T05:39:27.056262+0000","last_became_peered":"2024-09-16T05:39:27.056262+0000","last_unstale":"2024-09-16T05:51:56.356395+0000","last_undegraded":"2024-09-16T05:51:56.356395+0000","last_fullsized":"2024-09-16T05:51:56.356395+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:41.799365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:01.358717+0000","last_change":"2024-09-16T05:39:26.060534+0000","last_active":"2024-09-16T05:52:01.358717+0000","last_peered":"2024-09-16T05:52:01.358717+0000","last_clean":"2024-09-16T05:52:01.358717+0000","last_became_active":"2024-09-16T05:39:26.060029+0000","last_became_peered":"2024-09-16T05:39:26.060029+0000","last_unstale":"2024-09-16T05:52:01.358717+0000","last_undegraded":"2024-09-16T05:52:01.358717+0000","last_fullsized":"2024-09-16T05:52:01.358717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:47.905126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":171,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.360460+0000","last_change":"2024-09-16T05:39:26.055632+0000","last_active":"2024-09-16T05:52:26.360460+0000","last_peered":"2024-09-16T05:52:26.360460+0000","last_clean":"2024-09-16T05:52:26.360460+0000","last_became_active":"2024-09-16T05:39:26.054759+0000","last_became_peered":"2024-09-16T05:39:26.054759+0000","last_unstale":"2024-09-16T05:52:26.360460+0000","last_undegraded":"2024-09-16T05:52:26.360460+0000","last_fullsized":"2024-09-16T05:52:26.360460+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:14.840382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.362449+0000","last_change":"2024-09-16T05:39:26.051119+0000","last_active":"2024-09-16T05:52:26.362449+0000","last_peered":"2024-09-16T05:52:26.362449+0000","last_clean":"2024-09-16T05:52:26.362449+0000","last_became_active":"2024-09-16T05:39:26.050738+0000","last_became_peered":"2024-09-16T05:39:26.050738+0000","last_unstale":"2024-09-16T05:52:26.362449+0000","last_undegraded":"2024-09-16T05:52:26.362449+0000","last_fullsized":"2024-09-16T05:52:26.362449+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:37.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607807,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640509,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34328,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337512,"statfs":{"total":96636764160,"available":96601612288,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640509,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640508,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640511,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34328,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337512,"statfs":{"total":96636764160,"available":96601612288,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:55:38.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T05:55:38.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.831+0000 7fd7d7f8a640 10 monclient: get_monmap_and_config 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.831+0000 7fd7d7f8a640 10 monclient: build_initial_monmap 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.831+0000 7fd7d7f8a640 10 monclient: monmap: 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:38.838292+0000 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:38.838292+0000 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:38.837 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 Processor -- start 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- start start 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient: init 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient(hunting): picked mon.noname-c con 0x7fd7d006cd20 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient(hunting): picked mon.noname-b con 0x7fd7d00a3220 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7d013cb80 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d00a2fe0 con 0x7fd7d013cb80 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d006ecb0 con 0x7fd7d006cd20 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d0073a30 con 0x7fd7d00a3220 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient(hunting): authenticate will time out at 2200.645426s 2024-09-16T05:55:38.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d6527640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd7d013cb80 0x7fd7d013ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39286/0 (socket says 172.21.15.6:39286) 2024-09-16T05:55:38.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d6527640 1 -- 172.21.15.6:0/296338739 learned_addr learned my addr 172.21.15.6:0/296338739 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:38.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2912679327 0 0) 0x7fd7d0073a30 con 0x7fd7d00a3220 2024-09-16T05:55:38.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient(hunting): my global_id is 164483 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7bc003530 con 0x7fd7d00a3220 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 221463000 0 0) 0x7fd7d00a2fe0 con 0x7fd7d013cb80 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient(hunting): my global_id is 194157 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d0073a30 con 0x7fd7d013cb80 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1054852891 0 0) 0x7fd7bc003530 con 0x7fd7d00a3220 2024-09-16T05:55:38.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 >> v1:172.21.15.63:6789/0 conn(0x7fd7d006cd20 legacy=0x7fd7d006d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 >> v1:172.21.15.6:6789/0 conn(0x7fd7d013cb80 legacy=0x7fd7d013ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: _finish_hunting 0 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 monclient: found mon.noname-b 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d00a29f0 con 0x7fd7d00a3220 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: _finish_auth 0 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: _check_auth_tickets 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd7c0001620 con 0x7fd7d00a3220 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:38.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: dump: 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 5 monclient: authenticate success, global_id 164483 2024-09-16T05:55:38.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7c0001880 con 0x7fd7d00a3220 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 1 -- 172.21.15.6:0/296338739 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd7c0001c90 con 0x7fd7d00a3220 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d4d24640 10 monclient: dump: 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.849 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 4 set_mon_vals no callback set 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient: shutdown 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/296338739 >> v1:172.21.15.6:6790/0 conn(0x7fd7d00a3220 legacy=0x7fd7d006d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/296338739 shutdown_connections 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/296338739 >> 172.21.15.6:0/296338739 conn(0x7fd7d00a3880 msgr2=0x7fd7d00a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/296338739 shutdown_connections 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/296338739 wait complete. 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient: build_initial_monmap 2024-09-16T05:55:38.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.835+0000 7fd7d7f8a640 10 monclient: monmap: 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:38.851 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 1 Processor -- start 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 1 -- start start 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: init 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient(hunting): picked mon.noname-b con 0x7fd7d006cd20 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7d00a3220 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient(hunting): picked mon.noname-c con 0x7fd7d013cb80 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d0138ac0 con 0x7fd7d00a3220 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d01cbfb0 con 0x7fd7d006cd20 2024-09-16T05:55:38.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d0137e80 con 0x7fd7d013cb80 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient(hunting): authenticate will time out at 2200.648608s 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d5525640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd7d00a3220 0x7fd7d01cb0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39300/0 (socket says 172.21.15.6:39300) 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d6527640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd7d013cb80 0x7fd7d01d3400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:39604/0 (socket says 172.21.15.6:39604) 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d5525640 1 -- 172.21.15.6:0/3972316929 learned_addr learned my addr 172.21.15.6:0/3972316929 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1971803153 0 0) 0x7fd7d0138ac0 con 0x7fd7d00a3220 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): my global_id is 194160 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7b4003530 con 0x7fd7d00a3220 2024-09-16T05:55:38.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1155263085 0 0) 0x7fd7d0137e80 con 0x7fd7d013cb80 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): my global_id is 164486 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d0138ac0 con 0x7fd7d013cb80 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 950729606 0 0) 0x7fd7b4003530 con 0x7fd7d00a3220 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7d0137e80 con 0x7fd7d00a3220 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd7c0001c10 con 0x7fd7d00a3220 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): dump: 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.854 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3419944742 0 0) 0x7fd7d0138ac0 con 0x7fd7d013cb80 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7b4003530 con 0x7fd7d013cb80 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd7c8001b80 con 0x7fd7d013cb80 2024-09-16T05:55:38.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): dump: 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3631166400 0 0) 0x7fd7d01cbfb0 con 0x7fd7d006cd20 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient(hunting): my global_id is 184261 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7d0138ac0 con 0x7fd7d006cd20 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3946951606 0 0) 0x7fd7d0137e80 con 0x7fd7d00a3220 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 >> v1:172.21.15.6:6790/0 conn(0x7fd7d013cb80 legacy=0x7fd7d01d3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 >> v1:172.21.15.63:6789/0 conn(0x7fd7d006cd20 legacy=0x7fd7d013b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient: _finish_hunting 0 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 monclient: found mon.a 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d01387e0 con 0x7fd7d00a3220 2024-09-16T05:55:38.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient: _finish_auth 0 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient: _check_auth_tickets 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 5 monclient: authenticate success, global_id 194160 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: _renew_subs 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d013b9f0 con 0x7fd7d00a3220 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: _renew_subs 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7d01cb7d0 con 0x7fd7d00a3220 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7c0001880 con 0x7fd7d00a3220 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:38.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd7c0003b90 con 0x7fd7d00a3220 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c7fff640 4 set_mon_vals no callback set 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 10 monclient: dump: 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:38.860 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b8005090 con 0x7fd7d00a3220 2024-09-16T05:55:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd7c0003d30 con 0x7fd7d00a3220 2024-09-16T05:55:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fd7c0044ec0 con 0x7fd7d00a3220 2024-09-16T05:55:38.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.839+0000 7fd7d5d26640 10 monclient: get_auth_request con 0x7fd7b4036bd0 auth_method 0 2024-09-16T05:55:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.843+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd7c00154f0 con 0x7fd7d00a3220 2024-09-16T05:55:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.843+0000 7fd7c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:38.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:38.843+0000 7fd7c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:39.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.011+0000 7fd7d7f8a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:55:39.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.011+0000 7fd7d7f8a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:55:39.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.011+0000 7fd7d7f8a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:39.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.011+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7b8005380 con 0x7fd7d00a3220 2024-09-16T05:55:39.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7c67fc640 1 -- 172.21.15.6:0/3972316929 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) ==== 74+0+20479 (unknown 1421847654 0 341976569) 0x7fd7c001d9a0 con 0x7fd7d00a3220 2024-09-16T05:55:39.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T05:55:39.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:55:39.019 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:55:39.019 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":259,"fsid":"22154726-d7f4-4d5e-b601-06ee1f3e7219","created":"2024-09-16T05:34:36.231874+0000","modified":"2024-09-16T05:41:10.457214+0000","last_up_change":"2024-09-16T05:34:46.347254+0000","last_in_change":"2024-09-16T05:38:02.369094+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":182,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T05:34:47.340654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T05:34:53.191455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-16T05:38:16.065675+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"barset","create_time":"2024-09-16T05:38:16.993254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"28573ac1-022a-424e-a024-a8ae14680604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":3069057739}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":3069057739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":3069057739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":3069057739}]},"public_addr":"172.21.15.6:6804/3069057739","cluster_addr":"172.21.15.6:6805/3069057739","heartbeat_back_addr":"172.21.15.6:6807/3069057739","heartbeat_front_addr":"172.21.15.6:6806/3069057739","state":["exists","up"]},{"osd":1,"uuid":"5c1e8581-2bb5-4ed6-a7df-a14ad2331b0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":4098222447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":4098222447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":4098222447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":4098222447}]},"public_addr":"172.21.15.6:6808/4098222447","cluster_addr":"172.21.15.6:6809/4098222447","heartbeat_back_addr":"172.21.15.6:6811/4098222447","heartbeat_front_addr":"172.21.15.6:6810/4098222447","state":["exists","up"]},{"osd":2,"uuid":"89377298-ce68-4fd4-bb63-2ffd60eeea3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":1572197727}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":1572197727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":1572197727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":1572197727}]},"public_addr":"172.21.15.6:6800/1572197727","cluster_addr":"172.21.15.6:6801/1572197727","heartbeat_back_addr":"172.21.15.6:6803/1572197727","heartbeat_front_addr":"172.21.15.6:6802/1572197727","state":["exists","up"]},{"osd":3,"uuid":"7c5624a2-d552-4295-b112-d87972274f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6800","nonce":709309215}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6801","nonce":709309215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6803","nonce":709309215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6802","nonce":709309215}]},"public_addr":"172.21.15.63:6800/709309215","cluster_addr":"172.21.15.63:6801/709309215","heartbeat_back_addr":"172.21.15.63:6803/709309215","heartbeat_front_addr":"172.21.15.63:6802/709309215","state":["exists","up"]},{"osd":4,"uuid":"52fa74b0-2bf6-4c82-acb0-4032eca1f08b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6804","nonce":118134224}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6805","nonce":118134224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6807","nonce":118134224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6806","nonce":118134224}]},"public_addr":"172.21.15.63:6804/118134224","cluster_addr":"172.21.15.63:6805/118134224","heartbeat_back_addr":"172.21.15.63:6807/118134224","heartbeat_front_addr":"172.21.15.63:6806/118134224","state":["exists","up"]},{"osd":5,"uuid":"a4dac512-ff9b-4381-9065-82bf9a711bdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6808","nonce":165284194}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6809","nonce":165284194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6811","nonce":165284194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.63:6810","nonce":165284194}]},"public_addr":"172.21.15.63:6808/165284194","cluster_addr":"172.21.15.63:6809/165284194","heartbeat_back_addr":"172.21.15.63:6811/165284194","heartbeat_front_addr":"172.21.15.63:6810/165284194","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"5a2f0584-0de7-4238-aba4-c51878b788e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.065166+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.182281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.088394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.071475+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.333358+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T05:34:44.397991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T05:55:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd7b4036bd0 legacy=0x7fd7b4039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7d7f8a640 10 monclient: shutdown 2024-09-16T05:55:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7d7f8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 >> v1:172.21.15.6:6789/0 conn(0x7fd7d00a3220 legacy=0x7fd7d01cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 shutdown_connections 2024-09-16T05:55:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 >> 172.21.15.6:0/3972316929 conn(0x7fd7d00a3880 msgr2=0x7fd7d013bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:39.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 shutdown_connections 2024-09-16T05:55:39.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:39.015+0000 7fd7d7f8a640 1 -- 172.21.15.6:0/3972316929 wait complete. 2024-09-16T05:55:40.038 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T05:55:40.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:55:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient: get_monmap_and_config 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient: build_initial_monmap 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient: monmap: 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:40.180009+0000 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:40.180009+0000 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:40.180 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 1 Processor -- start 2024-09-16T05:55:40.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 1 -- start start 2024-09-16T05:55:40.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient: init 2024-09-16T05:55:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:55:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0dc136a20 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient(hunting): picked mon.noname-b con 0x7fe0dc139900 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient(hunting): picked mon.noname-c con 0x7fe0dc13d410 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0dc137510 con 0x7fe0dc136a20 2024-09-16T05:55:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0dc0a4cf0 con 0x7fe0dc13d410 2024-09-16T05:55:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0dc0a4a70 con 0x7fe0dc139900 2024-09-16T05:55:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient(hunting): authenticate will time out at 2201.986981s 2024-09-16T05:55:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e1308640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe0dc136a20 0x7fe0dc136e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40736/0 (socket says 172.21.15.6:40736) 2024-09-16T05:55:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e1308640 1 -- 172.21.15.6:0/3595637976 learned_addr learned my addr 172.21.15.6:0/3595637976 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e0b07640 1 --1- 172.21.15.6:0/3595637976 >> v1:172.21.15.6:6790/0 conn(0x7fe0dc139900 0x7fe0dc13bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52818/0 (socket says 172.21.15.6:52818) 2024-09-16T05:55:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895091039 0 0) 0x7fe0dc0a4a70 con 0x7fe0dc139900 2024-09-16T05:55:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient(hunting): my global_id is 164489 2024-09-16T05:55:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c8003530 con 0x7fe0dc139900 2024-09-16T05:55:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081591361 0 0) 0x7fe0dc137510 con 0x7fe0dc136a20 2024-09-16T05:55:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient(hunting): my global_id is 194163 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0dc0a4a70 con 0x7fe0dc136a20 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1151186446 0 0) 0x7fe0c8003530 con 0x7fe0dc139900 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 >> v1:172.21.15.63:6789/0 conn(0x7fe0dc13d410 legacy=0x7fe0dc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 >> v1:172.21.15.6:6789/0 conn(0x7fe0dc136a20 legacy=0x7fe0dc136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: _finish_hunting 0 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 monclient: found mon.noname-b 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0dc0a07a0 con 0x7fe0dc139900 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: _finish_auth 0 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: _check_auth_tickets 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0d0001620 con 0x7fe0dc139900 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: dump: 2024-09-16T05:55:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 5 monclient: authenticate success, global_id 164489 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0d0001880 con 0x7fe0dc139900 2024-09-16T05:55:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 1 -- 172.21.15.6:0/3595637976 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0d0001bd0 con 0x7fe0dc139900 2024-09-16T05:55:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0e356c640 4 set_mon_vals no callback set 2024-09-16T05:55:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.175+0000 7fe0cffff640 10 monclient: dump: 2024-09-16T05:55:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.196 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: shutdown 2024-09-16T05:55:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- 172.21.15.6:0/3595637976 >> v1:172.21.15.6:6790/0 conn(0x7fe0dc139900 legacy=0x7fe0dc13bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- 172.21.15.6:0/3595637976 shutdown_connections 2024-09-16T05:55:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- 172.21.15.6:0/3595637976 >> 172.21.15.6:0/3595637976 conn(0x7fe0dc0a5a10 msgr2=0x7fe0dc0a5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- 172.21.15.6:0/3595637976 shutdown_connections 2024-09-16T05:55:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- 172.21.15.6:0/3595637976 wait complete. 2024-09-16T05:55:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: build_initial_monmap 2024-09-16T05:55:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: monmap: 2024-09-16T05:55:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 Processor -- start 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- start start 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: init 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient(hunting): picked mon.noname-c con 0x7fe0dc136a20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:40.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0dc139900 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient(hunting): picked mon.noname-b con 0x7fe0dc13d410 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0dc120de0 con 0x7fe0dc139900 2024-09-16T05:55:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0dc09f420 con 0x7fe0dc13d410 2024-09-16T05:55:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0dc0a7a90 con 0x7fe0dc136a20 2024-09-16T05:55:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient(hunting): authenticate will time out at 2201.990069s 2024-09-16T05:55:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e1308640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe0dc136a20 0x7fe0dc135380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52830/0 (socket says 172.21.15.6:52830) 2024-09-16T05:55:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e1308640 1 -- 172.21.15.6:0/2282646053 learned_addr learned my addr 172.21.15.6:0/2282646053 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883073514 0 0) 0x7fe0dc0a7a90 con 0x7fe0dc136a20 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): my global_id is 164492 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c0003530 con 0x7fe0dc136a20 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4185622607 0 0) 0x7fe0dc120de0 con 0x7fe0dc139900 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): my global_id is 194166 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0dc0a7a90 con 0x7fe0dc139900 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3405323323 0 0) 0x7fe0dc0a7a90 con 0x7fe0dc139900 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0dc120de0 con 0x7fe0dc139900 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3747289352 0 0) 0x7fe0c0003530 con 0x7fe0dc136a20 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0dc0a7a90 con 0x7fe0dc136a20 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0d0001520 con 0x7fe0dc139900 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): dump: 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.203 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3547795804 0 0) 0x7fe0dc09f420 con 0x7fe0dc13d410 2024-09-16T05:55:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): my global_id is 184267 2024-09-16T05:55:40.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c0003530 con 0x7fe0dc13d410 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0d80018e0 con 0x7fe0dc136a20 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient(hunting): dump: 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.205 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.206 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1930702080 0 0) 0x7fe0dc120de0 con 0x7fe0dc139900 2024-09-16T05:55:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 >> v1:172.21.15.6:6790/0 conn(0x7fe0dc136a20 legacy=0x7fe0dc135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 >> v1:172.21.15.63:6789/0 conn(0x7fe0dc13d410 legacy=0x7fe0dc1d4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient: _finish_hunting 0 2024-09-16T05:55:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 monclient: found mon.a 2024-09-16T05:55:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0dc0a74d0 con 0x7fe0dc139900 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient: _finish_auth 0 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient: _check_auth_tickets 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 5 monclient: authenticate success, global_id 194166 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: _renew_subs 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0dc0a0100 con 0x7fe0dc139900 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: _renew_subs 2024-09-16T05:55:40.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0dc1364c0 con 0x7fe0dc139900 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0d0004aa0 con 0x7fe0dc139900 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cf7fe640 4 set_mon_vals no callback set 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe0d0004cd0 con 0x7fe0dc139900 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 10 monclient: dump: 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.209 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.179+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe0d0004ef0 con 0x7fe0dc139900 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.183+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fe0d0044f30 con 0x7fe0dc139900 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.183+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6804/3069057739 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0bc0053b0 con 0x7fe0bc001640 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.183+0000 7fe0e1b09640 10 monclient: get_auth_request con 0x7fe0bc001640 auth_method 0 2024-09-16T05:55:40.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.183+0000 7fe0e1308640 10 monclient: get_auth_request con 0x7fe0c0036cc0 auth_method 0 2024-09-16T05:55:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.183+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== osd.0 v1:172.21.15.6:6804/3069057739 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe0bc0053b0 con 0x7fe0bc001640 2024-09-16T05:55:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 --> v1:172.21.15.6:6804/3069057739 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe0bc0070a0 con 0x7fe0bc001640 2024-09-16T05:55:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0cdffb640 1 -- 172.21.15.6:0/2282646053 <== osd.0 v1:172.21.15.6:6804/3069057739 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fe0bc0070a0 con 0x7fe0bc001640 2024-09-16T05:55:40.212 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:40.212 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:55:40.212 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:40.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 >> v1:172.21.15.6:6804/3069057739 conn(0x7fe0bc001640 legacy=0x7fe0bc003b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe0c0036cc0 legacy=0x7fe0c0039180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 10 monclient: shutdown 2024-09-16T05:55:40.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:40.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 >> v1:172.21.15.6:6789/0 conn(0x7fe0dc139900 legacy=0x7fe0dc1d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e1b09640 1 -- 172.21.15.6:0/2282646053 reap_dead start 2024-09-16T05:55:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 shutdown_connections 2024-09-16T05:55:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 >> 172.21.15.6:0/2282646053 conn(0x7fe0dc0a5a10 msgr2=0x7fe0dc1374a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 shutdown_connections 2024-09-16T05:55:40.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.195+0000 7fe0e356c640 1 -- 172.21.15.6:0/2282646053 wait complete. 2024-09-16T05:55:40.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient: get_monmap_and_config 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient: build_initial_monmap 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient: monmap: 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:40.335481+0000 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:40.335481+0000 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.335 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:40.335 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:40.335 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:40.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 1 Processor -- start 2024-09-16T05:55:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 1 -- start start 2024-09-16T05:55:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient: init 2024-09-16T05:55:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:55:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient(hunting): picked mon.noname-a con 0x7f8f4009bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient(hunting): picked mon.noname-b con 0x7f8f4009bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient(hunting): picked mon.noname-c con 0x7f8f401397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f400a1350 con 0x7f8f4009bb40 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f400a10d0 con 0x7f8f401397f0 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f4009d940 con 0x7f8f4009bf20 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient(hunting): authenticate will time out at 2202.142811s 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4658e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8f4009bb40 0x7f8f4009e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40760/0 (socket says 172.21.15.6:40760) 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4658e640 1 -- 172.21.15.6:0/65814851 learned_addr learned my addr 172.21.15.6:0/65814851 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 289045598 0 0) 0x7f8f400a1350 con 0x7f8f4009bb40 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): my global_id is 194169 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f2c005740 con 0x7f8f4009bb40 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023969063 0 0) 0x7f8f4009d940 con 0x7f8f4009bf20 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): my global_id is 164495 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f400a1350 con 0x7f8f4009bf20 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 938222183 0 0) 0x7f8f400a10d0 con 0x7f8f401397f0 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient(hunting): my global_id is 184270 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f4009d940 con 0x7f8f401397f0 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3583488786 0 0) 0x7f8f2c005740 con 0x7f8f4009bb40 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 >> v1:172.21.15.6:6790/0 conn(0x7f8f4009bf20 legacy=0x7f8f4009c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 >> v1:172.21.15.63:6789/0 conn(0x7f8f401397f0 legacy=0x7f8f4013bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: _finish_hunting 0 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 monclient: found mon.noname-a 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4009dc20 con 0x7f8f4009bb40 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: _finish_auth 0 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: _check_auth_tickets 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:40.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8f3c001620 con 0x7f8f4009bb40 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 5 monclient: authenticate success, global_id 194169 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: dump: 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.342 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f3c0018d0 con 0x7f8f4009bb40 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 1 -- 172.21.15.6:0/65814851 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8f3c001ce0 con 0x7f8f4009bb40 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f4558c640 10 monclient: dump: 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.343 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.331+0000 7f8f487f2640 4 set_mon_vals no callback set 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: shutdown 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- 172.21.15.6:0/65814851 >> v1:172.21.15.6:6789/0 conn(0x7f8f4009bb40 legacy=0x7f8f4009e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- 172.21.15.6:0/65814851 shutdown_connections 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- 172.21.15.6:0/65814851 >> 172.21.15.6:0/65814851 conn(0x7f8f400a1650 msgr2=0x7f8f400a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- 172.21.15.6:0/65814851 shutdown_connections 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- 172.21.15.6:0/65814851 wait complete. 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: build_initial_monmap 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: monmap: 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.345 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:40.346 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 Processor -- start 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- start start 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: init 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient(hunting): picked mon.noname-a con 0x7f8f4009bb40 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient(hunting): picked mon.noname-c con 0x7f8f4009bf20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient(hunting): picked mon.noname-b con 0x7f8f401397f0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f40071ef0 con 0x7f8f4009bb40 2024-09-16T05:55:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f400740c0 con 0x7f8f401397f0 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f40071c10 con 0x7f8f4009bf20 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient(hunting): authenticate will time out at 2202.146082s 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f45d8d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8f4009bf20 0x7f8f401cbe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52852/0 (socket says 172.21.15.6:52852) 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f45d8d640 1 -- 172.21.15.6:0/394841642 learned_addr learned my addr 172.21.15.6:0/394841642 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f4658e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8f4009bb40 0x7f8f4012f730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40772/0 (socket says 172.21.15.6:40772) 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3733960409 0 0) 0x7f8f40071c10 con 0x7f8f4009bf20 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): my global_id is 164498 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f24003530 con 0x7f8f4009bf20 2024-09-16T05:55:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2770470588 0 0) 0x7f8f40071ef0 con 0x7f8f4009bb40 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): my global_id is 194172 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f40071c10 con 0x7f8f4009bb40 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4099458387 0 0) 0x7f8f40071c10 con 0x7f8f4009bb40 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f40071ef0 con 0x7f8f4009bb40 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8f3c001a80 con 0x7f8f4009bb40 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): dump: 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2783685517 0 0) 0x7f8f24003530 con 0x7f8f4009bf20 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f40071c10 con 0x7f8f4009bf20 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8f30001a10 con 0x7f8f4009bf20 2024-09-16T05:55:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): dump: 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.353 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2002131466 0 0) 0x7f8f400740c0 con 0x7f8f401397f0 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient(hunting): my global_id is 184273 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f24003530 con 0x7f8f401397f0 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3673038728 0 0) 0x7f8f40071ef0 con 0x7f8f4009bb40 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 >> v1:172.21.15.6:6790/0 conn(0x7f8f4009bf20 legacy=0x7f8f401cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 >> v1:172.21.15.63:6789/0 conn(0x7f8f401397f0 legacy=0x7f8f401cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 monclient: found mon.a 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f400704e0 con 0x7f8f4009bb40 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient: _finish_auth 0 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 5 monclient: authenticate success, global_id 194172 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: _renew_subs 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8f400a3950 con 0x7f8f4009bb40 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: _renew_subs 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8f4012fe40 con 0x7f8f4009bb40 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f3c0018d0 con 0x7f8f4009bb40 2024-09-16T05:55:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8f3c004de0 con 0x7f8f4009bb40 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f44d8b640 4 set_mon_vals no callback set 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.335+0000 7f8f36ffd640 10 monclient: dump: 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.339+0000 7f8f487f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.339+0000 7f8f487f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.339+0000 7f8f487f2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.339+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f28005090 con 0x7f8f4009bb40 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.339+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8f3c00fb80 con 0x7f8f4009bb40 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.339+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f8f3c044df0 con 0x7f8f4009bb40 2024-09-16T05:55:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.339+0000 7f8f45d8d640 10 monclient: get_auth_request con 0x7f8f24036e10 auth_method 0 2024-09-16T05:55:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.343+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8f3c019090 con 0x7f8f4009bb40 2024-09-16T05:55:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.343+0000 7f8f36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.343+0000 7f8f36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:40.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.507+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f8f28000ba0 con 0x7f8f24036e10 2024-09-16T05:55:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.507+0000 7f8f36ffd640 1 -- 172.21.15.6:0/394841642 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f8f28000ba0 con 0x7f8f24036e10 2024-09-16T05:55:40.512 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T05:55:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.511+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8f24036e10 legacy=0x7f8f240392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.511+0000 7f8f487f2640 10 monclient: shutdown 2024-09-16T05:55:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.511+0000 7f8f487f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.511+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 >> v1:172.21.15.6:6789/0 conn(0x7f8f4009bb40 legacy=0x7f8f4012f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.511+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 shutdown_connections 2024-09-16T05:55:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.511+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 >> 172.21.15.6:0/394841642 conn(0x7f8f400a1650 msgr2=0x7f8f401325b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.511+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 shutdown_connections 2024-09-16T05:55:40.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.511+0000 7f8f487f2640 1 -- 172.21.15.6:0/394841642 wait complete. 2024-09-16T05:55:40.531 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T05:55:40.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:55:40.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient: get_monmap_and_config 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient: build_initial_monmap 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient: monmap: 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:40.656291+0000 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:40.656291+0000 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:40.655 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:40.656 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 1 Processor -- start 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 1 -- start start 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient: init 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf4136a20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf4139900 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf413d410 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf4137510 con 0x7f3cf4139900 2024-09-16T05:55:40.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf40a4cf0 con 0x7f3cf413d410 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf40a4a70 con 0x7f3cf4136a20 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cfbccf640 10 monclient(hunting): authenticate will time out at 2202.463570s 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cf9a6b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3cf4136a20 0x7f3cf4136e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52856/0 (socket says 172.21.15.6:52856) 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.651+0000 7f3cf9a6b640 1 -- 172.21.15.6:0/3412843793 learned_addr learned my addr 172.21.15.6:0/3412843793 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241544110 0 0) 0x7f3cf40a4a70 con 0x7f3cf4136a20 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): my global_id is 164501 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ce0003530 con 0x7f3cf4136a20 2024-09-16T05:55:40.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2089637167 0 0) 0x7f3cf4137510 con 0x7f3cf4139900 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): my global_id is 194175 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf40a4a70 con 0x7f3cf4139900 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531550622 0 0) 0x7f3cf40a4cf0 con 0x7f3cf413d410 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient(hunting): my global_id is 184276 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf4137510 con 0x7f3cf413d410 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 767391617 0 0) 0x7f3cf40a4a70 con 0x7f3cf4139900 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 >> v1:172.21.15.6:6790/0 conn(0x7f3cf4136a20 legacy=0x7f3cf4136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 >> v1:172.21.15.63:6789/0 conn(0x7f3cf413d410 legacy=0x7f3cf413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: _finish_hunting 0 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 monclient: found mon.noname-a 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf40a07a0 con 0x7f3cf4139900 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: _finish_auth 0 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: _check_auth_tickets 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3ce4001600 con 0x7f3cf4139900 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: dump: 2024-09-16T05:55:40.663 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 5 monclient: authenticate success, global_id 194175 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ce4001860 con 0x7f3cf4139900 2024-09-16T05:55:40.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 1 -- 172.21.15.6:0/3412843793 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3ce4001c30 con 0x7f3cf4139900 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 4 set_mon_vals no callback set 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf8a69640 10 monclient: dump: 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.665 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: shutdown 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3412843793 >> v1:172.21.15.6:6789/0 conn(0x7f3cf4139900 legacy=0x7f3cf413bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3412843793 shutdown_connections 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3412843793 >> 172.21.15.6:0/3412843793 conn(0x7f3cf40a5a10 msgr2=0x7f3cf40a5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3412843793 shutdown_connections 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3412843793 wait complete. 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: build_initial_monmap 2024-09-16T05:55:40.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: monmap: 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 Processor -- start 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- start start 2024-09-16T05:55:40.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: init 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf4136a20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf4139900 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf413d410 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf41208c0 con 0x7f3cf4139900 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf4120ba0 con 0x7f3cf413d410 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf409f630 con 0x7f3cf4136a20 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient(hunting): authenticate will time out at 2202.466850s 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf9a6b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3cf4136a20 0x7f3cf41cb7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52858/0 (socket says 172.21.15.6:52858) 2024-09-16T05:55:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cf9a6b640 1 -- 172.21.15.6:0/3792975185 learned_addr learned my addr 172.21.15.6:0/3792975185 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:40.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594083629 0 0) 0x7f3cf409f630 con 0x7f3cf4136a20 2024-09-16T05:55:40.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): my global_id is 164504 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd8003530 con 0x7f3cf4136a20 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2902444847 0 0) 0x7f3cf41208c0 con 0x7f3cf4139900 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): my global_id is 194178 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf409f630 con 0x7f3cf4139900 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4063966613 0 0) 0x7f3cd8003530 con 0x7f3cf4136a20 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cf41208c0 con 0x7f3cf4136a20 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3cf0001a40 con 0x7f3cf4136a20 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:40.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): dump: 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:40.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3400337499 0 0) 0x7f3cf409f630 con 0x7f3cf4139900 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cd8003530 con 0x7f3cf4139900 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3ce4001500 con 0x7f3cf4139900 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): dump: 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.674 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586513169 0 0) 0x7f3cf4120ba0 con 0x7f3cf413d410 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient(hunting): my global_id is 184279 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf409f630 con 0x7f3cf413d410 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2769214598 0 0) 0x7f3cf41208c0 con 0x7f3cf4136a20 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 >> v1:172.21.15.63:6789/0 conn(0x7f3cf413d410 legacy=0x7f3cf41d3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 >> v1:172.21.15.6:6789/0 conn(0x7f3cf4139900 legacy=0x7f3cf41d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 monclient: found mon.c 2024-09-16T05:55:40.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf40a6e10 con 0x7f3cf4136a20 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient: _finish_auth 0 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 5 monclient: authenticate success, global_id 164504 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: _renew_subs 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf413cea0 con 0x7f3cf4136a20 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: _renew_subs 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3cf41205e0 con 0x7f3cf4136a20 2024-09-16T05:55:40.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.655+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3cf0001be0 con 0x7f3cf4136a20 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f3cf0004c40 con 0x7f3cf4136a20 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 10 monclient: dump: 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.677 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cebfff640 4 set_mon_vals no callback set 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f3cf000f9e0 con 0x7f3cf4136a20 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cfbccf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f3cdc000f90 con 0x7f3cf4136a20 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f3cf0044e40 con 0x7f3cf4136a20 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6808/4098222447 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3cd80446c0 con 0x7f3cd8040a90 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (unknown 1031510311 0 0) 0x7f3cf00452a0 con 0x7f3cf4136a20 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cfa26c640 10 monclient: get_auth_request con 0x7f3cd8040a90 auth_method 0 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cf926a640 10 monclient: get_auth_request con 0x7f3cd8036c40 auth_method 0 2024-09-16T05:55:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.659+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== osd.1 v1:172.21.15.6:6808/4098222447 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3cd80446c0 con 0x7f3cd8040a90 2024-09-16T05:55:40.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.671+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 --> v1:172.21.15.6:6808/4098222447 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3cdc002cc0 con 0x7f3cd8040a90 2024-09-16T05:55:40.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.671+0000 7f3cea7fc640 1 -- 172.21.15.6:0/3792975185 <== osd.1 v1:172.21.15.6:6808/4098222447 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f3cdc002cc0 con 0x7f3cd8040a90 2024-09-16T05:55:40.679 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:40.679 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:55:40.679 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.671+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 >> v1:172.21.15.6:6808/4098222447 conn(0x7f3cd8040a90 legacy=0x7f3cd8042ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.671+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 >> v1:172.21.15.63:6812/3643207211 conn(0x7f3cd8036c40 legacy=0x7f3cd8039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.671+0000 7f3cfbccf640 10 monclient: shutdown 2024-09-16T05:55:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.671+0000 7f3cfbccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.671+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 >> v1:172.21.15.6:6790/0 conn(0x7f3cf4136a20 legacy=0x7f3cf41cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.675+0000 7f3cfa26c640 1 -- 172.21.15.6:0/3792975185 reap_dead start 2024-09-16T05:55:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.675+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 shutdown_connections 2024-09-16T05:55:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.675+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 >> 172.21.15.6:0/3792975185 conn(0x7f3cf40a5a10 msgr2=0x7f3cf40a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.675+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 shutdown_connections 2024-09-16T05:55:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.675+0000 7f3cfbccf640 1 -- 172.21.15.6:0/3792975185 wait complete. 2024-09-16T05:55:40.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T05:55:40.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient: get_monmap_and_config 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient: build_initial_monmap 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient: monmap: 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:40.811179+0000 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:40.811179+0000 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:40.810 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 Processor -- start 2024-09-16T05:55:40.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- start start 2024-09-16T05:55:40.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient: init 2024-09-16T05:55:40.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:40.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:40.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient(hunting): picked mon.noname-b con 0x7fab8406cd20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient(hunting): picked mon.noname-a con 0x7fab840a3220 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient(hunting): picked mon.noname-c con 0x7fab8413cb80 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab840a2fe0 con 0x7fab840a3220 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab8406ecb0 con 0x7fab8413cb80 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab84073a30 con 0x7fab8406cd20 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient(hunting): authenticate will time out at 2202.617932s 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab7affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fab840a3220 0x7fab8406d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40806/0 (socket says 172.21.15.6:40806) 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab7affd640 1 -- 172.21.15.6:0/1149176531 learned_addr learned my addr 172.21.15.6:0/1149176531 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab837fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fab8406cd20 0x7fab8406d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52860/0 (socket says 172.21.15.6:52860) 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2281694785 0 0) 0x7fab840a2fe0 con 0x7fab840a3220 2024-09-16T05:55:40.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): my global_id is 194181 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab70003530 con 0x7fab840a3220 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2939967523 0 0) 0x7fab84073a30 con 0x7fab8406cd20 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): my global_id is 164507 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab840a2fe0 con 0x7fab8406cd20 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708393880 0 0) 0x7fab8406ecb0 con 0x7fab8413cb80 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient(hunting): my global_id is 184282 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab84073a30 con 0x7fab8413cb80 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3150933712 0 0) 0x7fab840a2fe0 con 0x7fab8406cd20 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 >> v1:172.21.15.63:6789/0 conn(0x7fab8413cb80 legacy=0x7fab8413ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 >> v1:172.21.15.6:6789/0 conn(0x7fab840a3220 legacy=0x7fab8406d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 monclient: found mon.noname-b 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab840a29f0 con 0x7fab8406cd20 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: _finish_auth 0 2024-09-16T05:55:40.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab74001580 con 0x7fab8406cd20 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 5 monclient: authenticate success, global_id 164507 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: dump: 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.822 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab74001860 con 0x7fab8406cd20 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 4 set_mon_vals no callback set 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 1 -- 172.21.15.6:0/1149176531 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab74001c70 con 0x7fab8406cd20 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab82ffd640 10 monclient: dump: 2024-09-16T05:55:40.823 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 10 monclient: shutdown 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- 172.21.15.6:0/1149176531 >> v1:172.21.15.6:6790/0 conn(0x7fab8406cd20 legacy=0x7fab8406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- 172.21.15.6:0/1149176531 shutdown_connections 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- 172.21.15.6:0/1149176531 >> 172.21.15.6:0/1149176531 conn(0x7fab840a3880 msgr2=0x7fab840a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- 172.21.15.6:0/1149176531 shutdown_connections 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.807+0000 7fab8a137640 1 -- 172.21.15.6:0/1149176531 wait complete. 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: build_initial_monmap 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: monmap: 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:40.825 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 1 Processor -- start 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 1 -- start start 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: init 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient(hunting): picked mon.noname-b con 0x7fab8406cd20 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient(hunting): picked mon.noname-c con 0x7fab840a3220 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient(hunting): picked mon.noname-a con 0x7fab8413cb80 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab84138c00 con 0x7fab8413cb80 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab84138ee0 con 0x7fab8406cd20 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab840a0350 con 0x7fab840a3220 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient(hunting): authenticate will time out at 2202.621237s 2024-09-16T05:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab7affd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fab840a3220 0x7fab8409b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52874/0 (socket says 172.21.15.6:52874) 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab7affd640 1 -- 172.21.15.6:0/3711413282 learned_addr learned my addr 172.21.15.6:0/3711413282 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370396673 0 0) 0x7fab84138c00 con 0x7fab8413cb80 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): my global_id is 194184 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab64003530 con 0x7fab8413cb80 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 944801152 0 0) 0x7fab840a0350 con 0x7fab840a3220 2024-09-16T05:55:40.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): my global_id is 164510 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab84138c00 con 0x7fab840a3220 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2801756927 0 0) 0x7fab84138c00 con 0x7fab840a3220 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab840a0350 con 0x7fab840a3220 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1866502071 0 0) 0x7fab64003530 con 0x7fab8413cb80 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab84138c00 con 0x7fab8413cb80 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab6c0018a0 con 0x7fab840a3220 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): dump: 2024-09-16T05:55:40.829 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab7c001bc0 con 0x7fab8413cb80 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): dump: 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.831 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2361077703 0 0) 0x7fab84138ee0 con 0x7fab8406cd20 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _negotiate 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient(hunting): my global_id is 184285 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab64003530 con 0x7fab8406cd20 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2214360811 0 0) 0x7fab84138c00 con 0x7fab8413cb80 2024-09-16T05:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 >> v1:172.21.15.6:6790/0 conn(0x7fab840a3220 legacy=0x7fab8409b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 >> v1:172.21.15.63:6789/0 conn(0x7fab8406cd20 legacy=0x7fab8409b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient: _finish_hunting 0 2024-09-16T05:55:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 monclient: found mon.a 2024-09-16T05:55:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab840a08a0 con 0x7fab8413cb80 2024-09-16T05:55:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient: _finish_auth 0 2024-09-16T05:55:40.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient: _check_auth_tickets 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 5 monclient: authenticate success, global_id 194184 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: _renew_subs 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fab84138920 con 0x7fab8413cb80 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: _renew_subs 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fab841d4af0 con 0x7fab8413cb80 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab7c0039a0 con 0x7fab8413cb80 2024-09-16T05:55:40.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fab7c003cf0 con 0x7fab8413cb80 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab827fc640 4 set_mon_vals no callback set 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 10 monclient: dump: 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:40.835 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fab7c00ea90 con 0x7fab8413cb80 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab68005090 con 0x7fab8413cb80 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fab7c043f10 con 0x7fab8413cb80 2024-09-16T05:55:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.811+0000 7fab837fe640 10 monclient: get_auth_request con 0x7fab64036b40 auth_method 0 2024-09-16T05:55:40.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.815+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fab7c0144e0 con 0x7fab8413cb80 2024-09-16T05:55:40.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.815+0000 7fab80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:40.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.815+0000 7fab80ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:40.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.975+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fab68000ba0 con 0x7fab64036b40 2024-09-16T05:55:40.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.975+0000 7fab80ff9640 1 -- 172.21.15.6:0/3711413282 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7fab68000ba0 con 0x7fab64036b40 2024-09-16T05:55:40.980 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T05:55:40.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.979+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 >> v1:172.21.15.63:6812/3643207211 conn(0x7fab64036b40 legacy=0x7fab64039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.979+0000 7fab8a137640 10 monclient: shutdown 2024-09-16T05:55:40.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.979+0000 7fab8a137640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:40.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.979+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 >> v1:172.21.15.6:6789/0 conn(0x7fab8413cb80 legacy=0x7fab8409be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:40.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.979+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 shutdown_connections 2024-09-16T05:55:40.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.979+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 >> 172.21.15.6:0/3711413282 conn(0x7fab840a3880 msgr2=0x7fab840a6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:40.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.979+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 shutdown_connections 2024-09-16T05:55:40.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:40.979+0000 7fab8a137640 1 -- 172.21.15.6:0/3711413282 wait complete. 2024-09-16T05:55:40.997 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T05:55:40.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:55:41.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: get_monmap_and_config 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: build_initial_monmap 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: monmap: 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:41.115251+0000 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:41.115251+0000 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.114 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:41.115 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:41.115 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 Processor -- start 2024-09-16T05:55:41.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- start start 2024-09-16T05:55:41.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: init 2024-09-16T05:55:41.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:41.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:41.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient(hunting): picked mon.noname-b con 0x7f6360136df0 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:41.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient(hunting): picked mon.noname-c con 0x7f63601371d0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient(hunting): picked mon.noname-a con 0x7f6360136740 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63600a1260 con 0x7f6360136740 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63600730b0 con 0x7f63601371d0 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f636009eaa0 con 0x7f6360136df0 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient(hunting): authenticate will time out at 2202.921879s 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6360136df0 0x7f6360072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52878/0 (socket says 172.21.15.6:52878) 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635ffff640 1 -- 172.21.15.6:0/3385253191 learned_addr learned my addr 172.21.15.6:0/3385253191 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f6364c88640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6360136740 0x7f636013fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40818/0 (socket says 172.21.15.6:40818) 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034233532 0 0) 0x7f63600a1260 con 0x7f6360136740 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient(hunting): my global_id is 194187 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f634c003530 con 0x7f6360136740 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278504740 0 0) 0x7f636009eaa0 con 0x7f6360136df0 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient(hunting): my global_id is 164513 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63600a1260 con 0x7f6360136df0 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1459755387 0 0) 0x7f634c003530 con 0x7f6360136740 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 >> v1:172.21.15.6:6790/0 conn(0x7f6360136df0 legacy=0x7f6360072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 >> v1:172.21.15.63:6789/0 conn(0x7f63601371d0 legacy=0x7f63600729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: _finish_hunting 0 2024-09-16T05:55:41.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 monclient: found mon.noname-a 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636009b1d0 con 0x7f6360136740 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: _finish_auth 0 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: _check_auth_tickets 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f63580019a0 con 0x7f6360136740 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 5 monclient: authenticate success, global_id 194187 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: dump: 2024-09-16T05:55:41.124 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6358001c10 con 0x7f6360136740 2024-09-16T05:55:41.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 1 -- 172.21.15.6:0/3385253191 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6358002020 con 0x7f6360136740 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f635effd640 10 monclient: dump: 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.126 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 4 set_mon_vals no callback set 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: shutdown 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- 172.21.15.6:0/3385253191 >> v1:172.21.15.6:6789/0 conn(0x7f6360136740 legacy=0x7f636013fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- 172.21.15.6:0/3385253191 shutdown_connections 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- 172.21.15.6:0/3385253191 >> 172.21.15.6:0/3385253191 conn(0x7f63600a1650 msgr2=0x7f63600a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- 172.21.15.6:0/3385253191 shutdown_connections 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 1 -- 172.21.15.6:0/3385253191 wait complete. 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: build_initial_monmap 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.111+0000 7f63666eb640 10 monclient: monmap: 2024-09-16T05:55:41.127 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 1 Processor -- start 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 1 -- start start 2024-09-16T05:55:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient: init 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient(hunting): picked mon.noname-c con 0x7f6360136740 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient(hunting): picked mon.noname-b con 0x7f6360136df0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient(hunting): picked mon.noname-a con 0x7f63601371d0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6360125c80 con 0x7f63601371d0 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63601243f0 con 0x7f6360136df0 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6360124ab0 con 0x7f6360136740 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient(hunting): authenticate will time out at 2202.924770s 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f6364c88640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f63601371d0 0x7f63601d41b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40832/0 (socket says 172.21.15.6:40832) 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f6364c88640 1 -- 172.21.15.6:0/2308326033 learned_addr learned my addr 172.21.15.6:0/2308326033 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:41.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635ffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6360136740 0x7f63601cb890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52884/0 (socket says 172.21.15.6:52884) 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1138588998 0 0) 0x7f6360125c80 con 0x7f63601371d0 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): my global_id is 194190 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6340003560 con 0x7f63601371d0 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1001589628 0 0) 0x7f6360124ab0 con 0x7f6360136740 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): my global_id is 164516 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6360125c80 con 0x7f6360136740 2024-09-16T05:55:41.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3019594627 0 0) 0x7f6340003560 con 0x7f63601371d0 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6360124ab0 con 0x7f63601371d0 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2091347390 0 0) 0x7f6360125c80 con 0x7f6360136740 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6340003560 con 0x7f6360136740 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6350001a00 con 0x7f6360136740 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): dump: 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342206994 0 0) 0x7f63601243f0 con 0x7f6360136df0 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): my global_id is 184291 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6360125c80 con 0x7f6360136df0 2024-09-16T05:55:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6358002000 con 0x7f63601371d0 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient(hunting): dump: 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3337297843 0 0) 0x7f6340003560 con 0x7f6360136740 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 >> v1:172.21.15.63:6789/0 conn(0x7f6360136df0 legacy=0x7f63601d0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 >> v1:172.21.15.6:6789/0 conn(0x7f63601371d0 legacy=0x7f63601d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: _finish_hunting 0 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 monclient: found mon.c 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6360125f60 con 0x7f6360136740 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: _finish_auth 0 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: _check_auth_tickets 2024-09-16T05:55:41.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 5 monclient: authenticate success, global_id 164516 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient: _renew_subs 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63600a4100 con 0x7f6360136740 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient: _renew_subs 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f63601259a0 con 0x7f6360136740 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6350001cb0 con 0x7f6360136740 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f6350004c40 con 0x7f6360136740 2024-09-16T05:55:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635e7fc640 4 set_mon_vals no callback set 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: dump: 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.136 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6344000f90 con 0x7f6360136740 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f635000f9e0 con 0x7f6360136740 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f6350044d60 con 0x7f6360136740 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6800/1572197727 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6340044650 con 0x7f6340040a20 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (unknown 1031510311 0 0) 0x7f6350045170 con 0x7f6360136740 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f6364c88640 10 monclient: get_auth_request con 0x7f6340040a20 auth_method 0 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635f7fe640 10 monclient: get_auth_request con 0x7f6340036bd0 auth_method 0 2024-09-16T05:55:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.115+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== osd.2 v1:172.21.15.6:6800/1572197727 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6340044650 con 0x7f6340040a20 2024-09-16T05:55:41.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 --> v1:172.21.15.6:6800/1572197727 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6344002cc0 con 0x7f6340040a20 2024-09-16T05:55:41.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f635cff9640 1 -- 172.21.15.6:0/2308326033 <== osd.2 v1:172.21.15.6:6800/1572197727 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f6344002cc0 con 0x7f6340040a20 2024-09-16T05:55:41.139 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:41.139 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:55:41.139 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:41.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 >> v1:172.21.15.6:6800/1572197727 conn(0x7f6340040a20 legacy=0x7f6340042e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 >> v1:172.21.15.63:6812/3643207211 conn(0x7f6340036bd0 legacy=0x7f6340039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 10 monclient: shutdown 2024-09-16T05:55:41.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:41.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 >> v1:172.21.15.6:6790/0 conn(0x7f6360136740 legacy=0x7f63601cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f6364c88640 1 -- 172.21.15.6:0/2308326033 reap_dead start 2024-09-16T05:55:41.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 shutdown_connections 2024-09-16T05:55:41.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 >> 172.21.15.6:0/2308326033 conn(0x7f63600a1650 msgr2=0x7f63600a2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:41.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 shutdown_connections 2024-09-16T05:55:41.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.131+0000 7f63666eb640 1 -- 172.21.15.6:0/2308326033 wait complete. 2024-09-16T05:55:41.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T05:55:41.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient: get_monmap_and_config 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient: build_initial_monmap 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient: monmap: 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:41.263310+0000 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:41.263310+0000 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.262 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:41.263 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:41.263 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:41.263 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 1 Processor -- start 2024-09-16T05:55:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 1 -- start start 2024-09-16T05:55:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient: init 2024-09-16T05:55:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient(hunting): picked mon.noname-b con 0x7fd8b80a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient(hunting): picked mon.noname-a con 0x7fd8b80a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient(hunting): picked mon.noname-c con 0x7fd8b80a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b806e520 con 0x7fd8b80a4290 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b80a0780 con 0x7fd8b80a1a00 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b80a4c70 con 0x7fd8b80a37a0 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient(hunting): authenticate will time out at 2203.070526s 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bdad3640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd8b80a37a0 0x7fd8b80a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52892/0 (socket says 172.21.15.6:52892) 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bdad3640 1 -- 172.21.15.6:0/1551024803 learned_addr learned my addr 172.21.15.6:0/1551024803 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96776048 0 0) 0x7fd8b80a4c70 con 0x7fd8b80a37a0 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient(hunting): my global_id is 164519 2024-09-16T05:55:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8a4003530 con 0x7fd8b80a37a0 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3917145591 0 0) 0x7fd8b806e520 con 0x7fd8b80a4290 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient(hunting): my global_id is 194193 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b80a4c70 con 0x7fd8b80a4290 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1037144519 0 0) 0x7fd8a4003530 con 0x7fd8b80a37a0 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 >> v1:172.21.15.63:6789/0 conn(0x7fd8b80a1a00 legacy=0x7fd8b813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 >> v1:172.21.15.6:6789/0 conn(0x7fd8b80a4290 legacy=0x7fd8b813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: _finish_hunting 0 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 monclient: found mon.noname-b 2024-09-16T05:55:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b80a65f0 con 0x7fd8b80a37a0 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: _finish_auth 0 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: _check_auth_tickets 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8a8001620 con 0x7fd8b80a37a0 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 5 monclient: authenticate success, global_id 164519 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: dump: 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8a80018c0 con 0x7fd8b80a37a0 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 1 -- 172.21.15.6:0/1551024803 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8a8001cd0 con 0x7fd8b80a37a0 2024-09-16T05:55:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bfd37640 4 set_mon_vals no callback set 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.259+0000 7fd8bcad1640 10 monclient: dump: 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: shutdown 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- 172.21.15.6:0/1551024803 >> v1:172.21.15.6:6790/0 conn(0x7fd8b80a37a0 legacy=0x7fd8b80a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- 172.21.15.6:0/1551024803 shutdown_connections 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- 172.21.15.6:0/1551024803 >> 172.21.15.6:0/1551024803 conn(0x7fd8b80a58a0 msgr2=0x7fd8b80a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- 172.21.15.6:0/1551024803 shutdown_connections 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- 172.21.15.6:0/1551024803 wait complete. 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: build_initial_monmap 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: monmap: 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:41.273 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 Processor -- start 2024-09-16T05:55:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- start start 2024-09-16T05:55:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: init 2024-09-16T05:55:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:55:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient(hunting): picked mon.noname-b con 0x7fd8b80a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient(hunting): picked mon.noname-c con 0x7fd8b80a37a0 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient(hunting): picked mon.noname-a con 0x7fd8b80a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b8137680 con 0x7fd8b80a4290 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b8135760 con 0x7fd8b80a1a00 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8b8134a20 con 0x7fd8b80a37a0 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient(hunting): authenticate will time out at 2203.074153s 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8be2d4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd8b80a4290 0x7fd8b81d3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40854/0 (socket says 172.21.15.6:40854) 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8be2d4640 1 -- 172.21.15.6:0/2696081047 learned_addr learned my addr 172.21.15.6:0/2696081047 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2340730564 0 0) 0x7fd8b8137680 con 0x7fd8b80a4290 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): my global_id is 194196 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c003530 con 0x7fd8b80a4290 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3748557664 0 0) 0x7fd8b8134a20 con 0x7fd8b80a37a0 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): my global_id is 164522 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b8137680 con 0x7fd8b80a37a0 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 812589278 0 0) 0x7fd89c003530 con 0x7fd8b80a4290 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8b8134a20 con 0x7fd8b80a4290 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8b0001cd0 con 0x7fd8b80a4290 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): dump: 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 930832553 0 0) 0x7fd8b8135760 con 0x7fd8b80a1a00 2024-09-16T05:55:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): my global_id is 184297 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89c003530 con 0x7fd8b80a1a00 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2689961429 0 0) 0x7fd8b8137680 con 0x7fd8b80a37a0 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8b8135760 con 0x7fd8b80a37a0 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8b4001950 con 0x7fd8b80a37a0 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient(hunting): dump: 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 364343939 0 0) 0x7fd8b8134a20 con 0x7fd8b80a4290 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 >> v1:172.21.15.6:6790/0 conn(0x7fd8b80a37a0 legacy=0x7fd8b81d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 >> v1:172.21.15.63:6789/0 conn(0x7fd8b80a1a00 legacy=0x7fd8b8136f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 monclient: found mon.a 2024-09-16T05:55:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b80a8070 con 0x7fd8b80a4290 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient: _finish_auth 0 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 5 monclient: authenticate success, global_id 194196 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: _renew_subs 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b80a7bb0 con 0x7fd8b80a4290 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: _renew_subs 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8b8134440 con 0x7fd8b80a4290 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8b00039a0 con 0x7fd8b80a4290 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd8b0003d70 con 0x7fd8b80a4290 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8affff640 4 set_mon_vals no callback set 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8ae7fc640 10 monclient: dump: 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.263+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a0005090 con 0x7fd8b80a4290 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.267+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd8b0003f10 con 0x7fd8b80a4290 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.267+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fd8b00451d0 con 0x7fd8b80a4290 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.267+0000 7fd8bdad3640 10 monclient: get_auth_request con 0x7fd89c036bd0 auth_method 0 2024-09-16T05:55:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.271+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd8b0015820 con 0x7fd8b80a4290 2024-09-16T05:55:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.271+0000 7fd8ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.271+0000 7fd8ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:41.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.435+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fd8a0000ba0 con 0x7fd89c036bd0 2024-09-16T05:55:41.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.435+0000 7fd8ae7fc640 1 -- 172.21.15.6:0/2696081047 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7fd8a0000ba0 con 0x7fd89c036bd0 2024-09-16T05:55:41.441 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T05:55:41.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.439+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd89c036bd0 legacy=0x7fd89c039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.439+0000 7fd8bfd37640 10 monclient: shutdown 2024-09-16T05:55:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.439+0000 7fd8bfd37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.439+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 >> v1:172.21.15.6:6789/0 conn(0x7fd8b80a4290 legacy=0x7fd8b81d3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.439+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 shutdown_connections 2024-09-16T05:55:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.439+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 >> 172.21.15.6:0/2696081047 conn(0x7fd8b80a58a0 msgr2=0x7fd8b80a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.439+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 shutdown_connections 2024-09-16T05:55:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.439+0000 7fd8bfd37640 1 -- 172.21.15.6:0/2696081047 wait complete. 2024-09-16T05:55:41.461 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T05:55:41.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:55:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.583+0000 7fdfd3192640 10 monclient: get_monmap_and_config 2024-09-16T05:55:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.583+0000 7fdfd3192640 10 monclient: build_initial_monmap 2024-09-16T05:55:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.583+0000 7fdfd3192640 10 monclient: monmap: 2024-09-16T05:55:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:41.590698+0000 2024-09-16T05:55:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:41.590698+0000 2024-09-16T05:55:41.589 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:41.590 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:41.590 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 Processor -- start 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- start start 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient: init 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient(hunting): picked mon.noname-c con 0x7fdfcc136a20 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient(hunting): picked mon.noname-b con 0x7fdfcc139900 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient(hunting): picked mon.noname-a con 0x7fdfcc13d410 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfcc137510 con 0x7fdfcc13d410 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfcc0a4cf0 con 0x7fdfcc136a20 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfcc0a4a70 con 0x7fdfcc139900 2024-09-16T05:55:41.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient(hunting): authenticate will time out at 2203.397536s 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcbfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdfcc139900 0x7fdfcc13bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52912/0 (socket says 172.21.15.6:52912) 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcbfff640 1 -- 172.21.15.6:0/3369169258 learned_addr learned my addr 172.21.15.6:0/3369169258 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd172f640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdfcc13d410 0x7fdfcc13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40868/0 (socket says 172.21.15.6:40868) 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1202852439 0 0) 0x7fdfcc137510 con 0x7fdfcc13d410 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient(hunting): my global_id is 194199 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfb4003530 con 0x7fdfcc13d410 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109673266 0 0) 0x7fdfcc0a4a70 con 0x7fdfcc139900 2024-09-16T05:55:41.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient(hunting): my global_id is 164525 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfcc137510 con 0x7fdfcc139900 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2690875980 0 0) 0x7fdfb4003530 con 0x7fdfcc13d410 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 >> v1:172.21.15.6:6790/0 conn(0x7fdfcc139900 legacy=0x7fdfcc13bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 >> v1:172.21.15.63:6789/0 conn(0x7fdfcc136a20 legacy=0x7fdfcc136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: _finish_hunting 0 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 monclient: found mon.noname-a 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc0a07a0 con 0x7fdfcc13d410 2024-09-16T05:55:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: _finish_auth 0 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: _check_auth_tickets 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdfc40019a0 con 0x7fdfcc13d410 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 5 monclient: authenticate success, global_id 194199 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: dump: 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.596 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdfc4001c50 con 0x7fdfcc13d410 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 1 -- 172.21.15.6:0/3369169258 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdfc4002020 con 0x7fdfcc13d410 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfcb7fe640 10 monclient: dump: 2024-09-16T05:55:41.597 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 4 set_mon_vals no callback set 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient: shutdown 2024-09-16T05:55:41.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- 172.21.15.6:0/3369169258 >> v1:172.21.15.6:6789/0 conn(0x7fdfcc13d410 legacy=0x7fdfcc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- 172.21.15.6:0/3369169258 shutdown_connections 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- 172.21.15.6:0/3369169258 >> 172.21.15.6:0/3369169258 conn(0x7fdfcc0a5a10 msgr2=0x7fdfcc0a5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- 172.21.15.6:0/3369169258 shutdown_connections 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 1 -- 172.21.15.6:0/3369169258 wait complete. 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient: build_initial_monmap 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.587+0000 7fdfd3192640 10 monclient: monmap: 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:41.599 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:41.600 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.600 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:41.600 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:41.600 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:41.600 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 1 Processor -- start 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 1 -- start start 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient: init 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient(hunting): picked mon.noname-c con 0x7fdfcc136a20 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient(hunting): picked mon.noname-a con 0x7fdfcc139900 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient(hunting): picked mon.noname-b con 0x7fdfcc13d410 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfcc120d00 con 0x7fdfcc139900 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfcc120fe0 con 0x7fdfcc13d410 2024-09-16T05:55:41.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfcc0a70f0 con 0x7fdfcc136a20 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient(hunting): authenticate will time out at 2203.400673s 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd0f2e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdfcc136a20 0x7fdfcc1350f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52922/0 (socket says 172.21.15.6:52922) 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd0f2e640 1 -- 172.21.15.6:0/3550707798 learned_addr learned my addr 172.21.15.6:0/3550707798 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1668900794 0 0) 0x7fdfcc0a70f0 con 0x7fdfcc136a20 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): my global_id is 164528 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfb8003530 con 0x7fdfcc136a20 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940017389 0 0) 0x7fdfcc120d00 con 0x7fdfcc139900 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): my global_id is 194202 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfcc0a70f0 con 0x7fdfcc139900 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3479596020 0 0) 0x7fdfb8003530 con 0x7fdfcc136a20 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfcc120d00 con 0x7fdfcc136a20 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdfc0001920 con 0x7fdfcc136a20 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): dump: 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.606 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3903407860 0 0) 0x7fdfcc0a70f0 con 0x7fdfcc139900 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfb8003530 con 0x7fdfcc139900 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdfbc0017c0 con 0x7fdfcc139900 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:41.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): dump: 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327837126 0 0) 0x7fdfcc120fe0 con 0x7fdfcc13d410 2024-09-16T05:55:41.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient(hunting): my global_id is 184303 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfcc0a70f0 con 0x7fdfcc13d410 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 875327312 0 0) 0x7fdfcc120d00 con 0x7fdfcc136a20 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 >> v1:172.21.15.63:6789/0 conn(0x7fdfcc13d410 legacy=0x7fdfcc1d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 >> v1:172.21.15.6:6789/0 conn(0x7fdfcc139900 legacy=0x7fdfcc1d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: _finish_hunting 0 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 monclient: found mon.c 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc13d010 con 0x7fdfcc136a20 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: _finish_auth 0 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: _check_auth_tickets 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 5 monclient: authenticate success, global_id 164528 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient: _renew_subs 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdfcc0a0100 con 0x7fdfcc136a20 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient: _renew_subs 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdfcc09e620 con 0x7fdfcc136a20 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdfc0003660 con 0x7fdfcc136a20 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:41.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fdfc00039b0 con 0x7fdfcc136a20 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfcaffd640 4 set_mon_vals no callback set 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: dump: 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.611 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fdfac000f90 con 0x7fdfcc136a20 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fdfc0003bd0 con 0x7fdfcc136a20 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fdfc0044eb0 con 0x7fdfcc136a20 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.63:6800/709309215 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfb8044780 con 0x7fdfb8040ab0 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (unknown 1031510311 0 0) 0x7fdfc0045320 con 0x7fdfcc136a20 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfc97fa640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfcbfff640 10 monclient: get_auth_request con 0x7fdfb8036c60 auth_method 0 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.591+0000 7fdfd172f640 10 monclient: get_auth_request con 0x7fdfb8040ab0 auth_method 0 2024-09-16T05:55:41.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.595+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== osd.3 v1:172.21.15.63:6800/709309215 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdfb8044780 con 0x7fdfb8040ab0 2024-09-16T05:55:41.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 --> v1:172.21.15.63:6800/709309215 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdfac002cc0 con 0x7fdfb8040ab0 2024-09-16T05:55:41.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfc97fa640 1 -- 172.21.15.6:0/3550707798 <== osd.3 v1:172.21.15.63:6800/709309215 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fdfac002cc0 con 0x7fdfb8040ab0 2024-09-16T05:55:41.613 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:41.613 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:55:41.613 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:41.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 >> v1:172.21.15.63:6800/709309215 conn(0x7fdfb8040ab0 legacy=0x7fdfb8042f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 >> v1:172.21.15.63:6812/3643207211 conn(0x7fdfb8036c60 legacy=0x7fdfb8039120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 10 monclient: shutdown 2024-09-16T05:55:41.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:41.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 >> v1:172.21.15.6:6790/0 conn(0x7fdfcc136a20 legacy=0x7fdfcc1350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd172f640 1 -- 172.21.15.6:0/3550707798 reap_dead start 2024-09-16T05:55:41.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 shutdown_connections 2024-09-16T05:55:41.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 >> 172.21.15.6:0/3550707798 conn(0x7fdfcc0a5a10 msgr2=0x7fdfcc13bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:41.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 shutdown_connections 2024-09-16T05:55:41.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.607+0000 7fdfd3192640 1 -- 172.21.15.6:0/3550707798 wait complete. 2024-09-16T05:55:41.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T05:55:41.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.735+0000 7f8d884b5640 10 monclient: get_monmap_and_config 2024-09-16T05:55:41.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.735+0000 7f8d884b5640 10 monclient: build_initial_monmap 2024-09-16T05:55:41.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.735+0000 7f8d884b5640 10 monclient: monmap: 2024-09-16T05:55:41.742 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:41.742 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.742 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:41.742289+0000 2024-09-16T05:55:41.742 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:41.742289+0000 2024-09-16T05:55:41.742 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:41.742 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.743 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:41.743 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:41.743 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:41.743 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 Processor -- start 2024-09-16T05:55:41.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- start start 2024-09-16T05:55:41.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient: init 2024-09-16T05:55:41.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:41.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:55:41.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient(hunting): picked mon.noname-c con 0x7f8d8009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:41.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient(hunting): picked mon.noname-b con 0x7f8d8009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient(hunting): picked mon.noname-a con 0x7f8d800a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d800a1370 con 0x7f8d800a0da0 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8009ead0 con 0x7f8d8009aa30 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8009edb0 con 0x7f8d8009e230 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient(hunting): authenticate will time out at 2203.548947s 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d85a50640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8d8009e230 0x7f8d801399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52934/0 (socket says 172.21.15.6:52934) 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d85a50640 1 -- 172.21.15.6:0/291009948 learned_addr learned my addr 172.21.15.6:0/291009948 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833273873 0 0) 0x7f8d8009edb0 con 0x7f8d8009e230 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient(hunting): my global_id is 164531 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d6c003530 con 0x7f8d8009e230 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568490817 0 0) 0x7f8d800a1370 con 0x7f8d800a0da0 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient(hunting): my global_id is 194205 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d8009edb0 con 0x7f8d800a0da0 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3683093115 0 0) 0x7f8d6c003530 con 0x7f8d8009e230 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 >> v1:172.21.15.63:6789/0 conn(0x7f8d8009aa30 legacy=0x7f8d8009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 >> v1:172.21.15.6:6789/0 conn(0x7f8d800a0da0 legacy=0x7f8d8013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: _finish_hunting 0 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 monclient: found mon.noname-b 2024-09-16T05:55:41.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8009dd50 con 0x7f8d8009e230 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: _finish_auth 0 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: _check_auth_tickets 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8d70001620 con 0x7f8d8009e230 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: dump: 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.753 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 5 monclient: authenticate success, global_id 164531 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d70001880 con 0x7f8d8009e230 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 1 -- 172.21.15.6:0/291009948 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8d70001bd0 con 0x7f8d8009e230 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d8524f640 10 monclient: dump: 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:41.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 4 set_mon_vals no callback set 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient: shutdown 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- 172.21.15.6:0/291009948 >> v1:172.21.15.6:6790/0 conn(0x7f8d8009e230 legacy=0x7f8d801399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- 172.21.15.6:0/291009948 shutdown_connections 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- 172.21.15.6:0/291009948 >> 172.21.15.6:0/291009948 conn(0x7f8d800a1650 msgr2=0x7f8d800a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- 172.21.15.6:0/291009948 shutdown_connections 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- 172.21.15.6:0/291009948 wait complete. 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient: build_initial_monmap 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient: monmap: 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:41.756 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 Processor -- start 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 1 -- start start 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.739+0000 7f8d884b5640 10 monclient: init 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:41.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient(hunting): picked mon.noname-b con 0x7f8d8009aa30 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient(hunting): picked mon.noname-c con 0x7f8d8009e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient(hunting): picked mon.noname-a con 0x7f8d800a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8013c8c0 con 0x7f8d800a0da0 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8013cba0 con 0x7f8d8009aa30 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d800718c0 con 0x7f8d8009e230 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient(hunting): authenticate will time out at 2203.552229s 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d86a52640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8d800a0da0 0x7f8d801d40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40896/0 (socket says 172.21.15.6:40896) 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d85a50640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8d8009e230 0x7f8d801d09a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52936/0 (socket says 172.21.15.6:52936) 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d86a52640 1 -- 172.21.15.6:0/3693783472 learned_addr learned my addr 172.21.15.6:0/3693783472 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:41.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2362570883 0 0) 0x7f8d800718c0 con 0x7f8d8009e230 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): my global_id is 164534 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d64003530 con 0x7f8d8009e230 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 837808257 0 0) 0x7f8d8013c8c0 con 0x7f8d800a0da0 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): my global_id is 194208 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d800718c0 con 0x7f8d800a0da0 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 552121244 0 0) 0x7f8d64003530 con 0x7f8d8009e230 2024-09-16T05:55:41.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d8013c8c0 con 0x7f8d8009e230 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 672605528 0 0) 0x7f8d8013cba0 con 0x7f8d8009aa30 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): my global_id is 184309 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d64003530 con 0x7f8d8009aa30 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8d70001520 con 0x7f8d8009e230 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): dump: 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.760 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 876816936 0 0) 0x7f8d800718c0 con 0x7f8d800a0da0 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d8013cba0 con 0x7f8d800a0da0 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8d780018c0 con 0x7f8d800a0da0 2024-09-16T05:55:41.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient(hunting): dump: 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1679244965 0 0) 0x7f8d8013cba0 con 0x7f8d800a0da0 2024-09-16T05:55:41.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 >> v1:172.21.15.6:6790/0 conn(0x7f8d8009e230 legacy=0x7f8d801d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 >> v1:172.21.15.63:6789/0 conn(0x7f8d8009aa30 legacy=0x7f8d801cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient: _finish_hunting 0 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 monclient: found mon.a 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8013ce80 con 0x7f8d800a0da0 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient: _finish_auth 0 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient: _check_auth_tickets 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 5 monclient: authenticate success, global_id 194208 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient: _renew_subs 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8013d580 con 0x7f8d800a0da0 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient: _renew_subs 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8d8009cd50 con 0x7f8d800a0da0 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d78003620 con 0x7f8d800a0da0 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f8d78003970 con 0x7f8d800a0da0 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d84a4e640 4 set_mon_vals no callback set 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 10 monclient: dump: 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.764 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d68005090 con 0x7f8d800a0da0 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f8d78003b10 con 0x7f8d800a0da0 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f8d78044d80 con 0x7f8d800a0da0 2024-09-16T05:55:41.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.743+0000 7f8d86251640 10 monclient: get_auth_request con 0x7f8d64036bf0 auth_method 0 2024-09-16T05:55:41.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.747+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f8d78074030 con 0x7f8d800a0da0 2024-09-16T05:55:41.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.747+0000 7f8d76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:41.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.747+0000 7f8d76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:41.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.903+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f8d68000ba0 con 0x7f8d64036bf0 2024-09-16T05:55:41.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.903+0000 7f8d76ffd640 1 -- 172.21.15.6:0/3693783472 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7f8d68000ba0 con 0x7f8d64036bf0 2024-09-16T05:55:41.909 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T05:55:41.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.907+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 >> v1:172.21.15.63:6812/3643207211 conn(0x7f8d64036bf0 legacy=0x7f8d640390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.907+0000 7f8d884b5640 10 monclient: shutdown 2024-09-16T05:55:41.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.907+0000 7f8d884b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:41.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.907+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 >> v1:172.21.15.6:6789/0 conn(0x7f8d800a0da0 legacy=0x7f8d801d40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:41.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.907+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 shutdown_connections 2024-09-16T05:55:41.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.907+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 >> 172.21.15.6:0/3693783472 conn(0x7f8d800a1650 msgr2=0x7f8d800a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:41.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.907+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 shutdown_connections 2024-09-16T05:55:41.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:41.907+0000 7f8d884b5640 1 -- 172.21.15.6:0/3693783472 wait complete. 2024-09-16T05:55:41.925 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T05:55:41.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:55:42.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient: get_monmap_and_config 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient: monmap: 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:42.040251+0000 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:42.040251+0000 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:42.039 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 1 Processor -- start 2024-09-16T05:55:42.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 1 -- start start 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient: init 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient(hunting): picked mon.noname-b con 0x7fe6ac134940 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient(hunting): picked mon.noname-a con 0x7fe6ac09e210 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient(hunting): picked mon.noname-c con 0x7fe6ac134410 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac072ef0 con 0x7fe6ac09e210 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac09ead0 con 0x7fe6ac134410 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac0a07c0 con 0x7fe6ac134940 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient(hunting): authenticate will time out at 2203.847019s 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6ab7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe6ac134940 0x7fe6ac134d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52950/0 (socket says 172.21.15.6:52950) 2024-09-16T05:55:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6ab7fe640 1 -- 172.21.15.6:0/3091207088 learned_addr learned my addr 172.21.15.6:0/3091207088 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3046761782 0 0) 0x7fe6ac0a07c0 con 0x7fe6ac134940 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient(hunting): my global_id is 164537 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe694003530 con 0x7fe6ac134940 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3993219185 0 0) 0x7fe6ac072ef0 con 0x7fe6ac09e210 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient(hunting): my global_id is 194211 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6ac0a07c0 con 0x7fe6ac09e210 2024-09-16T05:55:42.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2054339960 0 0) 0x7fe6ac0a07c0 con 0x7fe6ac09e210 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 >> v1:172.21.15.6:6790/0 conn(0x7fe6ac134940 legacy=0x7fe6ac134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 >> v1:172.21.15.63:6789/0 conn(0x7fe6ac134410 legacy=0x7fe6ac13ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 monclient: found mon.noname-a 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac1415d0 con 0x7fe6ac09e210 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe6a0001620 con 0x7fe6ac09e210 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 5 monclient: authenticate success, global_id 194211 2024-09-16T05:55:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: dump: 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.046 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6a0001810 con 0x7fe6ac09e210 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 1 -- 172.21.15.6:0/3091207088 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe6a0001c20 con 0x7fe6ac09e210 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6b1ee1640 4 set_mon_vals no callback set 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.035+0000 7fe6aa7fc640 10 monclient: dump: 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.047 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: shutdown 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/3091207088 >> v1:172.21.15.6:6789/0 conn(0x7fe6ac09e210 legacy=0x7fe6ac139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/3091207088 shutdown_connections 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/3091207088 >> 172.21.15.6:0/3091207088 conn(0x7fe6ac0a1650 msgr2=0x7fe6ac0a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/3091207088 shutdown_connections 2024-09-16T05:55:42.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/3091207088 wait complete. 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: monmap: 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 Processor -- start 2024-09-16T05:55:42.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- start start 2024-09-16T05:55:42.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: init 2024-09-16T05:55:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient(hunting): picked mon.noname-c con 0x7fe6ac09e210 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient(hunting): picked mon.noname-a con 0x7fe6ac134410 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient(hunting): picked mon.noname-b con 0x7fe6ac134940 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac133b30 con 0x7fe6ac134410 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac0740c0 con 0x7fe6ac134940 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6ab7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe6ac09e210 0x7fe6ac133420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52966/0 (socket says 172.21.15.6:52966) 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6ab7fe640 1 -- 172.21.15.6:0/405014961 learned_addr learned my addr 172.21.15.6:0/405014961 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac13e270 con 0x7fe6ac09e210 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient(hunting): authenticate will time out at 2203.850327s 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 216907516 0 0) 0x7fe6ac13e270 con 0x7fe6ac09e210 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): my global_id is 164540 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe698003560 con 0x7fe6ac09e210 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 857364764 0 0) 0x7fe6ac133b30 con 0x7fe6ac134410 2024-09-16T05:55:42.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): my global_id is 194214 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6ac13e270 con 0x7fe6ac134410 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1118524203 0 0) 0x7fe698003560 con 0x7fe6ac09e210 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6ac133b30 con 0x7fe6ac09e210 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe69c0013c0 con 0x7fe6ac09e210 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): dump: 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.055 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2030877845 0 0) 0x7fe6ac13e270 con 0x7fe6ac134410 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe698003560 con 0x7fe6ac134410 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe6a0001c20 con 0x7fe6ac134410 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): dump: 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889024792 0 0) 0x7fe6ac0740c0 con 0x7fe6ac134940 2024-09-16T05:55:42.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient(hunting): my global_id is 184315 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6ac13e270 con 0x7fe6ac134940 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1347648345 0 0) 0x7fe6ac133b30 con 0x7fe6ac09e210 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 >> v1:172.21.15.63:6789/0 conn(0x7fe6ac134940 legacy=0x7fe6ac1d3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 >> v1:172.21.15.6:6789/0 conn(0x7fe6ac134410 legacy=0x7fe6ac1d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 monclient: found mon.c 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac13d860 con 0x7fe6ac09e210 2024-09-16T05:55:42.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 5 monclient: authenticate success, global_id 164540 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: _renew_subs 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ac09d8e0 con 0x7fe6ac09e210 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: _renew_subs 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6ac1cc3f0 con 0x7fe6ac09e210 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe69c0016b0 con 0x7fe6ac09e210 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe69c004c40 con 0x7fe6ac09e210 2024-09-16T05:55:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 10 monclient: dump: 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6a9ffb640 4 set_mon_vals no callback set 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe68c000f90 con 0x7fe6ac09e210 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.039+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe69c00f9e0 con 0x7fe6ac09e210 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.043+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fe69c044ee0 con 0x7fe6ac09e210 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.043+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.63:6804/118134224 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe698044740 con 0x7fe698040a70 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.043+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (unknown 1031510311 0 0) 0x7fe69c045340 con 0x7fe6ac09e210 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.043+0000 7fe693fff640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.043+0000 7fe6aaffd640 10 monclient: get_auth_request con 0x7fe698036c20 auth_method 0 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.043+0000 7fe6abfff640 10 monclient: get_auth_request con 0x7fe698040a70 auth_method 0 2024-09-16T05:55:42.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.043+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== osd.4 v1:172.21.15.63:6804/118134224 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe698044740 con 0x7fe698040a70 2024-09-16T05:55:42.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.055+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 --> v1:172.21.15.63:6804/118134224 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe68c002cc0 con 0x7fe698040a70 2024-09-16T05:55:42.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe693fff640 1 -- 172.21.15.6:0/405014961 <== osd.4 v1:172.21.15.63:6804/118134224 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fe68c002cc0 con 0x7fe698040a70 2024-09-16T05:55:42.062 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:42.062 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:55:42.063 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:42.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 >> v1:172.21.15.63:6804/118134224 conn(0x7fe698040a70 legacy=0x7fe698042ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe698036c20 legacy=0x7fe6980390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 10 monclient: shutdown 2024-09-16T05:55:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 >> v1:172.21.15.6:6790/0 conn(0x7fe6ac09e210 legacy=0x7fe6ac133420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6abfff640 1 -- 172.21.15.6:0/405014961 reap_dead start 2024-09-16T05:55:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 shutdown_connections 2024-09-16T05:55:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 >> 172.21.15.6:0/405014961 conn(0x7fe6ac0a1650 msgr2=0x7fe6ac140a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 shutdown_connections 2024-09-16T05:55:42.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.059+0000 7fe6b1ee1640 1 -- 172.21.15.6:0/405014961 wait complete. 2024-09-16T05:55:42.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T05:55:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.183+0000 7fe89f9d2640 10 monclient: get_monmap_and_config 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.183+0000 7fe89f9d2640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.183+0000 7fe89f9d2640 10 monclient: monmap: 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:42.190406+0000 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:42.190406+0000 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:42.190 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:42.190 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 Processor -- start 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- start start 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: init 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient(hunting): picked mon.noname-b con 0x7fe89809bb40 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient(hunting): picked mon.noname-c con 0x7fe89809bf20 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient(hunting): picked mon.noname-a con 0x7fe8981397f0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8980a1350 con 0x7fe8981397f0 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8980a10d0 con 0x7fe89809bf20 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe89809d940 con 0x7fe89809bb40 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient(hunting): authenticate will time out at 2203.996951s 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89d76e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe89809bb40 0x7fe89809e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52974/0 (socket says 172.21.15.6:52974) 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89d76e640 1 -- 172.21.15.6:0/1507807546 learned_addr learned my addr 172.21.15.6:0/1507807546 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4109692436 0 0) 0x7fe8980a1350 con 0x7fe8981397f0 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient(hunting): my global_id is 194217 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe884005740 con 0x7fe8981397f0 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993943325 0 0) 0x7fe89809d940 con 0x7fe89809bb40 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient(hunting): my global_id is 164543 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8980a1350 con 0x7fe89809bb40 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3928235610 0 0) 0x7fe884005740 con 0x7fe8981397f0 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 >> v1:172.21.15.6:6790/0 conn(0x7fe89809bb40 legacy=0x7fe89809e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 >> v1:172.21.15.63:6789/0 conn(0x7fe89809bf20 legacy=0x7fe89809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 monclient: found mon.noname-a 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89809dc20 con 0x7fe8981397f0 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe8900019a0 con 0x7fe8981397f0 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: dump: 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 5 monclient: authenticate success, global_id 194217 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe890001c10 con 0x7fe8981397f0 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 1 -- 172.21.15.6:0/1507807546 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe890001fe0 con 0x7fe8981397f0 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 4 set_mon_vals no callback set 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe88ffff640 10 monclient: dump: 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: shutdown 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1507807546 >> v1:172.21.15.6:6789/0 conn(0x7fe8981397f0 legacy=0x7fe89813bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1507807546 shutdown_connections 2024-09-16T05:55:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1507807546 >> 172.21.15.6:0/1507807546 conn(0x7fe8980a1650 msgr2=0x7fe8980a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1507807546 shutdown_connections 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1507807546 wait complete. 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: monmap: 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 Processor -- start 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 1 -- start start 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: init 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.187+0000 7fe89f9d2640 10 monclient(hunting): picked mon.noname-b con 0x7fe89809bb40 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient(hunting): picked mon.noname-a con 0x7fe89809bf20 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient(hunting): picked mon.noname-c con 0x7fe8981397f0 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8980a3cc0 con 0x7fe89809bf20 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8980740c0 con 0x7fe89809bb40 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8981c7470 con 0x7fe8981397f0 2024-09-16T05:55:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient(hunting): authenticate will time out at 2204.000089s 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89cf6d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe89809bf20 0x7fe8981cc120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40928/0 (socket says 172.21.15.6:40928) 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89cf6d640 1 -- 172.21.15.6:0/1676246711 learned_addr learned my addr 172.21.15.6:0/1676246711 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89df6f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe8981397f0 0x7fe8981cf850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52982/0 (socket says 172.21.15.6:52982) 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404078326 0 0) 0x7fe8980a3cc0 con 0x7fe89809bf20 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): my global_id is 194220 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87c003530 con 0x7fe89809bf20 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2641673414 0 0) 0x7fe8981c7470 con 0x7fe8981397f0 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): my global_id is 164546 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8980a3cc0 con 0x7fe8981397f0 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1220032939 0 0) 0x7fe87c003530 con 0x7fe89809bf20 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8981c7470 con 0x7fe89809bf20 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe888001980 con 0x7fe89809bf20 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): dump: 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.205 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232610503 0 0) 0x7fe8980740c0 con 0x7fe89809bb40 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): my global_id is 184321 2024-09-16T05:55:42.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87c003530 con 0x7fe89809bb40 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3939197600 0 0) 0x7fe8980a3cc0 con 0x7fe8981397f0 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8980740c0 con 0x7fe8981397f0 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe890001f60 con 0x7fe8981397f0 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient(hunting): dump: 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.207 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3120628979 0 0) 0x7fe8981c7470 con 0x7fe89809bf20 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 >> v1:172.21.15.6:6790/0 conn(0x7fe8981397f0 legacy=0x7fe8981cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 >> v1:172.21.15.63:6789/0 conn(0x7fe89809bb40 legacy=0x7fe898072490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 monclient: found mon.a 2024-09-16T05:55:42.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe898137b40 con 0x7fe89809bf20 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 5 monclient: authenticate success, global_id 194220 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient: _renew_subs 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe89806ef60 con 0x7fe89809bf20 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient: _renew_subs 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe898072ba0 con 0x7fe89809bf20 2024-09-16T05:55:42.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe8880038d0 con 0x7fe89809bf20 2024-09-16T05:55:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fe888003b00 con 0x7fe89809bf20 2024-09-16T05:55:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88f7fe640 4 set_mon_vals no callback set 2024-09-16T05:55:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 10 monclient: dump: 2024-09-16T05:55:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.210 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.211 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fe888003ca0 con 0x7fe89809bf20 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe878005090 con 0x7fe89809bf20 2024-09-16T05:55:42.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fe888044e80 con 0x7fe89809bf20 2024-09-16T05:55:42.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.191+0000 7fe89d76e640 10 monclient: get_auth_request con 0x7fe87c03afb0 auth_method 0 2024-09-16T05:55:42.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.195+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fe888015470 con 0x7fe89809bf20 2024-09-16T05:55:42.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.195+0000 7fe88dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:42.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.195+0000 7fe88dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.363+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fe878000ba0 con 0x7fe87c03afb0 2024-09-16T05:55:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.363+0000 7fe88dffb640 1 -- 172.21.15.6:0/1676246711 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7fe878000ba0 con 0x7fe87c03afb0 2024-09-16T05:55:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T05:55:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.367+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 >> v1:172.21.15.63:6812/3643207211 conn(0x7fe87c03afb0 legacy=0x7fe87c03d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.367+0000 7fe89f9d2640 10 monclient: shutdown 2024-09-16T05:55:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.367+0000 7fe89f9d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.367+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 >> v1:172.21.15.6:6789/0 conn(0x7fe89809bf20 legacy=0x7fe8981cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.367+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 shutdown_connections 2024-09-16T05:55:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.367+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 >> 172.21.15.6:0/1676246711 conn(0x7fe8980a1650 msgr2=0x7fe89812fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.367+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 shutdown_connections 2024-09-16T05:55:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.367+0000 7fe89f9d2640 1 -- 172.21.15.6:0/1676246711 wait complete. 2024-09-16T05:55:42.389 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T05:55:42.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T05:55:42.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: get_monmap_and_config 2024-09-16T05:55:42.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: monmap: 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:42.511014+0000 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:42.511014+0000 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:42.510 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 Processor -- start 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- start start 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: init 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient(hunting): picked mon.noname-b con 0x7fbf7809c110 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient(hunting): picked mon.noname-c con 0x7fbf7809ed00 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient(hunting): picked mon.noname-a con 0x7fbf78139e40 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf780a1420 con 0x7fbf78139e40 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf780a0d20 con 0x7fbf7809ed00 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7809da60 con 0x7fbf7809c110 2024-09-16T05:55:42.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient(hunting): authenticate will time out at 2204.317408s 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7d1cf640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbf7809c110 0x7fbf7809c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:52992/0 (socket says 172.21.15.6:52992) 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7d1cf640 1 -- 172.21.15.6:0/1218324562 learned_addr learned my addr 172.21.15.6:0/1218324562 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635333355 0 0) 0x7fbf7809da60 con 0x7fbf7809c110 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient(hunting): my global_id is 164549 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf64003530 con 0x7fbf7809c110 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3136738456 0 0) 0x7fbf780a1420 con 0x7fbf78139e40 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient(hunting): my global_id is 194223 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf7809da60 con 0x7fbf78139e40 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1771975862 0 0) 0x7fbf64003530 con 0x7fbf7809c110 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 >> v1:172.21.15.63:6789/0 conn(0x7fbf7809ed00 legacy=0x7fbf78138720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 >> v1:172.21.15.6:6789/0 conn(0x7fbf78139e40 legacy=0x7fbf7813c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 monclient: found mon.noname-b 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf780a07c0 con 0x7fbf7809c110 2024-09-16T05:55:42.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf68001620 con 0x7fbf7809c110 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 5 monclient: authenticate success, global_id 164549 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: dump: 2024-09-16T05:55:42.520 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf68001880 con 0x7fbf7809c110 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 4 set_mon_vals no callback set 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 1 -- 172.21.15.6:0/1218324562 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf68001c90 con 0x7fbf7809c110 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf6ffff640 10 monclient: dump: 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.522 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: shutdown 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- 172.21.15.6:0/1218324562 >> v1:172.21.15.6:6790/0 conn(0x7fbf7809c110 legacy=0x7fbf7809c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- 172.21.15.6:0/1218324562 shutdown_connections 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- 172.21.15.6:0/1218324562 >> 172.21.15.6:0/1218324562 conn(0x7fbf780a1650 msgr2=0x7fbf780a1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- 172.21.15.6:0/1218324562 shutdown_connections 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 1 -- 172.21.15.6:0/1218324562 wait complete. 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.507+0000 7fbf7f433640 10 monclient: monmap: 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.523 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 1 Processor -- start 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 1 -- start start 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient: init 2024-09-16T05:55:42.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient(hunting): picked mon.noname-a con 0x7fbf7809c110 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient(hunting): picked mon.noname-c con 0x7fbf7809ed00 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient(hunting): picked mon.noname-b con 0x7fbf78139e40 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7811d950 con 0x7fbf7809c110 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf7811bfc0 con 0x7fbf78139e40 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf781398f0 con 0x7fbf7809ed00 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient(hunting): authenticate will time out at 2204.320761s 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7d1cf640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbf7809c110 0x7fbf7811b1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40938/0 (socket says 172.21.15.6:40938) 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7d1cf640 1 -- 172.21.15.6:0/875496526 learned_addr learned my addr 172.21.15.6:0/875496526 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2959591072 0 0) 0x7fbf7811d950 con 0x7fbf7809c110 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): my global_id is 194226 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf5c003530 con 0x7fbf7809c110 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4227713075 0 0) 0x7fbf781398f0 con 0x7fbf7809ed00 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): my global_id is 164552 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf7811d950 con 0x7fbf7809ed00 2024-09-16T05:55:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3726134840 0 0) 0x7fbf7811d950 con 0x7fbf7809ed00 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf781398f0 con 0x7fbf7809ed00 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf740017c0 con 0x7fbf7809ed00 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): dump: 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.527 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1778067189 0 0) 0x7fbf5c003530 con 0x7fbf7809c110 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf7811d950 con 0x7fbf7809c110 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf68001bd0 con 0x7fbf7809c110 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): dump: 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.528 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101049337 0 0) 0x7fbf7811bfc0 con 0x7fbf78139e40 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient(hunting): my global_id is 184327 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf5c003530 con 0x7fbf78139e40 2024-09-16T05:55:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 855351523 0 0) 0x7fbf781398f0 con 0x7fbf7809ed00 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 >> v1:172.21.15.63:6789/0 conn(0x7fbf78139e40 legacy=0x7fbf781cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 >> v1:172.21.15.6:6789/0 conn(0x7fbf7809c110 legacy=0x7fbf7811b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 monclient: found mon.c 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf780740c0 con 0x7fbf7809ed00 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 5 monclient: authenticate success, global_id 164552 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient: _renew_subs 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf78133f40 con 0x7fbf7809ed00 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient: _renew_subs 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf78139410 con 0x7fbf7809ed00 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf74003620 con 0x7fbf7809ed00 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fbf74003970 con 0x7fbf7809ed00 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: dump: 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6f7fe640 4 set_mon_vals no callback set 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbf58000f90 con 0x7fbf7809ed00 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fbf7400e790 con 0x7fbf7809ed00 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fbf74043c50 con 0x7fbf7809ed00 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.63:6808/165284194 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf5c0446d0 con 0x7fbf5c040a00 2024-09-16T05:55:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (unknown 1031510311 0 0) 0x7fbf74044060 con 0x7fbf7809ed00 2024-09-16T05:55:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf6dffb640 10 monclient: handle_get_version_reply finishing 1 version 259 2024-09-16T05:55:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7d1cf640 10 monclient: get_auth_request con 0x7fbf5c036bb0 auth_method 0 2024-09-16T05:55:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.511+0000 7fbf7d9d0640 10 monclient: get_auth_request con 0x7fbf5c040a00 auth_method 0 2024-09-16T05:55:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.515+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== osd.5 v1:172.21.15.63:6808/165284194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbf5c0446d0 con 0x7fbf5c040a00 2024-09-16T05:55:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 --> v1:172.21.15.63:6808/165284194 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbf58002cc0 con 0x7fbf5c040a00 2024-09-16T05:55:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf6dffb640 1 -- 172.21.15.6:0/875496526 <== osd.5 v1:172.21.15.63:6808/165284194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fbf58002cc0 con 0x7fbf5c040a00 2024-09-16T05:55:42.534 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-16T05:55:42.534 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T05:55:42.534 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 >> v1:172.21.15.63:6808/165284194 conn(0x7fbf5c040a00 legacy=0x7fbf5c042e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 >> v1:172.21.15.63:6812/3643207211 conn(0x7fbf5c036bb0 legacy=0x7fbf5c039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 10 monclient: shutdown 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 >> v1:172.21.15.6:6790/0 conn(0x7fbf7809ed00 legacy=0x7fbf7811b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7d9d0640 1 -- 172.21.15.6:0/875496526 reap_dead start 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 shutdown_connections 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 >> 172.21.15.6:0/875496526 conn(0x7fbf780a1650 msgr2=0x7fbf7806fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 shutdown_connections 2024-09-16T05:55:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.527+0000 7fbf7f433640 1 -- 172.21.15.6:0/875496526 wait complete. 2024-09-16T05:55:42.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T05:55:42.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient: get_monmap_and_config 2024-09-16T05:55:42.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient: monmap: 2024-09-16T05:55:42.663 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.663 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.663 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:42.664233+0000 2024-09-16T05:55:42.663 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:42.664233+0000 2024-09-16T05:55:42.663 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.664 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.664 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.664 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:42.664 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:42.664 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 1 Processor -- start 2024-09-16T05:55:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 1 -- start start 2024-09-16T05:55:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient: init 2024-09-16T05:55:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T05:55:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient(hunting): picked mon.noname-b con 0x7fc4b009ed10 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient(hunting): picked mon.noname-a con 0x7fc4b0130300 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient(hunting): picked mon.noname-c con 0x7fc4b0139de0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b009c220 con 0x7fc4b0130300 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b00a1500 con 0x7fc4b0139de0 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b00a17e0 con 0x7fc4b009ed10 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b7859640 10 monclient(hunting): authenticate will time out at 2204.471147s 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b55f5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc4b009ed10 0x7fc4b009f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53002/0 (socket says 172.21.15.6:53002) 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4b55f5640 1 -- 172.21.15.6:0/1081902983 learned_addr learned my addr 172.21.15.6:0/1081902983 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4201289656 0 0) 0x7fc4b00a17e0 con 0x7fc4b009ed10 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient(hunting): my global_id is 164555 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc49c003530 con 0x7fc4b009ed10 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3260536861 0 0) 0x7fc4b009c220 con 0x7fc4b0130300 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient(hunting): my global_id is 194229 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4b00a17e0 con 0x7fc4b0130300 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1836576656 0 0) 0x7fc4b00a17e0 con 0x7fc4b0130300 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 >> v1:172.21.15.6:6790/0 conn(0x7fc4b009ed10 legacy=0x7fc4b009f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 >> v1:172.21.15.63:6789/0 conn(0x7fc4b0139de0 legacy=0x7fc4b013c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 monclient: found mon.noname-a 2024-09-16T05:55:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.659+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b00754f0 con 0x7fc4b0130300 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4a0001620 con 0x7fc4b0130300 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 5 monclient: authenticate success, global_id 194229 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: dump: 2024-09-16T05:55:42.675 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc4a0001900 con 0x7fc4b0130300 2024-09-16T05:55:42.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 1 -- 172.21.15.6:0/1081902983 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4a0001d10 con 0x7fc4b0130300 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 4 set_mon_vals no callback set 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a7fff640 10 monclient: dump: 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.677 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: shutdown 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- 172.21.15.6:0/1081902983 >> v1:172.21.15.6:6789/0 conn(0x7fc4b0130300 legacy=0x7fc4b01386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- 172.21.15.6:0/1081902983 shutdown_connections 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- 172.21.15.6:0/1081902983 >> 172.21.15.6:0/1081902983 conn(0x7fc4b00a1f00 msgr2=0x7fc4b00a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- 172.21.15.6:0/1081902983 shutdown_connections 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- 172.21.15.6:0/1081902983 wait complete. 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: monmap: 2024-09-16T05:55:42.678 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 Processor -- start 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- start start 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: init 2024-09-16T05:55:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient(hunting): picked mon.noname-b con 0x7fc4b009ed10 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient(hunting): picked mon.noname-a con 0x7fc4b0130300 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient(hunting): picked mon.noname-c con 0x7fc4b0139de0 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b0134760 con 0x7fc4b0130300 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b0133590 con 0x7fc4b009ed10 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4b01391a0 con 0x7fc4b0139de0 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient(hunting): authenticate will time out at 2204.474207s 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b4df4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc4b0130300 0x7fc4b01cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40954/0 (socket says 172.21.15.6:40954) 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b5df6640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc4b0139de0 0x7fc4b01cf8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53010/0 (socket says 172.21.15.6:53010) 2024-09-16T05:55:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b4df4640 1 -- 172.21.15.6:0/3263854371 learned_addr learned my addr 172.21.15.6:0/3263854371 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962031984 0 0) 0x7fc4b0134760 con 0x7fc4b0130300 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): my global_id is 194232 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc494003530 con 0x7fc4b0130300 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453432073 0 0) 0x7fc4b01391a0 con 0x7fc4b0139de0 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): my global_id is 164558 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4b0134760 con 0x7fc4b0139de0 2024-09-16T05:55:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2672551322 0 0) 0x7fc4b0133590 con 0x7fc4b009ed10 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): my global_id is 184333 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4b01391a0 con 0x7fc4b009ed10 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3250432811 0 0) 0x7fc494003530 con 0x7fc4b0130300 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc4b0133590 con 0x7fc4b0130300 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2521352195 0 0) 0x7fc4b0134760 con 0x7fc4b0139de0 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc494003530 con 0x7fc4b0139de0 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4a0001c50 con 0x7fc4b0130300 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:55:42.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): dump: 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4a8001b60 con 0x7fc4b0139de0 2024-09-16T05:55:42.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient(hunting): dump: 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1399476251 0 0) 0x7fc4b0133590 con 0x7fc4b0130300 2024-09-16T05:55:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 >> v1:172.21.15.6:6790/0 conn(0x7fc4b0139de0 legacy=0x7fc4b01cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 >> v1:172.21.15.63:6789/0 conn(0x7fc4b009ed10 legacy=0x7fc4b012cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 monclient: found mon.a 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b009ce70 con 0x7fc4b0130300 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 5 monclient: authenticate success, global_id 194232 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: _renew_subs 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b0132a30 con 0x7fc4b0130300 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: _renew_subs 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc4b0131f20 con 0x7fc4b0130300 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc4a0001900 con 0x7fc4b0130300 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fc4a0003b50 con 0x7fc4b0130300 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a77fe640 4 set_mon_vals no callback set 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.663+0000 7fc4a5ffb640 10 monclient: dump: 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.667+0000 7fc4b7859640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.667+0000 7fc4b7859640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.667+0000 7fc4b7859640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.667+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc490005090 con 0x7fc4b0130300 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.667+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fc4a0003d70 con 0x7fc4b0130300 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.667+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fc4a0044ec0 con 0x7fc4b0130300 2024-09-16T05:55:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.667+0000 7fc4b55f5640 10 monclient: get_auth_request con 0x7fc494036bc0 auth_method 0 2024-09-16T05:55:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.671+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fc4a0015550 con 0x7fc4b0130300 2024-09-16T05:55:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.671+0000 7fc4a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.671+0000 7fc4a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.835+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fc490000ba0 con 0x7fc494036bc0 2024-09-16T05:55:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4a5ffb640 1 -- 172.21.15.6:0/3263854371 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7fc490000ba0 con 0x7fc494036bc0 2024-09-16T05:55:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T05:55:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 >> v1:172.21.15.63:6812/3643207211 conn(0x7fc494036bc0 legacy=0x7fc494039080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4b7859640 10 monclient: shutdown 2024-09-16T05:55:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4b7859640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 >> v1:172.21.15.6:6789/0 conn(0x7fc4b0130300 legacy=0x7fc4b01cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 shutdown_connections 2024-09-16T05:55:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 >> 172.21.15.6:0/3263854371 conn(0x7fc4b00a1f00 msgr2=0x7fc4b009db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 shutdown_connections 2024-09-16T05:55:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.839+0000 7fc4b7859640 1 -- 172.21.15.6:0/3263854371 wait complete. 2024-09-16T05:55:42.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:55:42.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.971+0000 7f1a48ca7640 10 monclient: get_monmap_and_config 2024-09-16T05:55:42.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.971+0000 7f1a48ca7640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.971+0000 7f1a48ca7640 10 monclient: monmap: 2024-09-16T05:55:42.976 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.977 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.977 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:55:42.977826+0000 2024-09-16T05:55:42.977 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:55:42.977826+0000 2024-09-16T05:55:42.977 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.977 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.977 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.977 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:55:42.978 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:55:42.978 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.971+0000 7f1a48ca7640 1 Processor -- start 2024-09-16T05:55:42.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.971+0000 7f1a48ca7640 1 -- start start 2024-09-16T05:55:42.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.971+0000 7f1a48ca7640 10 monclient: init 2024-09-16T05:55:42.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:55:42.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): picked mon.noname-a con 0x7f1a4409aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): picked mon.noname-b con 0x7f1a4409e230 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): picked mon.noname-c con 0x7f1a440a0da0 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a440a1370 con 0x7f1a4409aa30 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a4409ead0 con 0x7f1a440a0da0 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a4409edb0 con 0x7f1a4409e230 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): authenticate will time out at 2204.784653s 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a41d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1a4409e230 0x7f1a441399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53020/0 (socket says 172.21.15.6:53020) 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a41d9b640 1 -- 172.21.15.6:0/2091812519 learned_addr learned my addr 172.21.15.6:0/2091812519 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4259c640 1 --1- 172.21.15.6:0/2091812519 >> v1:172.21.15.6:6789/0 conn(0x7f1a4409aa30 0x7f1a4409ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40960/0 (socket says 172.21.15.6:40960) 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1667598690 0 0) 0x7f1a4409edb0 con 0x7f1a4409e230 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient(hunting): my global_id is 164561 2024-09-16T05:55:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a30003530 con 0x7f1a4409e230 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 834483584 0 0) 0x7f1a440a1370 con 0x7f1a4409aa30 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient(hunting): my global_id is 194235 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a4409edb0 con 0x7f1a4409aa30 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3933457513 0 0) 0x7f1a30003530 con 0x7f1a4409e230 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 >> v1:172.21.15.63:6789/0 conn(0x7f1a440a0da0 legacy=0x7f1a4413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 >> v1:172.21.15.6:6789/0 conn(0x7f1a4409aa30 legacy=0x7f1a4409ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 monclient: found mon.noname-b 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4409dd50 con 0x7f1a4409e230 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1a2c001620 con 0x7f1a4409e230 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: dump: 2024-09-16T05:55:42.988 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 5 monclient: authenticate success, global_id 164561 2024-09-16T05:55:42.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a2c001810 con 0x7f1a4409e230 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 1 -- 172.21.15.6:0/2091812519 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1a2c001be0 con 0x7f1a4409e230 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4159a640 10 monclient: dump: 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.990 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: get_monmap_and_config success 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 4 set_mon_vals no callback set 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: shutdown 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- 172.21.15.6:0/2091812519 >> v1:172.21.15.6:6790/0 conn(0x7f1a4409e230 legacy=0x7f1a441399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- 172.21.15.6:0/2091812519 shutdown_connections 2024-09-16T05:55:42.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- 172.21.15.6:0/2091812519 >> 172.21.15.6:0/2091812519 conn(0x7f1a440a1650 msgr2=0x7f1a440a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- 172.21.15.6:0/2091812519 shutdown_connections 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- 172.21.15.6:0/2091812519 wait complete. 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: build_initial_monmap 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: monmap: 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:55:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 Processor -- start 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- start start 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: init 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: _reopen_session rank -1 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): picked mon.noname-c con 0x7f1a4409aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): picked mon.noname-b con 0x7f1a4409e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): picked mon.noname-a con 0x7f1a440a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a4409c990 con 0x7f1a440a0da0 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a4413b020 con 0x7f1a4409e230 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a4413ce70 con 0x7f1a4409aa30 2024-09-16T05:55:42.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): _renew_subs 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a48ca7640 10 monclient(hunting): authenticate will time out at 2204.787700s 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4259c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1a4409aa30 0x7f1a4409bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53032/0 (socket says 172.21.15.6:53032) 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.975+0000 7f1a4259c640 1 -- 172.21.15.6:0/1888472262 learned_addr learned my addr 172.21.15.6:0/1888472262 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2538619283 0 0) 0x7f1a4413ce70 con 0x7f1a4409aa30 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): my global_id is 164564 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a24003530 con 0x7f1a4409aa30 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2679370868 0 0) 0x7f1a4409c990 con 0x7f1a440a0da0 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): my global_id is 194238 2024-09-16T05:55:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a4413ce70 con 0x7f1a440a0da0 2024-09-16T05:55:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 57549164 0 0) 0x7f1a24003530 con 0x7f1a4409aa30 2024-09-16T05:55:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a4409c990 con 0x7f1a4409aa30 2024-09-16T05:55:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1a340016f0 con 0x7f1a4409aa30 2024-09-16T05:55:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T05:55:42.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): dump: 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 433023575 0 0) 0x7f1a4413ce70 con 0x7f1a440a0da0 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a24003530 con 0x7f1a440a0da0 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1a3c001800 con 0x7f1a440a0da0 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): dump: 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:42.997 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352545881 0 0) 0x7f1a4413b020 con 0x7f1a4409e230 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _negotiate 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient(hunting): my global_id is 184339 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a4413ce70 con 0x7f1a4409e230 2024-09-16T05:55:42.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2187929501 0 0) 0x7f1a4409c990 con 0x7f1a4409aa30 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 >> v1:172.21.15.63:6789/0 conn(0x7f1a4409e230 legacy=0x7f1a4409c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 >> v1:172.21.15.6:6789/0 conn(0x7f1a440a0da0 legacy=0x7f1a441d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient: _finish_hunting 0 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 monclient: found mon.c 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a440740c0 con 0x7f1a4409aa30 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient: _finish_auth 0 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient: _check_auth_tickets 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 5 monclient: authenticate success, global_id 164564 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 10 monclient: _renew_subs 2024-09-16T05:55:42.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4413c8b0 con 0x7f1a4409aa30 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 10 monclient: _renew_subs 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a4413d5f0 con 0x7f1a4409aa30 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a34004b70 con 0x7f1a4409aa30 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7f1a34004ec0 con 0x7f1a4409aa30 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a40d99640 4 set_mon_vals no callback set 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 10 monclient: dump: 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:55:43.000 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7f1a3400fc60 con 0x7f1a4409aa30 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a28005090 con 0x7f1a4409aa30 2024-09-16T05:55:43.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7f1a340451d0 con 0x7f1a4409aa30 2024-09-16T05:55:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.979+0000 7f1a41d9b640 10 monclient: get_auth_request con 0x7f1a24036c70 auth_method 0 2024-09-16T05:55:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.983+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7f1a340157a0 con 0x7f1a4409aa30 2024-09-16T05:55:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.983+0000 7f1a3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:55:43.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:42.983+0000 7f1a3affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:55:43.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.147+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a28000ba0 con 0x7f1a24036c70 2024-09-16T05:55:43.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.151+0000 7f1a3affd640 1 -- 172.21.15.6:0/1888472262 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77554 (unknown 2965378022 0 4196536367) 0x7f1a28000ba0 con 0x7f1a24036c70 2024-09-16T05:55:43.157 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:55:43.157 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:55:43.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.155+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 >> v1:172.21.15.63:6812/3643207211 conn(0x7f1a24036c70 legacy=0x7f1a24039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:43.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.155+0000 7f1a48ca7640 10 monclient: shutdown 2024-09-16T05:55:43.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.155+0000 7f1a48ca7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:55:43.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.155+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 >> v1:172.21.15.6:6790/0 conn(0x7f1a4409aa30 legacy=0x7f1a4409bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:55:43.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.155+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 shutdown_connections 2024-09-16T05:55:43.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.155+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 >> 172.21.15.6:0/1888472262 conn(0x7f1a440a1650 msgr2=0x7f1a44071d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:55:43.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.155+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 shutdown_connections 2024-09-16T05:55:43.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:55:43.155+0000 7f1a48ca7640 1 -- 172.21.15.6:0/1888472262 wait complete. 2024-09-16T05:55:43.175 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":882,"stamp":"2024-09-16T05:55:41.627969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205768,"kb_used_data":18640,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566025272,"statfs":{"total":579820584960,"available":579609878528,"internally_reserved":0,"allocated":19087360,"data_stored":15446128,"data_compressed":1794170,"data_compressed_allocated":2719744,"data_compressed_original":5914624,"omap_allocated":9566,"internal_metadata":191486626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003860"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":289,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:01.384560+0000","last_change":"2024-09-16T05:39:26.074706+0000","last_active":"2024-09-16T05:55:01.384560+0000","last_peered":"2024-09-16T05:55:01.384560+0000","last_clean":"2024-09-16T05:55:01.384560+0000","last_became_active":"2024-09-16T05:39:26.073689+0000","last_became_peered":"2024-09-16T05:39:26.073689+0000","last_unstale":"2024-09-16T05:55:01.384560+0000","last_undegraded":"2024-09-16T05:55:01.384560+0000","last_fullsized":"2024-09-16T05:55:01.384560+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:27.376074+0000","last_change":"2024-09-16T05:39:26.053781+0000","last_active":"2024-09-16T05:52:27.376074+0000","last_peered":"2024-09-16T05:52:27.376074+0000","last_clean":"2024-09-16T05:52:27.376074+0000","last_became_active":"2024-09-16T05:39:26.053397+0000","last_became_peered":"2024-09-16T05:39:26.053397+0000","last_unstale":"2024-09-16T05:52:27.376074+0000","last_undegraded":"2024-09-16T05:52:27.376074+0000","last_fullsized":"2024-09-16T05:52:27.376074+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:58.904858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":295,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:01.496660+0000","last_change":"2024-09-16T05:39:26.070406+0000","last_active":"2024-09-16T05:55:01.496660+0000","last_peered":"2024-09-16T05:55:01.496660+0000","last_clean":"2024-09-16T05:55:01.496660+0000","last_became_active":"2024-09-16T05:39:26.069114+0000","last_became_peered":"2024-09-16T05:39:26.069114+0000","last_unstale":"2024-09-16T05:55:01.496660+0000","last_undegraded":"2024-09-16T05:55:01.496660+0000","last_fullsized":"2024-09-16T05:55:01.496660+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:27.376003+0000","last_change":"2024-09-16T05:39:26.056400+0000","last_active":"2024-09-16T05:52:27.376003+0000","last_peered":"2024-09-16T05:52:27.376003+0000","last_clean":"2024-09-16T05:52:27.376003+0000","last_became_active":"2024-09-16T05:39:26.055883+0000","last_became_peered":"2024-09-16T05:39:26.055883+0000","last_unstale":"2024-09-16T05:52:27.376003+0000","last_undegraded":"2024-09-16T05:52:27.376003+0000","last_fullsized":"2024-09-16T05:52:27.376003+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:38.780469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":224,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:22.364898+0000","last_change":"2024-09-16T05:39:27.057197+0000","last_active":"2024-09-16T05:51:22.364898+0000","last_peered":"2024-09-16T05:51:22.364898+0000","last_clean":"2024-09-16T05:51:22.364898+0000","last_became_active":"2024-09-16T05:39:27.056198+0000","last_became_peered":"2024-09-16T05:39:27.056198+0000","last_unstale":"2024-09-16T05:51:22.364898+0000","last_undegraded":"2024-09-16T05:51:22.364898+0000","last_fullsized":"2024-09-16T05:51:22.364898+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:54.298350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":277,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:06.387942+0000","last_change":"2024-09-16T05:39:26.074867+0000","last_active":"2024-09-16T05:55:06.387942+0000","last_peered":"2024-09-16T05:55:06.387942+0000","last_clean":"2024-09-16T05:55:06.387942+0000","last_became_active":"2024-09-16T05:39:26.073067+0000","last_became_peered":"2024-09-16T05:39:26.073067+0000","last_unstale":"2024-09-16T05:55:06.387942+0000","last_undegraded":"2024-09-16T05:55:06.387942+0000","last_fullsized":"2024-09-16T05:55:06.387942+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":176,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.362499+0000","last_change":"2024-09-16T05:39:26.061744+0000","last_active":"2024-09-16T05:52:26.362499+0000","last_peered":"2024-09-16T05:52:26.362499+0000","last_clean":"2024-09-16T05:52:26.362499+0000","last_became_active":"2024-09-16T05:39:26.061165+0000","last_became_peered":"2024-09-16T05:39:26.061165+0000","last_unstale":"2024-09-16T05:52:26.362499+0000","last_undegraded":"2024-09-16T05:52:26.362499+0000","last_fullsized":"2024-09-16T05:52:26.362499+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:26.707122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:54:01.373619+0000","last_change":"2024-09-16T05:39:26.053923+0000","last_active":"2024-09-16T05:54:01.373619+0000","last_peered":"2024-09-16T05:54:01.373619+0000","last_clean":"2024-09-16T05:54:01.373619+0000","last_became_active":"2024-09-16T05:39:26.053013+0000","last_became_peered":"2024-09-16T05:39:26.053013+0000","last_unstale":"2024-09-16T05:54:01.373619+0000","last_undegraded":"2024-09-16T05:54:01.373619+0000","last_fullsized":"2024-09-16T05:54:01.373619+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":209,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:37.377442+0000","last_change":"2024-09-16T05:39:26.059457+0000","last_active":"2024-09-16T05:52:37.377442+0000","last_peered":"2024-09-16T05:52:37.377442+0000","last_clean":"2024-09-16T05:52:37.377442+0000","last_became_active":"2024-09-16T05:39:26.058997+0000","last_became_peered":"2024-09-16T05:39:26.058997+0000","last_unstale":"2024-09-16T05:52:37.377442+0000","last_undegraded":"2024-09-16T05:52:37.377442+0000","last_fullsized":"2024-09-16T05:52:37.377442+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:25.563929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":213,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:06.368302+0000","last_change":"2024-09-16T05:39:26.049544+0000","last_active":"2024-09-16T05:53:06.368302+0000","last_peered":"2024-09-16T05:53:06.368302+0000","last_clean":"2024-09-16T05:53:06.368302+0000","last_became_active":"2024-09-16T05:39:26.049092+0000","last_became_peered":"2024-09-16T05:39:26.049092+0000","last_unstale":"2024-09-16T05:53:06.368302+0000","last_undegraded":"2024-09-16T05:53:06.368302+0000","last_fullsized":"2024-09-16T05:53:06.368302+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:29.953136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":226,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:57.376805+0000","last_change":"2024-09-16T05:39:26.057943+0000","last_active":"2024-09-16T05:52:57.376805+0000","last_peered":"2024-09-16T05:52:57.376805+0000","last_clean":"2024-09-16T05:52:57.376805+0000","last_became_active":"2024-09-16T05:39:26.057564+0000","last_became_peered":"2024-09-16T05:39:26.057564+0000","last_unstale":"2024-09-16T05:52:57.376805+0000","last_undegraded":"2024-09-16T05:52:57.376805+0000","last_fullsized":"2024-09-16T05:52:57.376805+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:01:19.835552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:54:01.373559+0000","last_change":"2024-09-16T05:39:26.060420+0000","last_active":"2024-09-16T05:54:01.373559+0000","last_peered":"2024-09-16T05:54:01.373559+0000","last_clean":"2024-09-16T05:54:01.373559+0000","last_became_active":"2024-09-16T05:39:26.059546+0000","last_became_peered":"2024-09-16T05:39:26.059546+0000","last_unstale":"2024-09-16T05:54:01.373559+0000","last_undegraded":"2024-09-16T05:54:01.373559+0000","last_fullsized":"2024-09-16T05:54:01.373559+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":173,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:54:01.373595+0000","last_change":"2024-09-16T05:39:26.074051+0000","last_active":"2024-09-16T05:54:01.373595+0000","last_peered":"2024-09-16T05:54:01.373595+0000","last_clean":"2024-09-16T05:54:01.373595+0000","last_became_active":"2024-09-16T05:39:26.073016+0000","last_became_peered":"2024-09-16T05:39:26.073016+0000","last_unstale":"2024-09-16T05:54:01.373595+0000","last_undegraded":"2024-09-16T05:54:01.373595+0000","last_fullsized":"2024-09-16T05:54:01.373595+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":290,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:01.384522+0000","last_change":"2024-09-16T05:39:26.077800+0000","last_active":"2024-09-16T05:55:01.384522+0000","last_peered":"2024-09-16T05:55:01.384522+0000","last_clean":"2024-09-16T05:55:01.384522+0000","last_became_active":"2024-09-16T05:39:26.076818+0000","last_became_peered":"2024-09-16T05:39:26.076818+0000","last_unstale":"2024-09-16T05:55:01.384522+0000","last_undegraded":"2024-09-16T05:55:01.384522+0000","last_fullsized":"2024-09-16T05:55:01.384522+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:16.505345+0000","last_clean_scrub_stamp":"2024-09-16T05:38:16.505345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":185,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:54:01.488785+0000","last_change":"2024-09-16T05:39:27.051741+0000","last_active":"2024-09-16T05:54:01.488785+0000","last_peered":"2024-09-16T05:54:01.488785+0000","last_clean":"2024-09-16T05:54:01.488785+0000","last_became_active":"2024-09-16T05:39:27.050732+0000","last_became_peered":"2024-09-16T05:39:27.050732+0000","last_unstale":"2024-09-16T05:54:01.488785+0000","last_undegraded":"2024-09-16T05:54:01.488785+0000","last_fullsized":"2024-09-16T05:54:01.488785+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:38:17.518279+0000","last_clean_scrub_stamp":"2024-09-16T05:38:17.518279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":181,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:41.154246+0000","last_change":"2024-09-16T05:55:41.154265+0000","last_active":"2024-09-16T05:55:41.154246+0000","last_peered":"2024-09-16T05:55:41.154246+0000","last_clean":"2024-09-16T05:55:41.154246+0000","last_became_active":"2024-09-16T05:39:26.052209+0000","last_became_peered":"2024-09-16T05:39:26.052209+0000","last_unstale":"2024-09-16T05:55:41.154246+0000","last_undegraded":"2024-09-16T05:55:41.154246+0000","last_fullsized":"2024-09-16T05:55:41.154246+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:41.154180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:41.154180+0000","last_clean_scrub_stamp":"2024-09-16T05:55:41.154180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:00:39.249246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":591,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:56.366813+0000","last_change":"2024-09-16T05:39:26.061773+0000","last_active":"2024-09-16T05:52:56.366813+0000","last_peered":"2024-09-16T05:52:56.366813+0000","last_clean":"2024-09-16T05:52:56.366813+0000","last_became_active":"2024-09-16T05:39:26.061101+0000","last_became_peered":"2024-09-16T05:39:26.061101+0000","last_unstale":"2024-09-16T05:52:56.366813+0000","last_undegraded":"2024-09-16T05:52:56.366813+0000","last_fullsized":"2024-09-16T05:52:56.366813+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:47.359645+0000","last_clean_scrub_stamp":"2024-09-16T05:34:47.359645+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:56:28.349578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":441,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:57.369715+0000","last_change":"2024-09-16T05:39:26.073172+0000","last_active":"2024-09-16T05:51:57.369715+0000","last_peered":"2024-09-16T05:51:57.369715+0000","last_clean":"2024-09-16T05:51:57.369715+0000","last_became_active":"2024-09-16T05:39:26.063463+0000","last_became_peered":"2024-09-16T05:39:26.063463+0000","last_unstale":"2024-09-16T05:51:57.369715+0000","last_undegraded":"2024-09-16T05:51:57.369715+0000","last_fullsized":"2024-09-16T05:51:57.369715+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:08.651405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:53:01.367585+0000","last_change":"2024-09-16T05:39:26.060982+0000","last_active":"2024-09-16T05:53:01.367585+0000","last_peered":"2024-09-16T05:53:01.367585+0000","last_clean":"2024-09-16T05:53:01.367585+0000","last_became_active":"2024-09-16T05:39:26.060550+0000","last_became_peered":"2024-09-16T05:39:26.060550+0000","last_unstale":"2024-09-16T05:53:01.367585+0000","last_undegraded":"2024-09-16T05:53:01.367585+0000","last_fullsized":"2024-09-16T05:53:01.367585+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:07:51.890578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":186,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:54:01.488749+0000","last_change":"2024-09-16T05:39:26.059723+0000","last_active":"2024-09-16T05:54:01.488749+0000","last_peered":"2024-09-16T05:54:01.488749+0000","last_clean":"2024-09-16T05:54:01.488749+0000","last_became_active":"2024-09-16T05:39:26.058771+0000","last_became_peered":"2024-09-16T05:39:26.058771+0000","last_unstale":"2024-09-16T05:54:01.488749+0000","last_undegraded":"2024-09-16T05:54:01.488749+0000","last_fullsized":"2024-09-16T05:54:01.488749+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":450,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:51:57.369673+0000","last_change":"2024-09-16T05:39:26.080898+0000","last_active":"2024-09-16T05:51:57.369673+0000","last_peered":"2024-09-16T05:51:57.369673+0000","last_clean":"2024-09-16T05:51:57.369673+0000","last_became_active":"2024-09-16T05:39:26.065589+0000","last_became_peered":"2024-09-16T05:39:26.065589+0000","last_unstale":"2024-09-16T05:51:57.369673+0000","last_undegraded":"2024-09-16T05:51:57.369673+0000","last_fullsized":"2024-09-16T05:51:57.369673+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:54.750367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":431,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:54:01.373637+0000","last_change":"2024-09-16T05:39:27.057353+0000","last_active":"2024-09-16T05:54:01.373637+0000","last_peered":"2024-09-16T05:54:01.373637+0000","last_clean":"2024-09-16T05:54:01.373637+0000","last_became_active":"2024-09-16T05:39:27.056262+0000","last_became_peered":"2024-09-16T05:39:27.056262+0000","last_unstale":"2024-09-16T05:54:01.373637+0000","last_undegraded":"2024-09-16T05:54:01.373637+0000","last_fullsized":"2024-09-16T05:54:01.373637+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:01.358717+0000","last_change":"2024-09-16T05:39:26.060534+0000","last_active":"2024-09-16T05:52:01.358717+0000","last_peered":"2024-09-16T05:52:01.358717+0000","last_clean":"2024-09-16T05:52:01.358717+0000","last_became_active":"2024-09-16T05:39:26.060029+0000","last_became_peered":"2024-09-16T05:39:26.060029+0000","last_unstale":"2024-09-16T05:52:01.358717+0000","last_undegraded":"2024-09-16T05:52:01.358717+0000","last_fullsized":"2024-09-16T05:52:01.358717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:47.905126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":172,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:54:36.378552+0000","last_change":"2024-09-16T05:39:26.055632+0000","last_active":"2024-09-16T05:54:36.378552+0000","last_peered":"2024-09-16T05:54:36.378552+0000","last_clean":"2024-09-16T05:54:36.378552+0000","last_became_active":"2024-09-16T05:39:26.054759+0000","last_became_peered":"2024-09-16T05:39:26.054759+0000","last_unstale":"2024-09-16T05:54:36.378552+0000","last_undegraded":"2024-09-16T05:54:36.378552+0000","last_fullsized":"2024-09-16T05:54:36.378552+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":175,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:52:26.362449+0000","last_change":"2024-09-16T05:39:26.051119+0000","last_active":"2024-09-16T05:52:26.362449+0000","last_peered":"2024-09-16T05:52:26.362449+0000","last_clean":"2024-09-16T05:52:26.362449+0000","last_became_active":"2024-09-16T05:39:26.050738+0000","last_became_peered":"2024-09-16T05:39:26.050738+0000","last_unstale":"2024-09-16T05:52:26.362449+0000","last_undegraded":"2024-09-16T05:52:26.362449+0000","last_fullsized":"2024-09-16T05:52:26.362449+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:34:53.884010+0000","last_clean_scrub_stamp":"2024-09-16T05:34:53.884010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:37.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607807,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640510,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34328,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337512,"statfs":{"total":96636764160,"available":96601612288,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640510,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640509,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34280,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337560,"statfs":{"total":96636764160,"available":96601661440,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640512,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34328,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337512,"statfs":{"total":96636764160,"available":96601612288,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:55:43.179 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-09-16T05:38:17.518279+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.180 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-16T05:38:16.505345+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.180 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2024-09-16T05:38:17.518279+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.180 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-16T05:38:16.505345+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.180 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-16T05:38:17.518279+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.180 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-16T05:38:16.505345+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.180 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-09-16T05:38:17.518279+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.180 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-16T05:38:16.505345+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.180 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-09-16T05:38:17.518279+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-16T05:38:16.505345+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-09-16T05:38:17.518279+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-16T05:38:16.505345+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-09-16T05:38:17.518279+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-09-16T05:38:16.505345+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-09-16T05:38:17.518279+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=38, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T05:34:47.359645+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=47, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T05:34:53.884010+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.181 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T05:34:53.884010+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.182 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T05:34:53.884010+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.182 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T05:34:53.884010+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.182 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T05:34:53.884010+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.182 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T05:34:53.884010+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.182 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T05:34:53.884010+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.182 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T05:34:53.884010+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=34, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=5, tm_min=55, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T05:55:43.182 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T05:56:03.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T05:56:03.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.310+0000 7fd22c6a4640 10 monclient: get_monmap_and_config 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.310+0000 7fd22c6a4640 10 monclient: build_initial_monmap 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.310+0000 7fd22c6a4640 10 monclient: monmap: 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:56:03.318286+0000 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:56:03.318286+0000 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:56:03.317 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:56:03.318 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 Processor -- start 2024-09-16T05:56:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- start start 2024-09-16T05:56:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient: init 2024-09-16T05:56:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient: _reopen_session rank -1 2024-09-16T05:56:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T05:56:03.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient(hunting): picked mon.noname-a con 0x7fd2240a37a0 addr v1:172.21.15.6:6789/0 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient(hunting): picked mon.noname-b con 0x7fd2240a4290 addr v1:172.21.15.6:6790/0 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient(hunting): picked mon.noname-c con 0x7fd2240a1a00 addr v1:172.21.15.63:6789/0 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22406e520 con 0x7fd2240a37a0 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2240a0780 con 0x7fd2240a1a00 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2240a4c70 con 0x7fd2240a4290 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient(hunting): _renew_subs 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient(hunting): authenticate will time out at 2225.125314s 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22a440640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd2240a37a0 0x7fd2240a3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43826/0 (socket says 172.21.15.6:43826) 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22a440640 1 -- 172.21.15.6:0/103823993 learned_addr learned my addr 172.21.15.6:0/103823993 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890340645 0 0) 0x7fd22406e520 con 0x7fd2240a37a0 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient(hunting): my global_id is 194241 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd210003530 con 0x7fd2240a37a0 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833939709 0 0) 0x7fd2240a4c70 con 0x7fd2240a4290 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient(hunting): my global_id is 164567 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd22406e520 con 0x7fd2240a4290 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4081258331 0 0) 0x7fd22406e520 con 0x7fd2240a4290 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 >> v1:172.21.15.63:6789/0 conn(0x7fd2240a1a00 legacy=0x7fd22413f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 >> v1:172.21.15.6:6789/0 conn(0x7fd2240a37a0 legacy=0x7fd2240a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: _finish_hunting 0 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 monclient: found mon.noname-b 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2240a65f0 con 0x7fd2240a4290 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: _finish_auth 0 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: _check_auth_tickets 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 5 monclient: authenticate success, global_id 164567 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd2140015a0 con 0x7fd2240a4290 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: dump: 2024-09-16T05:56:03.327 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd214001800 con 0x7fd2240a4290 2024-09-16T05:56:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: handle_config config(0 keys) 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient: get_monmap_and_config success 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 1 -- 172.21.15.6:0/103823993 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd214001bd0 con 0x7fd2240a4290 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 4 set_mon_vals no callback set 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22943e640 10 monclient: dump: 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient: shutdown 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- 172.21.15.6:0/103823993 >> v1:172.21.15.6:6790/0 conn(0x7fd2240a4290 legacy=0x7fd22413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- 172.21.15.6:0/103823993 shutdown_connections 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- 172.21.15.6:0/103823993 >> 172.21.15.6:0/103823993 conn(0x7fd2240a58a0 msgr2=0x7fd2240a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:56:03.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- 172.21.15.6:0/103823993 shutdown_connections 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- 172.21.15.6:0/103823993 wait complete. 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient: build_initial_monmap 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient: monmap: 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:56:03.331 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 Processor -- start 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 1 -- start start 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.314+0000 7fd22c6a4640 10 monclient: init 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient: _reopen_session rank -1 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient(hunting): picked mon.noname-c con 0x7fd2240a1a00 addr v1:172.21.15.6:6790/0 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient(hunting): picked mon.noname-b con 0x7fd2240a37a0 addr v1:172.21.15.63:6789/0 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient(hunting): picked mon.noname-a con 0x7fd2240a4290 addr v1:172.21.15.6:6789/0 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd224132b30 con 0x7fd2240a4290 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd224133cd0 con 0x7fd2240a37a0 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd224135240 con 0x7fd2240a1a00 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient(hunting): _renew_subs 2024-09-16T05:56:03.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient(hunting): authenticate will time out at 2225.128444s 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22ac41640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd2240a4290 0x7fd2241d3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43832/0 (socket says 172.21.15.6:43832) 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22ac41640 1 -- 172.21.15.6:0/319347877 learned_addr learned my addr 172.21.15.6:0/319347877 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22a440640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fd2240a1a00 0x7fd224131d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53254/0 (socket says 172.21.15.6:53254) 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3564974264 0 0) 0x7fd224132b30 con 0x7fd2240a4290 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): my global_id is 194244 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd208003530 con 0x7fd2240a4290 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3574694543 0 0) 0x7fd224135240 con 0x7fd2240a1a00 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): my global_id is 164570 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd224132b30 con 0x7fd2240a1a00 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3140114258 0 0) 0x7fd208003530 con 0x7fd2240a4290 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd224135240 con 0x7fd2240a4290 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3242560142 0 0) 0x7fd224132b30 con 0x7fd2240a1a00 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd208003530 con 0x7fd2240a1a00 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2269210887 0 0) 0x7fd224133cd0 con 0x7fd2240a37a0 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): my global_id is 184345 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd224132b30 con 0x7fd2240a37a0 2024-09-16T05:56:03.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd21c001c80 con 0x7fd2240a4290 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): dump: 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd220001100 con 0x7fd2240a1a00 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient(hunting): dump: 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.336 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3082449078 0 0) 0x7fd224135240 con 0x7fd2240a4290 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 >> v1:172.21.15.6:6790/0 conn(0x7fd2240a1a00 legacy=0x7fd224131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 >> v1:172.21.15.63:6789/0 conn(0x7fd2240a37a0 legacy=0x7fd224132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient: _finish_hunting 0 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 monclient: found mon.a 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd224135800 con 0x7fd2240a4290 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient: _finish_auth 0 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient: _check_auth_tickets 2024-09-16T05:56:03.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 5 monclient: authenticate success, global_id 194244 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient: _renew_subs 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd224134570 con 0x7fd2240a4290 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient: _renew_subs 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd224133a80 con 0x7fd2240a4290 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd21c002000 con 0x7fd2240a4290 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient: handle_config config(0 keys) 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fd21c003df0 con 0x7fd2240a4290 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:56:03.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 10 monclient: dump: 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd228c3d640 4 set_mon_vals no callback set 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:56:03.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:56:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20c005090 con 0x7fd2240a4290 2024-09-16T05:56:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fd21c003f90 con 0x7fd2240a4290 2024-09-16T05:56:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fd21c045160 con 0x7fd2240a4290 2024-09-16T05:56:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.318+0000 7fd22a440640 10 monclient: get_auth_request con 0x7fd208036c50 auth_method 0 2024-09-16T05:56:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.322+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fd224133a80 con 0x7fd2240a4290 2024-09-16T05:56:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.322+0000 7fd21affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:56:03.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.322+0000 7fd21affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:56:03.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.482+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 --> v1:172.21.15.63:6812/3643207211 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd20c000ba0 con 0x7fd208036c50 2024-09-16T05:56:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd21affd640 1 -- 172.21.15.6:0/319347877 <== mgr.4105 v1:172.21.15.63:6812/3643207211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77980 (unknown 2965378022 0 1206932759) 0x7fd20c000ba0 con 0x7fd208036c50 2024-09-16T05:56:03.495 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-16T05:56:03.496 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-16T05:56:03.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 >> v1:172.21.15.63:6812/3643207211 conn(0x7fd208036c50 legacy=0x7fd208039110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd22c6a4640 10 monclient: shutdown 2024-09-16T05:56:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd22c6a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:56:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 >> v1:172.21.15.6:6789/0 conn(0x7fd2240a4290 legacy=0x7fd2241d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 shutdown_connections 2024-09-16T05:56:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 >> 172.21.15.6:0/319347877 conn(0x7fd2240a58a0 msgr2=0x7fd2240a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:56:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 shutdown_connections 2024-09-16T05:56:03.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.494+0000 7fd22c6a4640 1 -- 172.21.15.6:0/319347877 wait complete. 2024-09-16T05:56:03.512 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":892,"stamp":"2024-09-16T05:56:01.634298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205744,"kb_used_data":18640,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":566025296,"statfs":{"total":579820584960,"available":579609903104,"internally_reserved":0,"allocated":19087360,"data_stored":15446128,"data_compressed":1794170,"data_compressed_allocated":2719744,"data_compressed_original":5914624,"omap_allocated":9566,"internal_metadata":191486626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003727"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":297,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:41.419799+0000","last_change":"2024-09-16T05:55:41.419928+0000","last_active":"2024-09-16T05:55:41.419799+0000","last_peered":"2024-09-16T05:55:41.419799+0000","last_clean":"2024-09-16T05:55:41.419799+0000","last_became_active":"2024-09-16T05:39:26.073689+0000","last_became_peered":"2024-09-16T05:39:26.073689+0000","last_unstale":"2024-09-16T05:55:41.419799+0000","last_undegraded":"2024-09-16T05:55:41.419799+0000","last_fullsized":"2024-09-16T05:55:41.419799+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:41.419602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:41.419602+0000","last_clean_scrub_stamp":"2024-09-16T05:55:41.419602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:57:38.040151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":181,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:43.222700+0000","last_change":"2024-09-16T05:55:43.222752+0000","last_active":"2024-09-16T05:55:43.222700+0000","last_peered":"2024-09-16T05:55:43.222700+0000","last_clean":"2024-09-16T05:55:43.222700+0000","last_became_active":"2024-09-16T05:39:26.053397+0000","last_became_peered":"2024-09-16T05:39:26.053397+0000","last_unstale":"2024-09-16T05:55:43.222700+0000","last_undegraded":"2024-09-16T05:55:43.222700+0000","last_fullsized":"2024-09-16T05:55:43.222700+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:43.222535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:43.222535+0000","last_clean_scrub_stamp":"2024-09-16T05:55:43.222535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:08.870685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":303,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:42.693829+0000","last_change":"2024-09-16T05:55:42.693852+0000","last_active":"2024-09-16T05:55:42.693829+0000","last_peered":"2024-09-16T05:55:42.693829+0000","last_clean":"2024-09-16T05:55:42.693829+0000","last_became_active":"2024-09-16T05:39:26.069114+0000","last_became_peered":"2024-09-16T05:39:26.069114+0000","last_unstale":"2024-09-16T05:55:42.693829+0000","last_undegraded":"2024-09-16T05:55:42.693829+0000","last_fullsized":"2024-09-16T05:55:42.693829+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:42.693743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:42.693743+0000","last_clean_scrub_stamp":"2024-09-16T05:55:42.693743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:22:59.308140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":181,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:44.259423+0000","last_change":"2024-09-16T05:55:44.259477+0000","last_active":"2024-09-16T05:55:44.259423+0000","last_peered":"2024-09-16T05:55:44.259423+0000","last_clean":"2024-09-16T05:55:44.259423+0000","last_became_active":"2024-09-16T05:39:26.055883+0000","last_became_peered":"2024-09-16T05:39:26.055883+0000","last_unstale":"2024-09-16T05:55:44.259423+0000","last_undegraded":"2024-09-16T05:55:44.259423+0000","last_fullsized":"2024-09-16T05:55:44.259423+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:44.259247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:44.259247+0000","last_clean_scrub_stamp":"2024-09-16T05:55:44.259247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:04.504792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":234,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:48.244378+0000","last_change":"2024-09-16T05:55:48.244521+0000","last_active":"2024-09-16T05:55:48.244378+0000","last_peered":"2024-09-16T05:55:48.244378+0000","last_clean":"2024-09-16T05:55:48.244378+0000","last_became_active":"2024-09-16T05:39:27.056198+0000","last_became_peered":"2024-09-16T05:39:27.056198+0000","last_unstale":"2024-09-16T05:55:48.244378+0000","last_undegraded":"2024-09-16T05:55:48.244378+0000","last_fullsized":"2024-09-16T05:55:48.244378+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:48.244147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:48.244147+0000","last_clean_scrub_stamp":"2024-09-16T05:55:48.244147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:21:35.912525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":285,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:42.465192+0000","last_change":"2024-09-16T05:55:42.465213+0000","last_active":"2024-09-16T05:55:42.465192+0000","last_peered":"2024-09-16T05:55:42.465192+0000","last_clean":"2024-09-16T05:55:42.465192+0000","last_became_active":"2024-09-16T05:39:26.073067+0000","last_became_peered":"2024-09-16T05:39:26.073067+0000","last_unstale":"2024-09-16T05:55:42.465192+0000","last_undegraded":"2024-09-16T05:55:42.465192+0000","last_fullsized":"2024-09-16T05:55:42.465192+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:42.465120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:42.465120+0000","last_clean_scrub_stamp":"2024-09-16T05:55:42.465120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:21:59.720321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":185,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:46.511849+0000","last_change":"2024-09-16T05:55:46.511979+0000","last_active":"2024-09-16T05:55:46.511849+0000","last_peered":"2024-09-16T05:55:46.511849+0000","last_clean":"2024-09-16T05:55:46.511849+0000","last_became_active":"2024-09-16T05:39:26.061165+0000","last_became_peered":"2024-09-16T05:39:26.061165+0000","last_unstale":"2024-09-16T05:55:46.511849+0000","last_undegraded":"2024-09-16T05:55:46.511849+0000","last_fullsized":"2024-09-16T05:55:46.511849+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:46.511601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:46.511601+0000","last_clean_scrub_stamp":"2024-09-16T05:55:46.511601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:41.102639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":180,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:42.183563+0000","last_change":"2024-09-16T05:55:42.183582+0000","last_active":"2024-09-16T05:55:42.183563+0000","last_peered":"2024-09-16T05:55:42.183563+0000","last_clean":"2024-09-16T05:55:42.183563+0000","last_became_active":"2024-09-16T05:39:26.053013+0000","last_became_peered":"2024-09-16T05:39:26.053013+0000","last_unstale":"2024-09-16T05:55:42.183563+0000","last_undegraded":"2024-09-16T05:55:42.183563+0000","last_fullsized":"2024-09-16T05:55:42.183563+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:42.183497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:42.183497+0000","last_clean_scrub_stamp":"2024-09-16T05:55:42.183497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:54.096870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":218,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:45.304665+0000","last_change":"2024-09-16T05:55:45.304772+0000","last_active":"2024-09-16T05:55:45.304665+0000","last_peered":"2024-09-16T05:55:45.304665+0000","last_clean":"2024-09-16T05:55:45.304665+0000","last_became_active":"2024-09-16T05:39:26.058997+0000","last_became_peered":"2024-09-16T05:39:26.058997+0000","last_unstale":"2024-09-16T05:55:45.304665+0000","last_undegraded":"2024-09-16T05:55:45.304665+0000","last_fullsized":"2024-09-16T05:55:45.304665+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:45.304414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:45.304414+0000","last_clean_scrub_stamp":"2024-09-16T05:55:45.304414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:52:46.785085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":222,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:42.536617+0000","last_change":"2024-09-16T05:55:42.536677+0000","last_active":"2024-09-16T05:55:42.536617+0000","last_peered":"2024-09-16T05:55:42.536617+0000","last_clean":"2024-09-16T05:55:42.536617+0000","last_became_active":"2024-09-16T05:39:26.049092+0000","last_became_peered":"2024-09-16T05:39:26.049092+0000","last_unstale":"2024-09-16T05:55:42.536617+0000","last_undegraded":"2024-09-16T05:55:42.536617+0000","last_fullsized":"2024-09-16T05:55:42.536617+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:42.536410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:42.536410+0000","last_clean_scrub_stamp":"2024-09-16T05:55:42.536410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:24.716269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":235,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:43.094693+0000","last_change":"2024-09-16T05:55:43.094751+0000","last_active":"2024-09-16T05:55:43.094693+0000","last_peered":"2024-09-16T05:55:43.094693+0000","last_clean":"2024-09-16T05:55:43.094693+0000","last_became_active":"2024-09-16T05:39:26.057564+0000","last_became_peered":"2024-09-16T05:39:26.057564+0000","last_unstale":"2024-09-16T05:55:43.094693+0000","last_undegraded":"2024-09-16T05:55:43.094693+0000","last_fullsized":"2024-09-16T05:55:43.094693+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:43.094512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:43.094512+0000","last_clean_scrub_stamp":"2024-09-16T05:55:43.094512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:24:51.254640+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":180,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:44.181357+0000","last_change":"2024-09-16T05:55:44.181409+0000","last_active":"2024-09-16T05:55:44.181357+0000","last_peered":"2024-09-16T05:55:44.181357+0000","last_clean":"2024-09-16T05:55:44.181357+0000","last_became_active":"2024-09-16T05:39:26.059546+0000","last_became_peered":"2024-09-16T05:39:26.059546+0000","last_unstale":"2024-09-16T05:55:44.181357+0000","last_undegraded":"2024-09-16T05:55:44.181357+0000","last_fullsized":"2024-09-16T05:55:44.181357+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:44.181183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:44.181183+0000","last_clean_scrub_stamp":"2024-09-16T05:55:44.181183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:04.577231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":181,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:45.141544+0000","last_change":"2024-09-16T05:55:45.141598+0000","last_active":"2024-09-16T05:55:45.141544+0000","last_peered":"2024-09-16T05:55:45.141544+0000","last_clean":"2024-09-16T05:55:45.141544+0000","last_became_active":"2024-09-16T05:39:26.073016+0000","last_became_peered":"2024-09-16T05:39:26.073016+0000","last_unstale":"2024-09-16T05:55:45.141544+0000","last_undegraded":"2024-09-16T05:55:45.141544+0000","last_fullsized":"2024-09-16T05:55:45.141544+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:45.141360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:45.141360+0000","last_clean_scrub_stamp":"2024-09-16T05:55:45.141360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:41.025162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":298,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:43.435165+0000","last_change":"2024-09-16T05:55:43.435217+0000","last_active":"2024-09-16T05:55:43.435165+0000","last_peered":"2024-09-16T05:55:43.435165+0000","last_clean":"2024-09-16T05:55:43.435165+0000","last_became_active":"2024-09-16T05:39:26.076818+0000","last_became_peered":"2024-09-16T05:39:26.076818+0000","last_unstale":"2024-09-16T05:55:43.435165+0000","last_undegraded":"2024-09-16T05:55:43.435165+0000","last_fullsized":"2024-09-16T05:55:43.435165+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:43.434955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:43.434955+0000","last_clean_scrub_stamp":"2024-09-16T05:55:43.434955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:40.329620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":193,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:43.652410+0000","last_change":"2024-09-16T05:55:43.652469+0000","last_active":"2024-09-16T05:55:43.652410+0000","last_peered":"2024-09-16T05:55:43.652410+0000","last_clean":"2024-09-16T05:55:43.652410+0000","last_became_active":"2024-09-16T05:39:27.050732+0000","last_became_peered":"2024-09-16T05:39:27.050732+0000","last_unstale":"2024-09-16T05:55:43.652410+0000","last_undegraded":"2024-09-16T05:55:43.652410+0000","last_fullsized":"2024-09-16T05:55:43.652410+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:43.652232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:43.652232+0000","last_clean_scrub_stamp":"2024-09-16T05:55:43.652232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:50.691957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":181,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:41.154246+0000","last_change":"2024-09-16T05:55:41.154265+0000","last_active":"2024-09-16T05:55:41.154246+0000","last_peered":"2024-09-16T05:55:41.154246+0000","last_clean":"2024-09-16T05:55:41.154246+0000","last_became_active":"2024-09-16T05:39:26.052209+0000","last_became_peered":"2024-09-16T05:39:26.052209+0000","last_unstale":"2024-09-16T05:55:41.154246+0000","last_undegraded":"2024-09-16T05:55:41.154246+0000","last_fullsized":"2024-09-16T05:55:41.154246+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:41.154180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:41.154180+0000","last_clean_scrub_stamp":"2024-09-16T05:55:41.154180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:00:39.249246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"259'119","reported_seq":602,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:47.564475+0000","last_change":"2024-09-16T05:55:47.564579+0000","last_active":"2024-09-16T05:55:47.564475+0000","last_peered":"2024-09-16T05:55:47.564475+0000","last_clean":"2024-09-16T05:55:47.564475+0000","last_became_active":"2024-09-16T05:39:26.061101+0000","last_became_peered":"2024-09-16T05:39:26.061101+0000","last_unstale":"2024-09-16T05:55:47.564475+0000","last_undegraded":"2024-09-16T05:55:47.564475+0000","last_fullsized":"2024-09-16T05:55:47.564475+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'119","last_scrub_stamp":"2024-09-16T05:55:47.564287+0000","last_deep_scrub":"259'119","last_deep_scrub_stamp":"2024-09-16T05:55:47.564287+0000","last_clean_scrub_stamp":"2024-09-16T05:55:47.564287+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:16:18.160516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"259'2","reported_seq":451,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:46.264386+0000","last_change":"2024-09-16T05:55:46.264572+0000","last_active":"2024-09-16T05:55:46.264386+0000","last_peered":"2024-09-16T05:55:46.264386+0000","last_clean":"2024-09-16T05:55:46.264386+0000","last_became_active":"2024-09-16T05:39:26.063463+0000","last_became_peered":"2024-09-16T05:39:26.063463+0000","last_unstale":"2024-09-16T05:55:46.264386+0000","last_undegraded":"2024-09-16T05:55:46.264386+0000","last_fullsized":"2024-09-16T05:55:46.264386+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'2","last_scrub_stamp":"2024-09-16T05:55:46.264182+0000","last_deep_scrub":"259'2","last_deep_scrub_stamp":"2024-09-16T05:55:46.264182+0000","last_clean_scrub_stamp":"2024-09-16T05:55:46.264182+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:44.248526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011975009999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":499,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:45.473336+0000","last_change":"2024-09-16T05:55:45.473391+0000","last_active":"2024-09-16T05:55:45.473336+0000","last_peered":"2024-09-16T05:55:45.473336+0000","last_clean":"2024-09-16T05:55:45.473336+0000","last_became_active":"2024-09-16T05:39:26.060550+0000","last_became_peered":"2024-09-16T05:39:26.060550+0000","last_unstale":"2024-09-16T05:55:45.473336+0000","last_undegraded":"2024-09-16T05:55:45.473336+0000","last_fullsized":"2024-09-16T05:55:45.473336+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:45.473157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:45.473157+0000","last_clean_scrub_stamp":"2024-09-16T05:55:45.473157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:16.831855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082819000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":194,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:41.649214+0000","last_change":"2024-09-16T05:55:41.649255+0000","last_active":"2024-09-16T05:55:41.649214+0000","last_peered":"2024-09-16T05:55:41.649214+0000","last_clean":"2024-09-16T05:55:41.649214+0000","last_became_active":"2024-09-16T05:39:26.058771+0000","last_became_peered":"2024-09-16T05:39:26.058771+0000","last_unstale":"2024-09-16T05:55:41.649214+0000","last_undegraded":"2024-09-16T05:55:41.649214+0000","last_fullsized":"2024-09-16T05:55:41.649214+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:41.649106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:41.649106+0000","last_clean_scrub_stamp":"2024-09-16T05:55:41.649106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:05:13.435637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001030029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"259'3","reported_seq":460,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:47.248696+0000","last_change":"2024-09-16T05:55:47.248851+0000","last_active":"2024-09-16T05:55:47.248696+0000","last_peered":"2024-09-16T05:55:47.248696+0000","last_clean":"2024-09-16T05:55:47.248696+0000","last_became_active":"2024-09-16T05:39:26.065589+0000","last_became_peered":"2024-09-16T05:39:26.065589+0000","last_unstale":"2024-09-16T05:55:47.248696+0000","last_undegraded":"2024-09-16T05:55:47.248696+0000","last_fullsized":"2024-09-16T05:55:47.248696+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'3","last_scrub_stamp":"2024-09-16T05:55:47.248401+0000","last_deep_scrub":"259'3","last_deep_scrub_stamp":"2024-09-16T05:55:47.248401+0000","last_clean_scrub_stamp":"2024-09-16T05:55:47.248401+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:33.840000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001996018,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":439,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:46.162921+0000","last_change":"2024-09-16T05:55:46.163070+0000","last_active":"2024-09-16T05:55:46.162921+0000","last_peered":"2024-09-16T05:55:46.162921+0000","last_clean":"2024-09-16T05:55:46.162921+0000","last_became_active":"2024-09-16T05:39:27.056262+0000","last_became_peered":"2024-09-16T05:39:27.056262+0000","last_unstale":"2024-09-16T05:55:46.162921+0000","last_undegraded":"2024-09-16T05:55:46.162921+0000","last_fullsized":"2024-09-16T05:55:46.162921+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:46.162686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:46.162686+0000","last_clean_scrub_stamp":"2024-09-16T05:55:46.162686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:32.241552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001540065,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":184,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:44.505488+0000","last_change":"2024-09-16T05:55:44.505543+0000","last_active":"2024-09-16T05:55:44.505488+0000","last_peered":"2024-09-16T05:55:44.505488+0000","last_clean":"2024-09-16T05:55:44.505488+0000","last_became_active":"2024-09-16T05:39:26.060029+0000","last_became_peered":"2024-09-16T05:39:26.060029+0000","last_unstale":"2024-09-16T05:55:44.505488+0000","last_undegraded":"2024-09-16T05:55:44.505488+0000","last_fullsized":"2024-09-16T05:55:44.505488+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:44.505315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:44.505315+0000","last_clean_scrub_stamp":"2024-09-16T05:55:44.505315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:09:04.115206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001073898,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":180,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:43.159634+0000","last_change":"2024-09-16T05:55:43.159692+0000","last_active":"2024-09-16T05:55:43.159634+0000","last_peered":"2024-09-16T05:55:43.159634+0000","last_clean":"2024-09-16T05:55:43.159634+0000","last_became_active":"2024-09-16T05:39:26.054759+0000","last_became_peered":"2024-09-16T05:39:26.054759+0000","last_unstale":"2024-09-16T05:55:43.159634+0000","last_undegraded":"2024-09-16T05:55:43.159634+0000","last_fullsized":"2024-09-16T05:55:43.159634+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:43.159461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:43.159461+0000","last_clean_scrub_stamp":"2024-09-16T05:55:43.159461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:58.108608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001156041,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":184,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-16T05:55:43.527450+0000","last_change":"2024-09-16T05:55:43.527510+0000","last_active":"2024-09-16T05:55:43.527450+0000","last_peered":"2024-09-16T05:55:43.527450+0000","last_clean":"2024-09-16T05:55:43.527450+0000","last_became_active":"2024-09-16T05:39:26.050738+0000","last_became_peered":"2024-09-16T05:39:26.050738+0000","last_unstale":"2024-09-16T05:55:43.527450+0000","last_undegraded":"2024-09-16T05:55:43.527450+0000","last_fullsized":"2024-09-16T05:55:43.527450+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T05:55:43.527260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T05:55:43.527260+0000","last_clean_scrub_stamp":"2024-09-16T05:55:43.527260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:56.601023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011964930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18560,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607811,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34272,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337568,"statfs":{"total":96636764160,"available":96601669632,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640514,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34324,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337516,"statfs":{"total":96636764160,"available":96601616384,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640514,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34272,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337568,"statfs":{"total":96636764160,"available":96601669632,"internally_reserved":0,"allocated":3162112,"data_stored":2398039,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640513,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3166208,"data_stored":2398058,"data_compressed":295935,"data_compressed_allocated":442368,"data_compressed_original":815104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640516,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34324,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337516,"statfs":{"total":96636764160,"available":96601616384,"internally_reserved":0,"allocated":3215360,"data_stored":2926967,"data_compressed":305215,"data_compressed_allocated":475136,"data_compressed_original":1327104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9280,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T05:56:03.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T05:56:03.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.634+0000 7fad46d75640 10 monclient: get_monmap_and_config 2024-09-16T05:56:03.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.634+0000 7fad46d75640 10 monclient: build_initial_monmap 2024-09-16T05:56:03.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.634+0000 7fad46d75640 10 monclient: monmap: 2024-09-16T05:56:03.641 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:56:03.642490+0000 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:56:03.642490+0000 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-c 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-09-16T05:56:03.642 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 Processor -- start 2024-09-16T05:56:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- start start 2024-09-16T05:56:03.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient: init 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient: _reopen_session rank -1 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient(hunting): picked mon.noname-a con 0x7fad4009aa30 addr v1:172.21.15.6:6789/0 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient(hunting): picked mon.noname-c con 0x7fad4009e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient(hunting): picked mon.noname-b con 0x7fad400a0da0 addr v1:172.21.15.6:6790/0 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad400a1370 con 0x7fad4009aa30 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4009ead0 con 0x7fad4009e230 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4009edb0 con 0x7fad400a0da0 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient(hunting): _renew_subs 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient(hunting): authenticate will time out at 2225.449226s 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad44b11640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fad4009aa30 0x7fad4009ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43838/0 (socket says 172.21.15.6:43838) 2024-09-16T05:56:03.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad44b11640 1 -- 172.21.15.6:0/698498170 learned_addr learned my addr 172.21.15.6:0/698498170 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255039437 0 0) 0x7fad4009edb0 con 0x7fad400a0da0 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient(hunting): my global_id is 164573 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad28003530 con 0x7fad400a0da0 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369992640 0 0) 0x7fad400a1370 con 0x7fad4009aa30 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient(hunting): my global_id is 194247 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad4009edb0 con 0x7fad4009aa30 2024-09-16T05:56:03.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1844864939 0 0) 0x7fad28003530 con 0x7fad400a0da0 2024-09-16T05:56:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 >> v1:172.21.15.63:6789/0 conn(0x7fad4009e230 legacy=0x7fad401399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 >> v1:172.21.15.6:6789/0 conn(0x7fad4009aa30 legacy=0x7fad4009ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: _finish_hunting 0 2024-09-16T05:56:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 monclient: found mon.noname-b 2024-09-16T05:56:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:56:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-09-16T05:56:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4009dd50 con 0x7fad400a0da0 2024-09-16T05:56:03.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: _finish_auth 0 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: _check_auth_tickets 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 5 monclient: authenticate success, global_id 164573 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad380018d0 con 0x7fad400a0da0 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: dump: 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.652 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.653 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.654 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.654 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.654 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T05:56:03.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad38001b40 con 0x7fad400a0da0 2024-09-16T05:56:03.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T05:56:03.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient: get_monmap_and_config success 2024-09-16T05:56:03.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 4 set_mon_vals no callback set 2024-09-16T05:56:03.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 1 -- 172.21.15.6:0/698498170 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad38001f10 con 0x7fad400a0da0 2024-09-16T05:56:03.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:56:03.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T05:56:03.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad377fe640 10 monclient: dump: 2024-09-16T05:56:03.656 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.656 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.656 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.656 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.657 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.657 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.657 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.657 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.657 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.657 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient: shutdown 2024-09-16T05:56:03.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:56:03.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- 172.21.15.6:0/698498170 >> v1:172.21.15.6:6790/0 conn(0x7fad400a0da0 legacy=0x7fad4013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- 172.21.15.6:0/698498170 shutdown_connections 2024-09-16T05:56:03.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- 172.21.15.6:0/698498170 >> 172.21.15.6:0/698498170 conn(0x7fad400a1650 msgr2=0x7fad400a1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:56:03.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- 172.21.15.6:0/698498170 shutdown_connections 2024-09-16T05:56:03.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- 172.21.15.6:0/698498170 wait complete. 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient: build_initial_monmap 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient: monmap: 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.noname-b 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 Processor -- start 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 1 -- start start 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.638+0000 7fad46d75640 10 monclient: init 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient: _reopen_session rank -1 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient(hunting): picked mon.noname-c con 0x7fad4009aa30 addr v1:172.21.15.6:6790/0 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient(hunting): picked mon.noname-b con 0x7fad4009e230 addr v1:172.21.15.63:6789/0 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient(hunting): picked mon.noname-a con 0x7fad400a0da0 addr v1:172.21.15.6:6789/0 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4011c610 con 0x7fad400a0da0 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad400740c0 con 0x7fad4009e230 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4011b9a0 con 0x7fad4009aa30 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient(hunting): _renew_subs 2024-09-16T05:56:03.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient(hunting): authenticate will time out at 2225.452331s 2024-09-16T05:56:03.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad44b11640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fad4009aa30 0x7fad4013b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:53278/0 (socket says 172.21.15.6:53278) 2024-09-16T05:56:03.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad44b11640 1 -- 172.21.15.6:0/4236173296 learned_addr learned my addr 172.21.15.6:0/4236173296 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-09-16T05:56:03.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4045423876 0 0) 0x7fad4011b9a0 con 0x7fad4009aa30 2024-09-16T05:56:03.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): my global_id is 164576 2024-09-16T05:56:03.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad2c003530 con 0x7fad4009aa30 2024-09-16T05:56:03.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034814614 0 0) 0x7fad4011c610 con 0x7fad400a0da0 2024-09-16T05:56:03.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): my global_id is 194250 2024-09-16T05:56:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad4011b9a0 con 0x7fad400a0da0 2024-09-16T05:56:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1000984979 0 0) 0x7fad4011b9a0 con 0x7fad400a0da0 2024-09-16T05:56:03.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad4011c610 con 0x7fad400a0da0 2024-09-16T05:56:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad38001e90 con 0x7fad400a0da0 2024-09-16T05:56:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:56:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T05:56:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): dump: 2024-09-16T05:56:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.664 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.665 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.665 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T05:56:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1268765336 0 0) 0x7fad2c003530 con 0x7fad4009aa30 2024-09-16T05:56:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad4011b9a0 con 0x7fad4009aa30 2024-09-16T05:56:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad3c001820 con 0x7fad4009aa30 2024-09-16T05:56:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T05:56:03.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T05:56:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): dump: 2024-09-16T05:56:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.667 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.668 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.1 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1860232012 0 0) 0x7fad400740c0 con 0x7fad4009e230 2024-09-16T05:56:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _negotiate 2024-09-16T05:56:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T05:56:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T05:56:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient(hunting): my global_id is 184351 2024-09-16T05:56:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad2c003530 con 0x7fad4009e230 2024-09-16T05:56:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1419401842 0 0) 0x7fad4011c610 con 0x7fad400a0da0 2024-09-16T05:56:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 >> v1:172.21.15.6:6790/0 conn(0x7fad4009aa30 legacy=0x7fad4013b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 >> v1:172.21.15.63:6789/0 conn(0x7fad4009e230 legacy=0x7fad4013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient: _finish_hunting 0 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 monclient: found mon.a 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4011c330 con 0x7fad400a0da0 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient: _finish_auth 0 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient: _check_auth_tickets 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 5 monclient: authenticate success, global_id 194250 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient: _renew_subs 2024-09-16T05:56:03.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fad40135250 con 0x7fad400a0da0 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fad38003cb0 con 0x7fad400a0da0 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient: handle_config config(0 keys) 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3917122089 0 0) 0x7fad38004000 con 0x7fad400a0da0 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 10 monclient: dump: 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 22154726-d7f4-4d5e-b601-06ee1f3e7219 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-09-16T05:34:04.807479+0000 2024-09-16T05:56:03.671 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.63:6789/0 mon.b 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad36ffd640 4 set_mon_vals no callback set 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient: _renew_subs 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fad4013c550 con 0x7fad400a0da0 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 3464729968 0 0) 0x7fad3800eec0 con 0x7fad400a0da0 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T05:56:03.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(259..259 src has 234..259) ==== 6817+0+0 (unknown 3701905584 0 0) 0x7fad38043f80 con 0x7fad400a0da0 2024-09-16T05:56:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:56:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad401387e0 con 0x7fad400a0da0 2024-09-16T05:56:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.642+0000 7fad44b11640 10 monclient: get_auth_request con 0x7fad2c036b70 auth_method 0 2024-09-16T05:56:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.646+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (unknown 1092875540 0 1081819029) 0x7fad38014670 con 0x7fad400a0da0 2024-09-16T05:56:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.646+0000 7fad357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T05:56:03.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.646+0000 7fad357fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T05:56:03.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.810+0000 7fad46d75640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T05:56:03.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.810+0000 7fad46d75640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T05:56:03.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.810+0000 7fad46d75640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-09-16T05:56:03.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.810+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 --> v1:172.21.15.6:6789/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fad4006bc10 con 0x7fad400a0da0 2024-09-16T05:56:03.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.818+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) ==== 123+0+0 (unknown 4099060384 0 0) 0x7fad38013f70 con 0x7fad400a0da0 2024-09-16T05:56:03.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.818+0000 7fad357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T05:56:03.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.818+0000 7fad357fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T05:56:03.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.818+0000 7fad357fa640 1 -- 172.21.15.6:0/4236173296 <== mon.0 v1:172.21.15.6:6789/0 11 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7fad38071030 con 0x7fad400a0da0 2024-09-16T05:56:03.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.818+0000 7fad357fa640 10 monclient: handle_config config(1 keys) 2024-09-16T05:56:03.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.818+0000 7fad367fc640 4 set_mon_vals no callback set 2024-09-16T05:56:03.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.818+0000 7fad367fc640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T05:56:03.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.822+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 >> v1:172.21.15.63:6812/3643207211 conn(0x7fad2c036b70 legacy=0x7fad2c039030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.822+0000 7fad46d75640 10 monclient: shutdown 2024-09-16T05:56:03.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.822+0000 7fad46d75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T05:56:03.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.822+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 >> v1:172.21.15.6:6789/0 conn(0x7fad400a0da0 legacy=0x7fad401d1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T05:56:03.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.822+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 shutdown_connections 2024-09-16T05:56:03.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.822+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 >> 172.21.15.6:0/4236173296 conn(0x7fad400a1650 msgr2=0x7fad401359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T05:56:03.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.822+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 shutdown_connections 2024-09-16T05:56:03.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-16T05:56:03.822+0000 7fad46d75640 1 -- 172.21.15.6:0/4236173296 wait complete. 2024-09-16T05:56:03.842 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T05:56:03.842 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T05:56:03.842 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T05:56:03.842 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:03.926 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T05:56:03.927 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T05:56:03.927 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.014 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T05:56:04.015 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T05:56:04.015 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.098 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T05:56:04.099 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T05:56:04.099 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.157 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T05:56:04.158 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T05:56:04.158 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.250 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T05:56:04.250 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T05:56:04.250 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.334 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T05:56:04.334 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T05:56:04.334 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T05:56:04.334 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.365 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T05:56:04.365 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T05:56:04.365 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T05:56:04.365 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.386 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T05:56:04.387 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T05:56:04.387 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.458 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T05:56:04.458 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T05:56:04.458 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T05:56:04.479 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T05:56:04.479 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T05:56:04.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2024-09-16T05:56:04.523 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:56:04.523 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T05:56:04.940 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:56:04.940 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T05:56:05.138 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:56:05.138 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T05:56:05.337 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:56:05.338 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T05:56:05.613 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:56:05.613 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T05:56:05.811 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:56:05.811 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T05:56:05.977 INFO:tasks.ceph:Archiving mon data... 2024-09-16T05:56:06.390 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/data/mon.a.tgz 2024-09-16T05:56:06.391 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-16T05:56:06.397 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.L6fq19a1LZ 2024-09-16T05:56:06.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.L6fq19a1LZ 2024-09-16T05:56:06.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.L6fq19a1LZ 2024-09-16T05:56:06.611 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.L6fq19a1LZ is 489KB 2024-09-16T05:56:06.672 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.L6fq19a1LZ 2024-09-16T05:56:06.677 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/data/mon.c.tgz 2024-09-16T05:56:06.677 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-16T05:56:06.726 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.082ZO5fB8f 2024-09-16T05:56:06.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.082ZO5fB8f 2024-09-16T05:56:06.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.082ZO5fB8f 2024-09-16T05:56:06.939 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.082ZO5fB8f is 493KB 2024-09-16T05:56:07.007 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.082ZO5fB8f 2024-09-16T05:56:07.014 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/data/mon.b.tgz 2024-09-16T05:56:07.014 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-09-16T05:56:07.019 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.ANbUCzIHC8 2024-09-16T05:56:07.020 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ANbUCzIHC8 2024-09-16T05:56:07.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.ANbUCzIHC8 2024-09-16T05:56:07.228 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.ANbUCzIHC8 is 494KB 2024-09-16T05:56:07.289 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.ANbUCzIHC8 2024-09-16T05:56:07.295 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T05:56:07.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T05:56:07.298 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T05:56:07.358 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T05:56:07.400 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T05:56:07.400 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T05:56:07.402 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/remote/smithi006/crash 2024-09-16T05:56:07.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T05:56:07.418 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/remote/smithi063/crash 2024-09-16T05:56:07.419 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T05:56:07.456 INFO:tasks.ceph:Compressing logs... 2024-09-16T05:56:07.456 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T05:56:07.467 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T05:56:07.485 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32161.log 2024-09-16T05:56:07.486 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32910.log 2024-09-16T05:56:07.487 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27790.log 2024-09-16T05:56:07.487 INFO:teuthology.orchestra.run.smithi006.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32161.log.gz 2024-09-16T05:56:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29721.log 2024-09-16T05:56:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32910.log: /var/log/ceph/ceph-client.admin.27790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31346.log 2024-09-16T05:56:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32369.log 2024-09-16T05:56:07.488 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27790.log.gz 2024-09-16T05:56:07.488 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32910.log.gz 2024-09-16T05:56:07.488 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29721.log.gz 2024-09-16T05:56:07.488 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29612.log 2024-09-16T05:56:07.491 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32369.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.32682.log 2024-09-16T05:56:07.491 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31346.log.gz 2024-09-16T05:56:07.491 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29612.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32369.log.gz 2024-09-16T05:56:07.491 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32474.log 2024-09-16T05:56:07.491 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29508.log 2024-09-16T05:56:07.491 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29612.log.gz 2024-09-16T05:56:07.491 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32682.log.gz 2024-09-16T05:56:07.491 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32317.log 2024-09-16T05:56:07.492 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32474.log.gz 2024-09-16T05:56:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31754.log 2024-09-16T05:56:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32734.log 2024-09-16T05:56:07.492 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29508.log.gz 2024-09-16T05:56:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31754.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.32317.log.gz 2024-09-16T05:56:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29799.log 2024-09-16T05:56:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32734.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31754.log.gz 2024-09-16T05:56:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29082.log 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32734.log.gz 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28514.log 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29082.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29799.log.gz 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29082.log.gz 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32988.log 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28514.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32630.log 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32988.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28514.log.gz 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29534.log 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32988.log.gz 2024-09-16T05:56:07.494 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.32630.log: /var/log/ceph/ceph-client.admin.28699.log 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30752.log 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28699.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32630.log.gz 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29534.log.gz 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32962.log 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28699.log.gz 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31294.log 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30752.log: /var/log/ceph/ceph-client.admin.32962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31033.log 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29108.log 2024-09-16T05:56:07.496 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.30752.log.gz 2024-09-16T05:56:07.497 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32962.log.gz 2024-09-16T05:56:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30544.log 2024-09-16T05:56:07.499 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31294.log.gz 2024-09-16T05:56:07.499 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31033.log.gz 2024-09-16T05:56:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32526.log 2024-09-16T05:56:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30179.log 2024-09-16T05:56:07.499 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29108.log.gz 2024-09-16T05:56:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30544.log: /var/log/ceph/ceph-client.admin.32526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32213.log 2024-09-16T05:56:07.500 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30544.log.gz 2024-09-16T05:56:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30179.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32526.log.gz 2024-09-16T05:56:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28169.log 2024-09-16T05:56:07.500 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30179.log.gz 2024-09-16T05:56:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31962.log 2024-09-16T05:56:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32421.log 2024-09-16T05:56:07.500 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32213.log.gz 2024-09-16T05:56:07.500 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28169.log.gz 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32044.log 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31962.log.gz 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29212.log 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29378.log 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32421.log.gz 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29212.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32044.log.gz 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29212.log.gz 2024-09-16T05:56:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28543.log 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30726.log 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29378.log.gz 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27942.log 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28543.log.gz 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28647.log 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25185.log 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27942.log.gz 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30726.log.gz 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31320.log 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28647.log: /var/log/ceph/ceph-client.admin.25185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31216.log 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28647.log.gz 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25185.log.gz 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32109.log 2024-09-16T05:56:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31216.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T05:56:07.507 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31320.log.gz 2024-09-16T05:56:07.512 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27968.log 2024-09-16T05:56:07.512 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31216.log.gz 2024-09-16T05:56:07.512 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32109.log.gz 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29238.log 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.27968.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28673.log 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29238.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27968.log.gz 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31610.log 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29238.log.gz 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28014.log 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28673.log: /var/log/ceph/ceph-client.admin.31610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31910.log 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28673.log.gz 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28014.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.31610.log.gz 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31528.log 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28014.log.gz 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31450.log 2024-09-16T05:56:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28803.log 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31528.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.31910.log.gz 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31676.log 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31884.log 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31450.log.gz -- replaced with /var/log/ceph/ceph-client.admin.31528.log.gz 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28803.log.gz 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.31676.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33274.log 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31884.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31676.log.gz 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29877.log 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33196.log 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33274.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31884.log.gz 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32395.log 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33196.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33274.log.gz 2024-09-16T05:56:07.514 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33378.log 2024-09-16T05:56:07.515 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29877.log.gz 2024-09-16T05:56:07.515 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33196.log.gz 2024-09-16T05:56:07.515 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28306.log 2024-09-16T05:56:07.515 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32552.log 2024-09-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33378.log: /var/log/ceph/ceph-client.admin.28306.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32395.log.gz 2024-09-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33014.log 2024-09-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28306.log.gz 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32552.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33378.log.gz 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30856.log 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31580.log 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33014.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32552.log.gz 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30596.log 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33014.log.gz 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31580.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30856.log.gz 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31398.log 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31580.log.gz 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29955.log 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31988.log 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31398.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30596.log.gz 2024-09-16T05:56:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29825.log 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31398.log.gz 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31988.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.29955.log.gz 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33300.log 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31988.log.gz 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28005.log 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31476.log 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33300.log: /var/log/ceph/ceph-client.admin.28005.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30153.log 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29825.log.gz 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33300.log.gz/var/log/ceph/ceph-client.admin.31476.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28005.log.gz 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31502.log 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30153.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.31476.log.gz 2024-09-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31780.log 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31502.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.30153.log.gz 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31085.log 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.31502.log.gz 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30830.log 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29586.log 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31780.log.gz 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31085.log.gz 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30890.log 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28338.log 2024-09-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30830.log.gz 2024-09-16T05:56:07.616 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29586.log: /var/log/ceph/ceph-client.admin.30890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30033.log 2024-09-16T05:56:07.616 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.30890.log.gz 2024-09-16T05:56:07.616 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29586.log.gz 2024-09-16T05:56:07.616 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27846.log/var/log/ceph/ceph-client.admin.28338.log: 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30283.log 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30033.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28338.log.gz 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29669.log 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30033.log.gz 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27846.log.gz 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29160.log 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28855.log 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29669.log: /var/log/ceph/ceph-client.admin.29160.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30283.log.gz 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29404.log 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29669.log.gz 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28855.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29160.log.gz 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30205.log 2024-09-16T05:56:07.617 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25108.log 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28855.log.gz 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29404.log: /var/log/ceph/ceph-client.admin.30205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28488.log 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29404.log.gz 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.25108.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.30205.log.gz --verbose 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.31190.log 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25108.log.gz 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30062.log 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29134.log 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31190.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28488.log.gz 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31554.log 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29134.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31190.log.gz 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32578.log 2024-09-16T05:56:07.618 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30062.log.gz 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31554.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.29134.log.gz -5 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30257.log 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32578.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31554.log.gz 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28621.log 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29326.log 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32578.log.gz 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30257.log.gz 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28621.log: /var/log/ceph/ceph-client.admin.32936.log 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28310.log 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29326.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28621.log.gz 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28595.log 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29326.log.gz 2024-09-16T05:56:07.619 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30778.log 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32936.log.gz 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28310.log: /var/log/ceph/ceph-client.admin.28595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28010.log 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28310.log.gz 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30778.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28595.log.gz 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28010.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.30778.log.gz 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30700.log 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31268.log 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28010.log.gz 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29030.log 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30700.log: /var/log/ceph/ceph-client.admin.31268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30622.log 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30700.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28751.log 2024-09-16T05:56:07.620 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31268.log.gz 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29030.log: /var/log/ceph/ceph-client.admin.30622.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29030.log.gz 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30622.log.gz 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30955.log 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31728.log 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28952.log 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28751.log.gz 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31728.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30955.log.gz 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32708.log 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31728.log.gz 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29456.log 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28952.log: /var/log/ceph/ceph-client.admin.32708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31137.log 2024-09-16T05:56:07.621 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29352.log 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28952.log.gz 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32708.log.gz 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31137.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29456.log.gz 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31137.log.gz 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28011.log 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30648.log 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31007.log 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29352.log.gz 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29851.log 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28011.log.gz 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24991.log 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30648.log.gz 2024-09-16T05:56:07.622 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31007.log: /var/log/ceph/ceph-client.admin.29851.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31007.log.gz 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29851.log.gz 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33326.log 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.24991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32239.log 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28315.log 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.24991.log.gz 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29773.log 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33326.log.gz 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28829.log 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9%/var/log/ceph/ceph-client.admin.28315.log: -- replaced with /var/log/ceph/ceph-client.admin.32239.log.gz 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29773.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28315.log.gz 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29773.log.gz 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T05:56:07.623 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30007.log 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28725.log 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28829.log.gz 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28366.log 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28725.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27871.log 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30007.log.gz 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28366.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31858.log 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28725.log.gz 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27871.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28366.log.gz 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27871.log.gz 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29430.log 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31806.log 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33428.log 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31858.log.gz 2024-09-16T05:56:07.624 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31806.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29430.log.gz 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28881.log 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31806.log.gz 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30361.log 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28881.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33248.log 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33428.log.gz 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30361.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28881.log.gz 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31936.log 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28365.log 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.30361.log.gz 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31936.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33248.log.gz 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32604.log 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.31936.log.gz 2024-09-16T05:56:07.625 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32858.log 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32604.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28365.log.gz 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30127.log 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29300.log 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32858.log: /var/log/ceph/ceph-client.admin.30127.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32604.log.gz 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32858.log.gz 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30127.log.gz 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32884.log 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28166.log 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32884.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29300.log.gz 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28166.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32884.log.gz 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28166.log.gz 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33222.log 2024-09-16T05:56:07.626 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30231.log 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31242.log 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5%/var/log/ceph/ceph-client.admin.30231.log: -- replaced with /var/log/ceph/ceph-client.admin.33222.log.gz 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30231.log.gz 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28462.log 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32500.log 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28462.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31242.log.gz 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28462.log.gz 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27916.log 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32135.log 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27916.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32500.log.gz 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27916.log.gz 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32291.log 2024-09-16T05:56:07.627 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29264.log 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32291.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32135.log.gz 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28009.log 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32291.log.gz 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28162.log 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31059.log 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29264.log.gz/var/log/ceph/ceph-client.admin.28009.log: 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30518.log 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28009.log.gz 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28162.log.gz 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30570.log 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7%/var/log/ceph/ceph-client.admin.30518.log: -- replaced with /var/log/ceph/ceph-client.admin.31059.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31111.log 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:07.628 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9%/var/log/ceph/ceph-client.admin.30570.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.30518.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.33170.log 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31111.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30570.log.gz 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31111.log.gz 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32265.log 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.audit.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33170.log.gz -5 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.28170.log 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32265.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32265.log.gz 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30387.log 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31702.log 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30387.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28170.log.gz 2024-09-16T05:56:07.629 INFO:teuthology.orchestra.run.smithi006.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30387.log.gz 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29638.log 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31372.log 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29638.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31702.log.gz 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29638.log.gz 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30804.log 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28569.log 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30804.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31372.log.gz 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30804.log.gz 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32014.log 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28150.log 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32014.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28569.log.gz 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32014.log.gz 2024-09-16T05:56:07.630 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28926.log 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32343.log 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28926.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28150.log.gz 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28926.log.gz 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32656.log 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33118.log 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32656.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.32343.log.gz 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32656.log.gz 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27896.log 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33118.log: 95.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33040.log 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33118.log.gz 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.27896.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27896.log.gz 2024-09-16T05:56:07.631 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33066.log 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33404.log 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28161.log 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33040.log.gz 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6%/var/log/ceph/ceph-client.admin.33404.log: -- replaced with /var/log/ceph/ceph-client.admin.33066.log.gz 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33404.log.gz 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30492.log 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29929.log 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30492.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29560.log 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28161.log.gz 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29929.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30492.log.gz 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29929.log.gz 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30465.log 2024-09-16T05:56:07.632 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29056.log 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31832.log/var/log/ceph/ceph-client.admin.30465.log: 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29560.log.gz 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29056.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30465.log.gz 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29056.log.gz 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33092.log 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30413.log 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30335.log 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31832.log.gz 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30413.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33092.log.gz 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30413.log.gz 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28000.log 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29004.log 2024-09-16T05:56:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30335.log: /var/log/ceph/ceph-client.admin.28000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32448.log 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30335.log.gz 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28000.log.gz 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29004.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29004.log.gz 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33352.log 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32806.log 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33352.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21391.log 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.32448.log.gz 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32806.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33352.log.gz 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.32806.log.gz 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32187.log 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33144.log 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.21391.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21391.log.gz 2024-09-16T05:56:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32187.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32187.log.gz 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30309.log 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29186.log 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32832.log 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33144.log.gz 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29186.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30309.log.gz 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29186.log.gz 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29747.log 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30439.log 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.32832.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.29747.log: -- /var/log/ceph/ceph-client.admin.30981.log 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30439.log: 88.9% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32832.log.gz 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29747.log.gz 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30439.log.gz 2024-09-16T05:56:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29903.log 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28777.log 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30981.log: /var/log/ceph/ceph-client.admin.29903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29981.log 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28777.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30981.log.gz 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29903.log.gz 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28777.log.gz 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29482.log 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28978.log 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29981.log: /var/log/ceph/ceph-client.admin.29482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29695.log 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr: 88.4%/var/log/ceph/ceph-client.admin.28978.log: -- replaced with /var/log/ceph/ceph-client.admin.29981.log.gz 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29482.log.gz 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28978.log.gz 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31424.log 2024-09-16T05:56:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31163.log 2024-09-16T05:56:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30674.log 2024-09-16T05:56:07.637 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29695.log.gz 2024-09-16T05:56:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.31163.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31424.log.gz 2024-09-16T05:56:07.637 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31163.log.gz 2024-09-16T05:56:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.30674.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30674.log.gz 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30194.log 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32293.log 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29641.log 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32293.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30194.log.gz 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.35823.log 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32293.log.gz 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30871.log 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29641.log: /var/log/ceph/ceph-client.bar.35823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34942.log 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34005.log 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30871.log.gz 2024-09-16T05:56:07.686 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31767.log 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34942.log.gz 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29641.log.gz 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34005.log: 92.8% -- replaced with /var/log/ceph/ceph-client.bar.35823.log.gz 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33596.log 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34005.log.gz 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36025.log 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34673.log 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33596.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31767.log.gz 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.36025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32775.log 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.36025.log.gz 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33596.log.gz 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33258.log 2024-09-16T05:56:07.687 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32775.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34673.log.gz 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32392.log 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28790.log 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33258.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32775.log.gz 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31887.log 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33258.log.gz 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28790.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32392.log.gz 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.35801.log 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28790.log.gz 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32051.log 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.bar.35801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29523.log 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31887.log.gz 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35687.log 2024-09-16T05:56:07.688 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.bar.35801.log.gz 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29523.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.32051.log.gz 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34486.log 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29523.log.gz/var/log/ceph/ceph-client.admin.35687.log: 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bazar.35365.log 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31037.log 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34486.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35687.log.gz 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.bazar.35365.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34173.log 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34486.log.gz 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32243.log 2024-09-16T05:56:07.689 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31037.log.gz 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31545.log 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30680.log 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.bazar.35365.log.gz 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34173.log.gz 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34270.log 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32243.log.gz 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31545.log.gz/var/log/ceph/ceph-client.admin.30680.log: 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33981.log 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.34270.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29764.log 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30680.log.gz 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34438.log 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34270.log.gz 2024-09-16T05:56:07.690 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28639.log 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33981.log.gz 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29764.log.gz 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30440.log 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34438.log.gz 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.33740.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29838.log 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28639.log.gz 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35341.log 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33740.log.gz 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30440.log.gz 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35847.log 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34318.log 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29838.log.gz 2024-09-16T05:56:07.691 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29180.log 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35341.log.gz 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34318.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35847.log.gz 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30748.log 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34318.log.gz 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29108.log 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33884.log 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30748.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29180.log.gz 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.29108.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30748.log.gz 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28910.log 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29108.log.gz 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6%/var/log/ceph/ceph-client.admin.35190.log: -- replaced with /var/log/ceph/ceph-client.admin.33884.log.gz 2024-09-16T05:56:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29666.log 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29228.log 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.29666.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31061.log 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28910.log.gz 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32724.log 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29666.log.gz 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29228.log.gz 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25211.log 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31327.log 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31061.log.gz 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.35735.log/var/log/ceph/ceph-client.admin.25211.log: 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32724.log.gz 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34197.log 2024-09-16T05:56:07.693 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25211.log.gz 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.bar.35735.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.31327.log.gz 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32217.log 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.34197.log: -- /var/log/ceph/ceph-client.admin.30318.log 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.bar.35735.log.gz 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32027.log 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34197.log.gz 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29378.log 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32217.log.gz 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32027.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30318.log.gz 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30772.log 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25259.log 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32027.log.gz 2024-09-16T05:56:07.694 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.30772.log: /var/log/ceph/ceph-client.admin.30466.log 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29378.log.gz 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.25259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34510.log 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30772.log.gz 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25259.log.gz 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28609.log 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30392.log 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.30466.log.gz 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33033.log 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34510.log.gz 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30242.log 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28609.log.gz 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33033.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30392.log.gz 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29402.log 2024-09-16T05:56:07.695 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29156.log 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33033.log.gz 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.29402.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33403.log 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30242.log.gz 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28527.log 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29402.log.gz 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.35632.log 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29156.log.gz 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28172.log 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33403.log.gz 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.foo.35632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34582.log 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28527.log.gz 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32125.log 2024-09-16T05:56:07.696 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28172.log.gz 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34582.log: gzip -5 --verbose -- 92.6% /var/log/ceph/ceph-client.admin.31719.log 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.foo.35632.log.gz 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29303.log 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34582.log.gz 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34246.log 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32125.log.gz 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29303.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31719.log.gz 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32267.log 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29303.log.gz 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31497.log 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29426.log 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32267.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34246.log.gz 2024-09-16T05:56:07.697 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34697.log 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31497.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32267.log.gz 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32489.log 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31497.log.gz 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29426.log.gz 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30847.log 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28399.log 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34697.log.gz 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30822.log 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32489.log.gz 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28838.log 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30847.log.gz 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30822.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28399.log.gz 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32149.log 2024-09-16T05:56:07.698 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34077.log 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30822.log.gz 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35118.log 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28838.log.gz 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31595.log 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32149.log.gz 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35118.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34077.log.gz 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28028.log 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32417.log 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35118.log.gz 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.28028.log: -- /var/log/ceph/ceph-client.admin.35262.log 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31595.log.gz 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31981.log 2024-09-16T05:56:07.699 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28028.log.gz 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35262.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32417.log.gz -5 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35062.log 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33692.log 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35262.log.gz 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28194.log 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.31981.log.gz 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33692.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.35062.log.gz -5 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.28375.log 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28194.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33692.log.gz 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31933.log 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33764.log 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28194.log.gz 2024-09-16T05:56:07.700 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5/var/log/ceph/ceph-client.admin.31933.log: --verbose -- /var/log/ceph/ceph-client.admin.34462.log 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28375.log.gz 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29328.log 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31933.log.gz 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34462.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33764.log.gz 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34414.log 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28964.log 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34462.log.gz 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28125.log 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29328.log.gz 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31425.log 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34414.log.gz 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28125.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28964.log.gz 2024-09-16T05:56:07.701 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35014.log 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31013.log 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28125.log.gz 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35086.log 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31425.log.gz 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31085.log 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35014.log.gz 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35086.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31013.log.gz 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28447.log 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25163.log 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31085.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35086.log.gz 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31693.log 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31085.log.gz 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.25163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35400.log 2024-09-16T05:56:07.702 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28447.log.gz 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.25163.log.gz 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32556.log 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34894.log 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31693.log.gz 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.32556.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30370.log 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35400.log.gz 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32904.log 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32556.log.gz 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34894.log.gz/var/log/ceph/ceph-client.admin.30370.log: 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30074.log 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31473.log 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30370.log.gz 2024-09-16T05:56:07.703 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30074.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32904.log.gz 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33186.log 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30074.log.gz 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.3.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31473.log.gz 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30122.log 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33114.log 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33186.log.gz 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29616.log 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30122.log.gz 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28262.log 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30146.log 2024-09-16T05:56:07.704 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33114.log.gz 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29616.log: /var/log/ceph/ceph-client.admin.28262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29499.log 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30146.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28262.log.gz 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.29616.log.gz --verbose 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr: -- /var/log/ceph/ceph-client.admin.30416.log 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30608.log 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30146.log.gz 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28814.log 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29499.log.gz 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30416.log.gz 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28497.log 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28814.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30608.log.gz 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33282.log 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28814.log.gz 2024-09-16T05:56:07.705 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31109.log 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30266.log 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32700.log 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28497.log.gz 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31109.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.33282.log.gz 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28705.log 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31109.log.gz 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.35711.log 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30266.log.gz 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28705.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32700.log.gz 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34822.log 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.bar.35711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30989.log 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.bar.35711.log.gz 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.28705.log.gz 2024-09-16T05:56:07.706 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33451.log 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36001.log 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34822.log.gz 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30989.log.gz 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33379.log 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.36001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33788.log 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33451.log.gz 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34721.log 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36001.log.gz 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33379.log.gz 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33138.log 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31863.log 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33788.log.gz 2024-09-16T05:56:07.707 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35310.log 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34721.log.gz 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33138.log.gz 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33909.log 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29790.log 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35310.log: /var/log/ceph/ceph-client.admin.33909.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31863.log.gz 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35376.log 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33909.log.gz 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29790.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35310.log.gz 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33210.log 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29954.log 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29790.log.gz 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29714.log 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33210.log.gz 2024-09-16T05:56:07.708 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35376.log.gz 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25139.log 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35976.log 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29954.log.gz 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.25139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30656.log 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29714.log.gz 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25139.log.gz 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28052.log 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31181.log 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30656.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35976.log.gz 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34637.log 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30656.log.gz 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28052.log.gz 2024-09-16T05:56:07.709 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33427.log 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34637.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31181.log.gz 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32465.log 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34846.log 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34637.log.gz 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31157.log 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33427.log.gz 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33306.log 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32465.log.gz 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34846.log.gz 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.31157.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35286.log 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34053.log 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33306.log.gz 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31157.log.gz 2024-09-16T05:56:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30943.log 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35286.log: /var/log/ceph/ceph-client.admin.34053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28423.log 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34053.log.gz 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35286.log.gz 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34918.log 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30943.log: /var/log/ceph/ceph-client.admin.28423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35238.log 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28423.log.gz 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30943.log.gz 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33087.log 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34918.log: /var/log/ceph/ceph-client.admin.35238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.35871.log 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34918.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35238.log.gz 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.35494.log 2024-09-16T05:56:07.711 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33087.log: /var/log/ceph/ceph-client.bar.35871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33162.log 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33087.log.gz 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.bar.35871.log.gz 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31279.log 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.foo.35494.log: /var/log/ceph/ceph-client.admin.33162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33860.log 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.foo.35494.log.gz 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33572.log 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33162.log.gz 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31279.log.gz 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33860.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32342.log 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33572.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33860.log.gz 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32983.log 2024-09-16T05:56:07.712 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32342.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33572.log.gz 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28238.log 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32342.log.gz 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33957.log 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33644.log 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32983.log.gz 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33957.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28238.log.gz 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30510.log 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33548.log 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33957.log.gz 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33644.log.gz 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33476.log 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30510.log: /var/log/ceph/ceph-client.admin.33548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32628.log 2024-09-16T05:56:07.713 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30510.log.gz 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34294.log 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33476.log: -- replaced with /var/log/ceph/ceph-client.admin.33548.log.gz 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32628.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33476.log.gz 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32628.log.gz 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28862.log 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31133.log 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34294.log.gz 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29475.log 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28862.log: /var/log/ceph/ceph-client.admin.31133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30292.log 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28862.log.gz 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31133.log.gz 2024-09-16T05:56:07.714 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28683.log 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33007.log 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29475.log.gz 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31645.log 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30292.log.gz 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28683.log.gz 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35038.log 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33007.log: gzip/var/log/ceph/ceph-client.admin.31645.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32076.log 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33007.log.gz 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.35516.log 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31645.log.gz 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28330.log 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35038.log.gz 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31449.log 2024-09-16T05:56:07.715 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32076.log.gz 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.foo.35516.log: /var/log/ceph/ceph-client.admin.28330.log: gzip 86.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.foo.35516.log.gz -- 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr: /var/log/ceph/ceph-client.admin.32802.log 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31399.log 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31449.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28330.log.gz/var/log/ceph/ceph-client.admin.32802.log: 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30026.log 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31449.log.gz 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32802.log.gz 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31399.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31399.log.gz 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32604.log 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.35779.log 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30026.log.gz 2024-09-16T05:56:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28748.log 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29591.log 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.bar.35779.log: /var/log/ceph/ceph-client.admin.28748.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32604.log.gz 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr:gzip 87.0% -- replaced with /var/log/ceph/ceph-client.bar.35779.log.gz -5 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31231.log 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28769.log 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28748.log.gz 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29591.log: /var/log/ceph/ceph-client.admin.31231.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34221.log 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29591.log.gz 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28886.log 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31231.log.gz 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34966.log 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28769.log.gz 2024-09-16T05:56:07.717 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28886.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34221.log.gz 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30895.log 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29036.log 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34966.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28886.log.gz 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31303.log 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34966.log.gz 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33836.log 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30895.log.gz 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31303.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29036.log.gz 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30098.log 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32003.log 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33836.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31303.log.gz 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34990.log 2024-09-16T05:56:07.718 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33836.log.gz 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30218.log 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30098.log.gz 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25235.log 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32003.log.gz 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.35472.log 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30218.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34990.log.gz 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.25235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28076.log 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30218.log.gz 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.foo.35472.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.25235.log.gz 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28934.log 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.foo.35472.log.gz 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33330.log 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30919.log 2024-09-16T05:56:07.719 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28934.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28076.log.gz 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.35538.log 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28934.log.gz 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33330.log.gz 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31909.log 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.foo.35538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29908.log 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30919.log.gz 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.35757.log 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31521.log 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31909.log.gz 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.bar.35757.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29908.log.gz 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34558.log 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.foo.35538.log.gz 87.4% -- replaced with /var/log/ceph/ceph-client.bar.35757.log.gz 2024-09-16T05:56:07.720 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30634.log 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31521.log: /var/log/ceph/ceph-client.admin.34558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32853.log 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30634.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34125.log 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31521.log.gz 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34558.log.gz 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34534.log 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30634.log.gz 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33668.log 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32853.log.gz 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34798.log 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34125.log.gz 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34534.log.gz 2024-09-16T05:56:07.721 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33668.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29084.log 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33668.log.gz 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29132.log 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34798.log.gz 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30170.log 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34149.log 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29084.log.gz 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34342.log 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29132.log.gz 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33500.log 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34149.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30170.log.gz 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34342.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34149.log.gz 2024-09-16T05:56:07.722 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34342.log.gz 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28553.log 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30488.log 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34769.log 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33500.log.gz 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34390.log 88.1% 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28553.log.gz 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr: 91.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29738.log 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34769.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.30488.log.gz 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29886.log 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29738.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34769.log.gz 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30965.log 2024-09-16T05:56:07.723 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34390.log.gz 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29738.log.gz 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28471.log 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25187.log 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30965.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29886.log.gz 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35142.log 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30965.log.gz 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.25187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29690.log 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28471.log.gz 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29451.log 87.2% 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25187.log.gz 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34366.log 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29690.log: /var/log/ceph/ceph-client.admin.29451.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35142.log.gz 2024-09-16T05:56:07.724 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29814.log 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29690.log.gz 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32101.log 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29451.log.gz 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29814.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.34366.log.gz -5 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32931.log 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5/var/log/ceph/ceph-client.admin.32101.log: --verbose -- /var/log/ceph/ceph-client.admin.28150.log 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29814.log.gz 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29932.log 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32101.log.gz 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28727.log 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32931.log.gz 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31621.log 2024-09-16T05:56:07.725 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28150.log.gz 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29932.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28727.log: /var/log/ceph/ceph-client.admin.28101.log 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31815.log 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.29932.log.gz 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28727.log.gz 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29978.log 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31621.log.gz 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28216.log 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28101.log.gz 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29277.log 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31815.log.gz 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31569.log 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29978.log.gz 2024-09-16T05:56:07.726 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29277.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28216.log.gz 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31957.log 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32652.log 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31569.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29277.log.gz 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30702.log 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31569.log.gz 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28988.log 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31957.log.gz 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33060.log 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30702.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32652.log.gz 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33716.log 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30702.log.gz 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31667.log 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28988.log.gz 2024-09-16T05:56:07.727 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29204.log 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33060.log.gz 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30724.log 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33716.log.gz 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30050.log 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31667.log.gz 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30724.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29204.log.gz 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32193.log 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30724.log.gz 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33234.log 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30050.log.gz 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.35424.log 2024-09-16T05:56:07.728 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33234.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32193.log.gz 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31353.log 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.foo.35424.log: /var/log/ceph/ceph-client.admin.32441.log 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33234.log.gz 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29354.log 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.foo.35424.log.gz 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32441.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31353.log.gz 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32368.log 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32441.log.gz 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32748.log 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33620.log 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29354.log.gz 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35166.log 2024-09-16T05:56:07.729 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32368.log.gz 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33620.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32748.log.gz 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29252.log 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33620.log.gz 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.4.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35166.log.gz 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34870.log 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34101.log 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31255.log 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29252.log.gz 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34101.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34870.log.gz 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34101.log.gz 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32676.log 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33524.log 2024-09-16T05:56:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28308.log 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31255.log.gz 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33524.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32676.log.gz 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32171.log 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28308.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30344.log 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33524.log.gz 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32171.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28308.log.gz 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31839.log 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32171.log.gz 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.35448.log 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30344.log.gz 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28583.log 2024-09-16T05:56:07.731 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.foo.35448.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31839.log.gz 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30002.log: 88.9% -- replaced with /var/log/ceph/ceph-client.foo.35448.log.gz 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gz 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28661.log 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35951.log 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33812.log 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28583.log.gz 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28661.log.gz 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35951.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35951.log.gz 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31207.log 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30797.log 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.33812.log: /var/log/ceph/ceph-client.admin.31207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32958.log 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30797.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33812.log.gz 2024-09-16T05:56:07.732 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31207.log.gz 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32877.log 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30797.log.gz 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28284.log 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32958.log: /var/log/ceph/ceph-client.admin.32877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35214.log 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28284.log: 87.7% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32877.log.gz -- replaced with /var/log/ceph/ceph-client.admin.32958.log.gz 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29012.log 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28284.log.gz 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32580.log 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35214.log: /var/log/ceph/ceph-client.admin.29012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30560.log 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.32580.log: 87.0% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35214.log.gz 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29012.log.gz 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32829.log 2024-09-16T05:56:07.733 INFO:teuthology.orchestra.run.smithi063.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32580.log.gz 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30584.log 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30560.log: /var/log/ceph/ceph-client.admin.32829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31743.log 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30584.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30560.log.gz 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32829.log.gz 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32318.log 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30584.log.gz 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35927.log 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31743.log: /var/log/ceph/ceph-client.admin.32318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35608.log 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35927.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.31743.log.gz 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32318.log.gz 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29862.log 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35927.log.gz 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34029.log 2024-09-16T05:56:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35608.log: /var/log/ceph/ceph-client.admin.29862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31791.log 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.34029.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35608.log.gz 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29862.log.gz 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28354.log 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34029.log.gz 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31791.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31791.log.gz 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33933.log 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30534.log 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28354.log: /var/log/ceph/ceph-client.admin.33933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29060.log 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30534.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28354.log.gz 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33933.log.gz 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31377.log 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30534.log.gz 2024-09-16T05:56:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29060.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29060.log.gz 2024-09-16T05:56:07.736 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34745.log 2024-09-16T05:56:07.736 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33355.log 2024-09-16T05:56:07.736 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31377.log: /var/log/ceph/ceph-client.admin.34745.log: /var/log/ceph/ceph-client.admin.33355.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.31377.log.gz 2024-09-16T05:56:07.736 INFO:teuthology.orchestra.run.smithi063.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34745.log.gz 2024-09-16T05:56:07.736 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33355.log.gz 2024-09-16T05:56:07.780 INFO:teuthology.orchestra.run.smithi063.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T05:56:08.074 INFO:teuthology.orchestra.run.smithi063.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T05:56:08.296 INFO:teuthology.orchestra.run.smithi006.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T05:56:08.343 INFO:teuthology.orchestra.run.smithi006.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T05:56:08.602 INFO:teuthology.orchestra.run.smithi063.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T05:56:08.617 INFO:teuthology.orchestra.run.smithi063.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T05:56:08.665 INFO:teuthology.orchestra.run.smithi006.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T05:56:08.907 INFO:teuthology.orchestra.run.smithi063.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T05:56:08.909 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-16T05:56:08.909 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m1.405s 2024-09-16T05:56:08.909 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m4.242s 2024-09-16T05:56:08.909 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m0.241s 2024-09-16T05:56:08.934 INFO:teuthology.orchestra.run.smithi006.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T05:56:08.993 INFO:teuthology.orchestra.run.smithi006.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T05:56:08.994 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-16T05:56:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m1.525s 2024-09-16T05:56:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m5.705s 2024-09-16T05:56:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.292s 2024-09-16T05:56:08.995 INFO:tasks.ceph:Archiving logs... 2024-09-16T05:56:08.995 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/remote/smithi006/log 2024-09-16T05:56:08.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T05:56:09.900 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/remote/smithi063/log 2024-09-16T05:56:09.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T05:56:10.716 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T05:56:10.727 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T05:56:10.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T05:56:10.730 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T05:56:10.795 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T05:56:10.804 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T05:56:10.804 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:56:10.806 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:56:10.817 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T05:56:10.817 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-09-16T05:56:10.817 INFO:teuthology.orchestra.run.smithi006.stdout:+hv01.front.sepi 67.205.162.81 3 u 55 64 377 0.083 -2.254 0.298 2024-09-16T05:56:10.818 INFO:teuthology.orchestra.run.smithi006.stdout:*hv02.front.sepi 63.231.80.2 3 u 57 64 377 0.107 -0.910 0.334 2024-09-16T05:56:10.818 INFO:teuthology.orchestra.run.smithi006.stdout:+hv03.front.sepi 74.6.168.72 3 u 44 64 377 0.087 -0.133 0.087 2024-09-16T05:56:10.818 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-16T05:56:10.819 INFO:teuthology.orchestra.run.smithi063.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T05:56:10.820 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================== 2024-09-16T05:56:10.820 INFO:teuthology.orchestra.run.smithi063.stdout:+hv01.front.sepi 67.205.162.81 3 u 60 64 377 0.088 -1.179 0.186 2024-09-16T05:56:10.820 INFO:teuthology.orchestra.run.smithi063.stdout:*hv02.front.sepi 63.231.80.2 3 u 53 64 377 0.082 -0.362 0.047 2024-09-16T05:56:10.820 INFO:teuthology.orchestra.run.smithi063.stdout:+hv03.front.sepi 74.6.168.72 3 u 56 64 377 0.102 +0.386 0.385 2024-09-16T05:56:10.820 INFO:teuthology.orchestra.run.smithi063.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-16T05:56:10.821 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T05:56:10.829 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T05:56:10.830 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T05:56:10.838 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T05:56:10.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T05:56:10.856 INFO:teuthology.task.internal:Duration was 1903.451501 seconds 2024-09-16T05:56:10.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T05:56:10.865 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T05:56:10.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T05:56:10.868 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T05:56:10.909 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T05:56:10.910 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-09-16T05:56:10.910 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T05:56:10.957 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-09-16T05:56:10.957 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T05:56:10.967 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T05:56:10.968 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T05:56:11.014 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T05:56:11.041 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T05:56:11.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T05:56:11.087 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T05:56:11.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T05:56:11.169 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T05:56:11.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T05:56:11.184 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T05:56:11.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T05:56:11.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:56:11.227 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:56:11.236 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-09-16T05:56:11.250 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-09-16T05:56:11.264 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:56:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:56:11.309 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:56:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:56:11.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T05:56:11.325 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T05:56:11.326 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/remote/smithi006 2024-09-16T05:56:11.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T05:56:11.375 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906162/remote/smithi063 2024-09-16T05:56:11.375 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T05:56:11.396 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T05:56:11.397 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T05:56:11.414 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T05:56:11.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T05:56:11.451 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T05:56:11.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T05:56:11.460 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T05:56:11.460 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T05:56:11.466 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T05:56:11.470 INFO:teuthology.orchestra.run.smithi006.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 05:56 /home/ubuntu/cephtest 2024-09-16T05:56:11.488 INFO:teuthology.orchestra.run.smithi063.stdout: 658636 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 05:56 /home/ubuntu/cephtest 2024-09-16T05:56:11.489 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T05:56:11.499 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T05:56:11.582 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits} duration: 1903.4515008926392 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T05:56:11.582 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T05:56:11.658 INFO:teuthology.run:pass